Userland interfaces¶

The DRM core exports several interfaces to applications, generally intended to be used through corresponding libdrm wrapper functions. In addition, drivers export device-specific interfaces for use by userspace drivers & device-aware applications through ioctls and sysfs files.

External interfaces include: memory mapping, context management, DMA operations, AGP management, vblank control, fence management, memory management, and output management.

Cover generic ioctls and sysfs layout here. We only need high-level info, since man pages should cover the rest.

libdrm Device Lookup¶

BEWARE THE DRAGONS! MIND THE TRAPDOORS!

In an attempt to warn anyone else who’s trying to figure out what’s going on here, I’ll try to summarize the story. First things first, let’s clear up the names, because the kernel internals, libdrm and the ioctls are all named differently:

GET_UNIQUE ioctl, implemented by drm_getunique is wrapped up in libdrm through the drmGetBusid function.

The libdrm drmSetBusid function is backed by the SET_UNIQUE ioctl. All that code is nerved in the kernel with drm_invalid_op().

The internal set_busid kernel functions and driver callbacks are exclusively use by the SET_VERSION ioctl, because only drm 1.0 (which is nerved) allowed userspace to set the busid through the above ioctl.

Other ioctls and functions involved are named consistently.

For anyone wondering what’s the difference between drm 1.1 and 1.4: Correctly handling pci domains in the busid on ppc. Doing this correctly was only implemented in libdrm in 2010, hence can’t be nerved yet. No one knows what’s special with drm 1.2 and 1.3.

Now the actual horror story of how device lookup in drm works. At large, there’s 2 different ways, either by busid, or by device driver name.

Opening by busid is fairly simple:

First call SET_VERSION to make sure pci domains are handled properly. As a side-effect this fills out the unique name in the master structure.
Call GET_UNIQUE to read out the unique name from the master structure, which matches the busid thanks to step 1. If it doesn’t, proceed to try the next device node.

Opening by name is slightly different:

Directly call VERSION to get the version and to match against the driver name returned by that ioctl. Note that SET_VERSION is not called, which means the the unique name for the master node just opening is _not_ filled out. This despite that with current drm device nodes are always bound to one device, and can’t be runtime assigned like with drm 1.0.
Match driver name. If it mismatches, proceed to the next device node.
Call GET_UNIQUE, and check whether the unique name has length zero (by checking that the first byte in the string is 0). If that’s not the case libdrm skips and proceeds to the next device node. Probably this is just copypasta from drm 1.0 times where a set unique name meant that the driver was in use already, but that’s just conjecture.

Long story short: To keep the open by name logic working, GET_UNIQUE must _not_ return a unique string when SET_VERSION hasn’t been called yet, otherwise libdrm breaks. Even when that unique string can’t ever change, and is totally irrelevant for actually opening the device because runtime assignable device instances were only support in drm 1.0, which is long dead. But the libdrm code in drmOpenByName somehow survived, hence this can’t be broken.

Primary Nodes, DRM Master and Authentication¶

struct drm_master is used to track groups of clients with open primary/legacy device nodes. For every struct drm_file which has had at least once successfully became the device master (either through the SET_MASTER IOCTL, or implicitly through opening the primary device node when no one else is the current master that time) there exists one drm_master. This is noted in drm_file.is_master. All other clients have just a pointer to the drm_master they are associated with.

In addition only one drm_master can be the current master for a drm_device. It can be switched through the DROP_MASTER and SET_MASTER IOCTL, or implicitly through closing/openeing the primary device node. See also drm_is_current_master().

Clients can authenticate against the current master (if it matches their own) using the GETMAGIC and AUTHMAGIC IOCTLs. Together with exchanging masters, this allows controlled access to the device for an entire group of mutually trusted clients.

bool drm_is_current_master(struct drm_file * fpriv)¶: checks whether priv is the current master

Parameters

struct drm_file * fpriv: DRM file private

Description

Checks whether fpriv is current master on its device. This decides whether a client is allowed to run DRM_MASTER IOCTLs.

Most of the modern IOCTL which require DRM_MASTER are for kernel modesetting - the current master is assumed to own the non-shareable display hardware.

struct drm_master * drm_master_get(struct drm_master * master)¶: reference a master pointer

Parameters

struct drm_master * master: struct drm_master

Description

Increments the reference count of master and returns a pointer to master.

void drm_master_put(struct drm_master ** master)¶: unreference and clear a master pointer

Parameters

struct drm_master ** master: pointer to a pointer of struct drm_master

Description

This decrements the drm_master behind master and sets it to NULL.

struct drm_master¶: drm master structure

Definition

struct drm_master {
  struct kref refcount;
  struct drm_device * dev;
  char * unique;
  int unique_len;
  struct idr magic_map;
  struct drm_lock_data lock;
  void * driver_priv;
};

Members

refcount: Refcount for this master object.
dev: Link back to the DRM device
unique: Unique identifier: e.g. busid. Protected by drm_device.master_mutex.
unique_len: Length of unique field. Protected by drm_device.master_mutex.
magic_map: Map of used authentication tokens. Protected by drm_device.master_mutex.
lock: DRI1 lock information.
driver_priv: Pointer to driver-private information.

Description

Note that master structures are only relevant for the legacy/primary device nodes, hence there can only be one per device, not one per drm_minor.

Open-Source Userspace Requirements¶

The DRM subsystem has stricter requirements than most other kernel subsystems on what the userspace side for new uAPI needs to look like. This section here explains what exactly those requirements are, and why they exist.

The short summary is that any addition of DRM uAPI requires corresponding open-sourced userspace patches, and those patches must be reviewed and ready for merging into a suitable and canonical upstream project.

GFX devices (both display and render/GPU side) are really complex bits of hardware, with userspace and kernel by necessity having to work together really closely. The interfaces, for rendering and modesetting, must be extremely wide and flexible, and therefore it is almost always impossible to precisely define them for every possible corner case. This in turn makes it really practically infeasible to differentiate between behaviour that’s required by userspace, and which must not be changed to avoid regressions, and behaviour which is only an accidental artifact of the current implementation.

Without access to the full source code of all userspace users that means it becomes impossible to change the implementation details, since userspace could depend upon the accidental behaviour of the current implementation in minute details. And debugging such regressions without access to source code is pretty much impossible. As a consequence this means:

The Linux kernel’s “no regression” policy holds in practice only for open-source userspace of the DRM subsystem. DRM developers are perfectly fine if closed-source blob drivers in userspace use the same uAPI as the open drivers, but they must do so in the exact same way as the open drivers. Creative (ab)use of the interfaces will, and in the past routinely has, lead to breakage.
Any new userspace interface must have an open-source implementation as demonstration vehicle.

The other reason for requiring open-source userspace is uAPI review. Since the kernel and userspace parts of a GFX stack must work together so closely, code review can only assess whether a new interface achieves its goals by looking at both sides. Making sure that the interface indeed covers the use-case fully leads to a few additional requirements:

The open-source userspace must not be a toy/test application, but the real thing. Specifically it needs to handle all the usual error and corner cases. These are often the places where new uAPI falls apart and hence essential to assess the fitness of a proposed interface.
The userspace side must be fully reviewed and tested to the standards of that userspace project. For e.g. mesa this means piglit testcases and review on the mailing list. This is again to ensure that the new interface actually gets the job done.
The userspace patches must be against the canonical upstream, not some vendor fork. This is to make sure that no one cheats on the review and testing requirements by doing a quick fork.
The kernel patch can only be merged after all the above requirements are met, but it must be merged before the userspace patches land. uAPI always flows from the kernel, doing things the other way round risks divergence of the uAPI definitions and header files.

These are fairly steep requirements, but have grown out from years of shared pain and experience with uAPI added hastily, and almost always regretted about just as fast. GFX devices change really fast, requiring a paradigm shift and entire new set of uAPI interfaces every few years at least. Together with the Linux kernel’s guarantee to keep existing userspace running for 10+ years this is already rather painful for the DRM subsystem, with multiple different uAPIs for the same thing co-existing. If we add a few more complete mistakes into the mix every year it would be entirely unmanageable.

Render nodes¶

DRM core provides multiple character-devices for user-space to use. Depending on which device is opened, user-space can perform a different set of operations (mainly ioctls). The primary node is always created and called card<num>. Additionally, a currently unused control node, called controlD<num> is also created. The primary node provides all legacy operations and historically was the only interface used by userspace. With KMS, the control node was introduced. However, the planned KMS control interface has never been written and so the control node stays unused to date.

With the increased use of offscreen renderers and GPGPU applications, clients no longer require running compositors or graphics servers to make use of a GPU. But the DRM API required unprivileged clients to authenticate to a DRM-Master prior to getting GPU access. To avoid this step and to grant clients GPU access without authenticating, render nodes were introduced. Render nodes solely serve render clients, that is, no modesetting or privileged ioctls can be issued on render nodes. Only non-global rendering commands are allowed. If a driver supports render nodes, it must advertise it via the DRIVER_RENDER DRM driver capability. If not supported, the primary node must be used for render clients together with the legacy drmAuth authentication procedure.

If a driver advertises render node support, DRM core will create a separate render node called renderD<num>. There will be one render node per device. No ioctls except PRIME-related ioctls will be allowed on this node. Especially GEM_OPEN will be explicitly prohibited. Render nodes are designed to avoid the buffer-leaks, which occur if clients guess the flink names or mmap offsets on the legacy interface. Additionally to this basic interface, drivers must mark their driver-dependent render-only ioctls as DRM_RENDER_ALLOW so render clients can use them. Driver authors must be careful not to allow any privileged ioctls on render nodes.

With render nodes, user-space can now control access to the render node via basic file-system access-modes. A running graphics server which authenticates clients on the privileged primary/legacy node is no longer required. Instead, a client can open the render node and is immediately granted GPU access. Communication between clients (or servers) is done via PRIME. FLINK from render node to legacy node is not supported. New clients must not use the insecure FLINK interface.

Besides dropping all modeset/global ioctls, render nodes also drop the DRM-Master concept. There is no reason to associate render clients with a DRM-Master as they are independent of any graphics server. Besides, they must work without any running master, anyway. Drivers must be able to run without a master object if they support render nodes. If, on the other hand, a driver requires shared state between clients which is visible to user-space and accessible beyond open-file boundaries, they cannot support render nodes.

IOCTL Support on Device Nodes¶

First things first, driver private IOCTLs should only be needed for drivers supporting rendering. Kernel modesetting is all standardized, and extended through properties. There are a few exceptions in some existing drivers, which define IOCTL for use by the display DRM master, but they all predate properties.

Now if you do have a render driver you always have to support it through driver private properties. There’s a few steps needed to wire all the things up.

First you need to define the structure for your IOCTL in your driver private UAPI header in include/uapi/drm/my_driver_drm.h:

struct my_driver_operation {
        u32 some_thing;
        u32 another_thing;
};

Please make sure that you follow all the best practices from Documentation/ioctl/botching-up-ioctls.txt. Note that drm_ioctl() automatically zero-extends structures, hence make sure you can add more stuff at the end, i.e. don’t put a variable sized array there.

Then you need to define your IOCTL number, using one of DRM_IO(), DRM_IOR(), DRM_IOW() or DRM_IOWR(). It must start with the DRM_IOCTL_ prefix:

##define DRM_IOCTL_MY_DRIVER_OPERATION  *         DRM_IOW(DRM_COMMAND_BASE, struct my_driver_operation)

DRM driver private IOCTL must be in the range from DRM_COMMAND_BASE to DRM_COMMAND_END. Finally you need an array of struct drm_ioctl_desc to wire up the handlers and set the access rights:

static const struct drm_ioctl_desc my_driver_ioctls[] = {
    DRM_IOCTL_DEF_DRV(MY_DRIVER_OPERATION, my_driver_operation,
            DRM_AUTH|DRM_RENDER_ALLOW),
};

And then assign this to the drm_driver.ioctls field in your driver structure.

See the separate chapter on file operations for how the driver-specific IOCTLs are wired up.

typedef int drm_ioctl_t(struct drm_device * dev, void * data, struct drm_file * file_priv)¶: DRM ioctl function type.

Parameters

struct drm_device * dev: DRM device inode
void * data: private pointer of the ioctl call
struct drm_file * file_priv: DRM file this ioctl was made on

Description

This is the DRM ioctl typedef. Note that drm_ioctl() has alrady copied data into kernel-space, and will also copy it back, depending upon the read/write settings in the ioctl command code.

typedef int drm_ioctl_compat_t(struct file * filp, unsigned int cmd, unsigned long arg)¶: compatibility DRM ioctl function type.

Parameters

struct file * filp: file pointer
unsigned int cmd: ioctl command code
unsigned long arg: DRM file this ioctl was made on

Description

Just a typedef to make declaring an array of compatibility handlers easier. New drivers shouldn’t screw up the structure layout for their ioctl structures and hence never need this.

enum drm_ioctl_flags¶: DRM ioctl flags

Constants

DRM_AUTH

This is for ioctl which are used for rendering, and require that the file descriptor is either for a render node, or if it’s a legacy/primary node, then it must be authenticated.

DRM_MASTER

This must be set for any ioctl which can change the modeset or display state. Userspace must call the ioctl through a primary node, while it is the active master.

Note that read-only modeset ioctl can also be called by unauthenticated clients, or when a master is not the currently active one.

DRM_ROOT_ONLY

Anything that could potentially wreak a master file descriptor needs to have this flag set. Current that’s only for the SETMASTER and DROPMASTER ioctl, which e.g. logind can call to force a non-behaving master (display compositor) into compliance.

This is equivalent to callers with the SYSADMIN capability.

DRM_CONTROL_ALLOW

Deprecated, do not use. Control nodes are in the process of getting removed.

DRM_UNLOCKED

Whether drm_ioctl_desc.func should be called with the DRM BKL held or not. Enforced as the default for all modern drivers, hence there should never be a need to set this flag.

DRM_RENDER_ALLOW

This is used for all ioctl needed for rendering only, for drivers which support render nodes. This should be all new render drivers, and hence it should be always set for any ioctl with DRM_AUTH set. Note though that read-only query ioctl might have this set, but have not set DRM_AUTH because they do not require authentication.

Description

Various flags that can be set in drm_ioctl_desc.flags to control how userspace can use a given ioctl.

struct drm_ioctl_desc¶: DRM driver ioctl entry

Definition

struct drm_ioctl_desc {
  unsigned int cmd;
  enum drm_ioctl_flags flags;
  drm_ioctl_t * func;
  const char * name;
};

Members

cmd: ioctl command number, without flags
flags: a bitmask of enum drm_ioctl_flags
func: handler for this ioctl
name: user-readable name for debug output

Description

For convenience it’s easier to create these using the DRM_IOCTL_DEF_DRV() macro.

DRM_IOCTL_DEF_DRV(ioctl, _func, _flags)¶: helper macro to fill out a struct drm_ioctl_desc

Parameters

ioctl: ioctl command suffix
_func: handler for the ioctl
_flags: a bitmask of enum drm_ioctl_flags

Description

Small helper macro to create a struct drm_ioctl_desc entry. The ioctl command number is constructed by prepending DRM_IOCTL\_ and passing that to DRM_IOCTL_NR().

int drm_noop(struct drm_device * dev, void * data, struct drm_file * file_priv)¶: DRM no-op ioctl implemntation

Parameters

struct drm_device * dev: DRM device for the ioctl
void * data: data pointer for the ioctl
struct drm_file * file_priv: DRM file for the ioctl call

Description

This no-op implementation for drm ioctls is useful for deprecated functionality where we can’t return a failure code because existing userspace checks the result of the ioctl, but doesn’t care about the action.

Always returns successfully with 0.

int drm_invalid_op(struct drm_device * dev, void * data, struct drm_file * file_priv)¶: DRM invalid ioctl implemntation

Parameters

struct drm_device * dev: DRM device for the ioctl
void * data: data pointer for the ioctl
struct drm_file * file_priv: DRM file for the ioctl call

Description

This no-op implementation for drm ioctls is useful for deprecated functionality where we really don’t want to allow userspace to call the ioctl any more. This is the case for old ums interfaces for drivers that transitioned to kms gradually and so kept the old legacy tables around. This only applies to radeon and i915 kms drivers, other drivers shouldn’t need to use this function.

Always fails with a return value of -EINVAL.

int drm_ioctl_permit(u32 flags, struct drm_file * file_priv)¶: Check ioctl permissions against caller

Parameters

u32 flags: ioctl permission flags.
struct drm_file * file_priv: Pointer to struct drm_file identifying the caller.

Description

Checks whether the caller is allowed to run an ioctl with the indicated permissions.

Return

Zero if allowed, -EACCES otherwise.

long drm_ioctl(struct file * filp, unsigned int cmd, unsigned long arg)¶: ioctl callback implementation for DRM drivers

Parameters

struct file * filp: file this ioctl is called on
unsigned int cmd: ioctl cmd number
unsigned long arg: user argument

Description

Looks up the ioctl function in the DRM core and the driver dispatch table, stored in drm_driver.ioctls. It checks for necessary permission by calling drm_ioctl_permit(), and dispatches to the respective function.

Return

Zero on success, negative error code on failure.

bool drm_ioctl_flags(unsigned int nr, unsigned int * flags): Check for core ioctl and return ioctl permission flags

Parameters

unsigned int nr: ioctl number
unsigned int * flags: where to return the ioctl permission flags

Description

This ioctl is only used by the vmwgfx driver to augment the access checks done by the drm core and insofar a pretty decent layering violation. This shouldn’t be used by any drivers.

Return

True if the nr corresponds to a DRM core ioctl number, false otherwise.

long drm_compat_ioctl(struct file * filp, unsigned int cmd, unsigned long arg)¶: 32bit IOCTL compatibility handler for DRM drivers

Parameters

struct file * filp: file this ioctl is called on
unsigned int cmd: ioctl cmd number
unsigned long arg: user argument

Description

Compatibility handler for 32 bit userspace running on 64 kernels. All actual IOCTL handling is forwarded to drm_ioctl(), while marshalling structures as appropriate. Note that this only handles DRM core IOCTLs, if the driver has botched IOCTL itself, it must handle those by wrapping this function.

Return

Zero on success, negative error code on failure.

Testing and validation¶

Validating changes with IGT¶

There’s a collection of tests that aims to cover the whole functionality of DRM drivers and that can be used to check that changes to DRM drivers or the core don’t regress existing functionality. This test suite is called IGT and its code can be found in https://cgit.freedesktop.org/drm/igt-gpu-tools/.

To build IGT, start by installing its build dependencies. In Debian-based systems:

# apt-get build-dep intel-gpu-tools

And in Fedora-based systems:

# dnf builddep intel-gpu-tools

Then clone the repository:

$ git clone git://anongit.freedesktop.org/drm/igt-gpu-tools

Configure the build system and start the build:

$ cd igt-gpu-tools && ./autogen.sh && make -j6

Download the piglit dependency:

$ ./scripts/run-tests.sh -d

And run the tests:

$ ./scripts/run-tests.sh -t kms -t core -s

run-tests.sh is a wrapper around piglit that will execute the tests matching the -t options. A report in HTML format will be available in ./results/html/index.html. Results can be compared with piglit.

Display CRC Support¶

DRM device drivers can provide to userspace CRC information of each frame as it reached a given hardware component (a CRC sampling “source”).

Userspace can control generation of CRCs in a given CRTC by writing to the file dri/0/crtc-N/crc/control in debugfs, with N being the index of the CRTC. Accepted values are source names (which are driver-specific) and the “auto” keyword, which will let the driver select a default source of frame CRCs for this CRTC.

Once frame CRC generation is enabled, userspace can capture them by reading the dri/0/crtc-N/crc/data file. Each line in that file contains the frame number in the first field and then a number of unsigned integer fields containing the CRC data. Fields are separated by a single space and the number of CRC fields is source-specific.

Note that though in some cases the CRC is computed in a specified way and on the frame contents as supplied by userspace (eDP 1.3), in general the CRC computation is performed in an unspecified way and on frame contents that have been already processed in also an unspecified way and thus userspace cannot rely on being able to generate matching CRC values for the frame contents that it submits. In this general case, the maximum userspace can do is to compare the reported CRCs of frames that should have the same contents.

On the driver side the implementation effort is minimal, drivers only need to implement drm_crtc_funcs.set_crc_source. The debugfs files are automatically set up if that vfunc is set. CRC samples need to be captured in the driver by calling drm_crtc_add_crc_entry().

int drm_crtc_add_crc_entry(struct drm_crtc * crtc, bool has_frame, uint32_t frame, uint32_t * crcs)¶: Add entry with CRC information for a frame

Parameters

struct drm_crtc * crtc: CRTC to which the frame belongs
bool has_frame: whether this entry has a frame number to go with
uint32_t frame: number of the frame these CRCs are about
uint32_t * crcs: array of CRC values, with length matching #drm_crtc_crc.values_cnt

Description

For each frame, the driver polls the source of CRCs for new data and calls this function to add them to the buffer from where userspace reads.

Debugfs Support¶

struct drm_info_list¶: debugfs info list entry

Definition

struct drm_info_list {
  const char * name;
  int (* show) (struct seq_file*, void*);
  u32 driver_features;
  void * data;
};

Members

name: file name
show: Show callback. seq_file->private will be set to the struct drm_info_node corresponding to the instance of this info on a given struct drm_minor.
driver_features: Required driver features for this entry
data: Driver-private data, should not be device-specific.

Description

This structure represents a debugfs file to be created by the drm core.

struct drm_info_node¶: Per-minor debugfs node structure

Definition

struct drm_info_node {
  struct drm_minor * minor;
  const struct drm_info_list * info_ent;
};

Members

minor: struct drm_minor for this node.
info_ent: template for this node.

Description

This structure represents a debugfs file, as an instantiation of a struct drm_info_list on a struct drm_minor.

FIXME:

No it doesn’t make a hole lot of sense that we duplicate debugfs entries for both the render and the primary nodes, but that’s how this has organically grown. It should probably be fixed, with a compatibility link, if needed.

int drm_debugfs_create_files(const struct drm_info_list * files, int count, struct dentry * root, struct drm_minor * minor)¶: Initialize a given set of debugfs files for DRM minor

Parameters

const struct drm_info_list * files: The array of files to create
int count: The number of files given
struct dentry * root: DRI debugfs dir entry.
struct drm_minor * minor: device minor number

Description

Create a given set of debugfs files represented by an array of struct drm_info_list in the given root directory. These files will be removed automatically on drm_debugfs_cleanup().

Sysfs Support¶

DRM provides very little additional support to drivers for sysfs interactions, beyond just all the standard stuff. Drivers who want to expose additional sysfs properties and property groups can attach them at either drm_device.dev or drm_connector.kdev.

Registration is automatically handled when calling drm_dev_register(), or drm_connector_register() in case of hot-plugged connectors. Unregistration is also automatically handled by drm_dev_unregister() and drm_connector_unregister().

void drm_sysfs_hotplug_event(struct drm_device * dev)¶: generate a DRM uevent

Parameters

struct drm_device * dev: DRM device

Description

Send a uevent for the DRM device specified by dev. Currently we only set HOTPLUG=1 in the uevent environment, but this could be expanded to deal with other types of events.

int drm_class_device_register(struct device * dev)¶: register new device with the DRM sysfs class

Parameters

struct device * dev: device to register

Description

Registers a new struct device within the DRM sysfs class. Essentially only used by ttm to have a place for its global settings. Drivers should never use this.

void drm_class_device_unregister(struct device * dev)¶: unregister device with the DRM sysfs class

Parameters

struct device * dev: device to unregister

Description

Unregisters a struct device from the DRM sysfs class. Essentially only used by ttm to have a place for its global settings. Drivers should never use this.

VBlank event handling¶

The DRM core exposes two vertical blank related ioctls:

DRM_IOCTL_WAIT_VBLANK: This takes a struct drm_wait_vblank structure as its argument, and it is used to block or request a signal when a specified vblank event occurs.
DRM_IOCTL_MODESET_CTL: This was only used for user-mode-settind drivers around modesetting changes to allow the kernel to update the vblank interrupt after mode setting, since on many devices the vertical blank counter is reset to 0 at some point during modeset. Modern drivers should not call this any more since with kernel mode setting it is a no-op.