Buffer Sharing and Synchronization¶

The dma-buf subsystem provides the framework for sharing buffers for hardware (DMA) access across multiple device drivers and subsystems, and for synchronizing asynchronous hardware access.

This is used, for example, by drm “prime” multi-GPU support, but is of course not limited to GPU use cases.

The three main components of this are: (1) dma-buf, representing a sg_table and exposed to userspace as a file descriptor to allow passing between devices, (2) fence, which provides a mechanism to signal when one device as finished access, and (3) reservation, which manages the shared or exclusive fence(s) associated with the buffer.

Shared DMA Buffers¶

This document serves as a guide to device-driver writers on what is the dma-buf buffer sharing API, how to use it for exporting and using shared buffers.

Any device driver which wishes to be a part of DMA buffer sharing, can do so as either the ‘exporter’ of buffers, or the ‘user’ or ‘importer’ of buffers.

Say a driver A wants to use buffers created by driver B, then we call B as the exporter, and A as buffer-user/importer.

The exporter

implements and manages operations in struct dma_buf_ops for the buffer,

allows other users to share the buffer by using dma_buf sharing APIs,

manages the details of buffer allocation, wrapped int a struct dma_buf,

decides about the actual backing storage where this allocation happens,

and takes care of any migration of scatterlist - for all (shared) users of this buffer.

The buffer-user

is one of (many) sharing users of the buffer.

doesn’t need to worry about how the buffer is allocated, or where.

and needs a mechanism to get access to the scatterlist that makes up this buffer in memory, mapped into its own address space, so it can access the same area of memory. This interface is provided by struct dma_buf_attachment.

Any exporters or users of the dma-buf buffer sharing framework must have a ‘select DMA_SHARED_BUFFER’ in their respective Kconfigs.

Userspace Interface Notes¶

Mostly a DMA buffer file descriptor is simply an opaque object for userspace, and hence the generic interface exposed is very minimal. There’s a few things to consider though:

Since kernel 3.12 the dma-buf FD supports the llseek system call, but only with offset=0 and whence=SEEK_END|SEEK_SET. SEEK_SET is supported to allow the usual size discover pattern size = SEEK_END(0); SEEK_SET(0). Every other llseek operation will report -EINVAL.

If llseek on dma-buf FDs isn’t support the kernel will report -ESPIPE for all cases. Userspace can use this to detect support for discovering the dma-buf size using llseek.
In order to avoid fd leaks on exec, the FD_CLOEXEC flag must be set on the file descriptor. This is not just a resource leak, but a potential security hole. It could give the newly exec’d application access to buffers, via the leaked fd, to which it should otherwise not be permitted access.

The problem with doing this via a separate fcntl() call, versus doing it atomically when the fd is created, is that this is inherently racy in a multi-threaded app[3]. The issue is made worse when it is library code opening/creating the file descriptor, as the application may not even be aware of the fd’s.

To avoid this problem, userspace must have a way to request O_CLOEXEC flag be set when the dma-buf fd is created. So any API provided by the exporting driver to create a dmabuf fd must provide a way to let userspace control setting of O_CLOEXEC flag passed in to dma_buf_fd().
Memory mapping the contents of the DMA buffer is also supported. See the discussion below on CPU Access to DMA Buffer Objects for the full details.
The DMA buffer FD is also pollable, see Fence Poll Support below for details.

Basic Operation and Device DMA Access¶

For device DMA access to a shared DMA buffer the usual sequence of operations is fairly simple:

The exporter defines his exporter instance using DEFINE_DMA_BUF_EXPORT_INFO() and calls dma_buf_export() to wrap a private buffer object into a dma_buf. It then exports that dma_buf to userspace as a file descriptor by calling dma_buf_fd().
Userspace passes this file-descriptors to all drivers it wants this buffer to share with: First the filedescriptor is converted to a dma_buf using dma_buf_get(). The the buffer is attached to the device using dma_buf_attach().

Up to this stage the exporter is still free to migrate or reallocate the backing storage.
Once the buffer is attached to all devices userspace can inniate DMA access to the shared buffer. In the kernel this is done by calling dma_buf_map_attachment() and dma_buf_unmap_attachment().
Once a driver is done with a shared buffer it needs to call dma_buf_detach() (after cleaning up any mappings) and then release the reference acquired with dma_buf_get by calling dma_buf_put().

For the detailed semantics exporters are expected to implement see dma_buf_ops.

CPU Access to DMA Buffer Objects¶

There are mutliple reasons for supporting CPU access to a dma buffer object:

Fallback operations in the kernel, for example when a device is connected over USB and the kernel needs to shuffle the data around first before sending it away. Cache coherency is handled by braketing any transactions with calls to dma_buf_begin_cpu_access() and dma_buf_end_cpu_access() access.

To support dma_buf objects residing in highmem cpu access is page-based using an api similar to kmap. Accessing a dma_buf is done in aligned chunks of PAGE_SIZE size. Before accessing a chunk it needs to be mapped, which returns a pointer in kernel virtual address space. Afterwards the chunk needs to be unmapped again. There is no limit on how often a given chunk can be mapped and unmapped, i.e. the importer does not need to call begin_cpu_access again before mapping the same chunk again.

Interfaces::

void *dma_buf_kmap(struct dma_buf *, unsigned long); void dma_buf_kunmap(struct dma_buf *, unsigned long, void *);

There are also atomic variants of these interfaces. Like for kmap they facilitate non-blocking fast-paths. Neither the importer nor the exporter (in the callback) is allowed to block when using these.

Interfaces::

void *dma_buf_kmap_atomic(struct dma_buf *, unsigned long); void dma_buf_kunmap_atomic(struct dma_buf *, unsigned long, void *);

For importers all the restrictions of using kmap apply, like the limited supply of kmap_atomic slots. Hence an importer shall only hold onto at max 2 atomic dma_buf kmaps at the same time (in any given process context).

dma_buf kmap calls outside of the range specified in begin_cpu_access are undefined. If the range is not PAGE_SIZE aligned, kmap needs to succeed on the partial chunks at the beginning and end but may return stale or bogus data outside of the range (in these partial chunks).

Note that these calls need to always succeed. The exporter needs to complete any preparations that might fail in begin_cpu_access.

For some cases the overhead of kmap can be too high, a vmap interface is introduced. This interface should be used very carefully, as vmalloc space is a limited resources on many architectures.

Interfaces::

void *dma_buf_vmap(struct dma_buf *dmabuf) void dma_buf_vunmap(struct dma_buf *dmabuf, void *vaddr)

The vmap call can fail if there is no vmap support in the exporter, or if it runs out of vmalloc space. Fallback to kmap should be implemented. Note that the dma-buf layer keeps a reference count for all vmap access and calls down into the exporter’s vmap function only when no vmapping exists, and only unmaps it once. Protection against concurrent vmap/vunmap calls is provided by taking the dma_buf->lock mutex.
For full compatibility on the importer side with existing userspace interfaces, which might already support mmap’ing buffers. This is needed in many processing pipelines (e.g. feeding a software rendered image into a hardware pipeline, thumbnail creation, snapshots, ...). Also, Android’s ION framework already supported this and for DMA buffer file descriptors to replace ION buffers mmap support was needed.

There is no special interfaces, userspace simply calls mmap on the dma-buf fd. But like for CPU access there’s a need to braket the actual access, which is handled by the ioctl (DMA_BUF_IOCTL_SYNC). Note that DMA_BUF_IOCTL_SYNC can fail with -EAGAIN or -EINTR, in which case it must be restarted.

Some systems might need some sort of cache coherency management e.g. when CPU and GPU domains are being accessed through dma-buf at the same time. To circumvent this problem there are begin/end coherency markers, that forward directly to existing dma-buf device drivers vfunc hooks. Userspace can make use of those markers through the DMA_BUF_IOCTL_SYNC ioctl. The sequence would be used like following:
mmap dma-buf fd

for each drawing/upload cycle in CPU 1. SYNC_START ioctl, 2. read/write to mmap area 3. SYNC_END ioctl. This can be repeated as often as you want (with the new data being consumed by say the GPU or the scanout device)

munmap once you don’t need the buffer any more
For correctness and optimal performance, it is always required to use SYNC_START and SYNC_END before and after, respectively, when accessing the mapped address. Userspace cannot rely on coherent access, even when there are systems where it just works without calling these ioctls.
And as a CPU fallback in userspace processing pipelines.

Similar to the motivation for kernel cpu access it is again important that the userspace code of a given importing subsystem can use the same interfaces with a imported dma-buf buffer object as with a native buffer object. This is especially important for drm where the userspace part of contemporary OpenGL, X, and other drivers is huge, and reworking them to use a different way to mmap a buffer rather invasive.

The assumption in the current dma-buf interfaces is that redirecting the initial mmap is all that’s needed. A survey of some of the existing subsystems shows that no driver seems to do any nefarious thing like syncing up with outstanding asynchronous processing on the device or allocating special resources at fault time. So hopefully this is good enough, since adding interfaces to intercept pagefaults and allow pte shootdowns would increase the complexity quite a bit.

Interface::

int dma_buf_mmap(struct dma_buf *, struct vm_area_struct *,

unsigned long);

If the importing subsystem simply provides a special-purpose mmap call to set up a mapping in userspace, calling do_mmap with dma_buf->file will equally achieve that for a dma-buf object.

Fence Poll Support¶

To support cross-device and cross-driver synchronization of buffer access implicit fences (represented internally in the kernel with struct fence) can be attached to a dma_buf. The glue for that and a few related things are provided in the reservation_object structure.

Userspace can query the state of these implicitly tracked fences using poll() and related system calls:

Checking for POLLIN, i.e. read access, can be use to query the state of the most recent write or exclusive fence.
Checking for POLLOUT, i.e. write access, can be used to query the state of all attached fences, shared and exclusive ones.

Note that this only signals the completion of the respective fences, i.e. the DMA transfers are complete. Cache flushing and any other necessary preparations before CPU access can begin still need to happen.

Kernel Functions and Structures Reference¶

struct dma_buf * dma_buf_export(const struct dma_buf_export_info * exp_info)¶: Creates a new dma_buf, and associates an anon file with this buffer, so it can be exported. Also connect the allocator specific data and ops to the buffer. Additionally, provide a name string for exporter; useful in debugging.

Parameters

const struct dma_buf_export_info * exp_info: [in] holds all the export related information provided by the exporter. see struct dma_buf_export_info for further details.

Description

Returns, on success, a newly created dma_buf object, which wraps the supplied private data and operations for dma_buf_ops. On either missing ops, or error in allocating struct dma_buf, will return negative error.

For most cases the easiest way to create exp_info is through the DEFINE_DMA_BUF_EXPORT_INFO macro.

int dma_buf_fd(struct dma_buf * dmabuf, int flags)¶: returns a file descriptor for the given dma_buf

Parameters

struct dma_buf * dmabuf: [in] pointer to dma_buf for which fd is required.
int flags: [in] flags to give to fd

Description

On success, returns an associated ‘fd’. Else, returns error.

struct dma_buf * dma_buf_get(int fd)¶: returns the dma_buf structure related to an fd

Parameters

int fd: [in] fd associated with the dma_buf to be returned

Description

On success, returns the dma_buf structure associated with an fd; uses file’s refcounting done by fget to increase refcount. returns ERR_PTR otherwise.

void dma_buf_put(struct dma_buf * dmabuf)¶: decreases refcount of the buffer

Parameters

struct dma_buf * dmabuf: [in] buffer to reduce refcount of

Description

Uses file’s refcounting done implicitly by fput().

If, as a result of this call, the refcount becomes 0, the ‘release’ file operation related to this fd is called. It calls dma_buf_ops.release vfunc in turn, and frees the memory allocated for dmabuf when exported.

struct dma_buf_attachment * dma_buf_attach(struct dma_buf * dmabuf, struct device * dev)¶: Add the device to dma_buf’s attachments list; optionally, calls attach() of dma_buf_ops to allow device-specific attach functionality

Parameters

struct dma_buf * dmabuf: [in] buffer to attach device to.
struct device * dev: [in] device to be attached.

Description

Returns struct dma_buf_attachment pointer for this attachment. Attachments must be cleaned up by calling dma_buf_detach().

Return

A pointer to newly created dma_buf_attachment on success, or a negative error code wrapped into a pointer on failure.

Note that this can fail if the backing storage of dmabuf is in a place not accessible to dev, and cannot be moved to a more suitable place. This is indicated with the error code -EBUSY.

void dma_buf_detach(struct dma_buf * dmabuf, struct dma_buf_attachment * attach)¶: Remove the given attachment from dmabuf’s attachments list; optionally calls detach() of dma_buf_ops for device-specific detach

Parameters

struct dma_buf * dmabuf: [in] buffer to detach from.
struct dma_buf_attachment * attach: [in] attachment to be detached; is free’d after this call.

Description

Clean up a device attachment obtained by calling dma_buf_attach().

struct sg_table * dma_buf_map_attachment(struct dma_buf_attachment * attach, enum dma_data_direction direction)¶: Returns the scatterlist table of the attachment; mapped into _device_ address space. Is a wrapper for map_dma_buf() of the dma_buf_ops.

Parameters

struct dma_buf_attachment * attach: [in] attachment whose scatterlist is to be returned
enum dma_data_direction direction: [in] direction of DMA transfer

Description

Returns sg_table containing the scatterlist to be returned; returns ERR_PTR on error. May return -EINTR if it is interrupted by a signal.

A mapping must be unmapped again using dma_buf_map_attachment(). Note that the underlying backing storage is pinned for as long as a mapping exists, therefore users/importers should not hold onto a mapping for undue amounts of time.

void dma_buf_unmap_attachment(struct dma_buf_attachment * attach, struct sg_table * sg_table, enum dma_data_direction direction)¶: unmaps and decreases usecount of the buffer;might deallocate the scatterlist associated. Is a wrapper for unmap_dma_buf() of dma_buf_ops.

Parameters

struct dma_buf_attachment * attach: [in] attachment to unmap buffer from
struct sg_table * sg_table: [in] scatterlist info of the buffer to unmap
enum dma_data_direction direction: [in] direction of DMA transfer

Description

This unmaps a DMA mapping for attached obtained by dma_buf_map_attachment().

int dma_buf_begin_cpu_access(struct dma_buf * dmabuf, enum dma_data_direction direction)¶: Must be called before accessing a dma_buf from the cpu in the kernel context. Calls begin_cpu_access to allow exporter-specific preparations. Coherency is only guaranteed in the specified range for the specified access direction.

Parameters

struct dma_buf * dmabuf: [in] buffer to prepare cpu access for.
enum dma_data_direction direction: [in] length of range for cpu access.

Description

After the cpu access is complete the caller should call dma_buf_end_cpu_access(). Only when cpu access is braketed by both calls is it guaranteed to be coherent with other DMA access.

Can return negative error values, returns 0 on success.

int dma_buf_end_cpu_access(struct dma_buf * dmabuf, enum dma_data_direction direction)¶: Must be called after accessing a dma_buf from the cpu in the kernel context. Calls end_cpu_access to allow exporter-specific actions. Coherency is only guaranteed in the specified range for the specified access direction.

Parameters

struct dma_buf * dmabuf: [in] buffer to complete cpu access for.
enum dma_data_direction direction: [in] length of range for cpu access.

Description

This terminates CPU access started with dma_buf_begin_cpu_access().

Can return negative error values, returns 0 on success.

void * dma_buf_kmap_atomic(struct dma_buf * dmabuf, unsigned long page_num)¶: Map a page of the buffer object into kernel address space. The same restrictions as for kmap_atomic and friends apply.

Parameters

struct dma_buf * dmabuf: [in] buffer to map page from.
unsigned long page_num: [in] page in PAGE_SIZE units to map.

Description

This call must always succeed, any necessary preparations that might fail need to be done in begin_cpu_access.

void dma_buf_kunmap_atomic(struct dma_buf * dmabuf, unsigned long page_num, void * vaddr)¶: Unmap a page obtained by dma_buf_kmap_atomic.

Parameters

struct dma_buf * dmabuf: [in] buffer to unmap page from.
unsigned long page_num: [in] page in PAGE_SIZE units to unmap.
void * vaddr: [in] kernel space pointer obtained from dma_buf_kmap_atomic.

Description

This call must always succeed.

void * dma_buf_kmap(struct dma_buf * dmabuf, unsigned long page_num)¶: Map a page of the buffer object into kernel address space. The same restrictions as for kmap and friends apply.

Parameters

struct dma_buf * dmabuf: [in] buffer to map page from.
unsigned long page_num: [in] page in PAGE_SIZE units to map.

Description

This call must always succeed, any necessary preparations that might fail need to be done in begin_cpu_access.

void dma_buf_kunmap(struct dma_buf * dmabuf, unsigned long page_num, void * vaddr)¶: Unmap a page obtained by dma_buf_kmap.

Parameters

struct dma_buf * dmabuf: [in] buffer to unmap page from.
unsigned long page_num: [in] page in PAGE_SIZE units to unmap.
void * vaddr: [in] kernel space pointer obtained from dma_buf_kmap.

Description

This call must always succeed.

int dma_buf_mmap(struct dma_buf * dmabuf, struct vm_area_struct * vma, unsigned long pgoff)¶: Setup up a userspace mmap with the given vma

Parameters

struct dma_buf * dmabuf: [in] buffer that should back the vma
struct vm_area_struct * vma: [in] vma for the mmap
unsigned long pgoff: [in] offset in pages where this mmap should start within the dma-buf buffer.

Description

This function adjusts the passed in vma so that it points at the file of the dma_buf operation. It also adjusts the starting pgoff and does bounds checking on the size of the vma. Then it calls the exporters mmap function to set up the mapping.

Can return negative error values, returns 0 on success.

void * dma_buf_vmap(struct dma_buf * dmabuf)¶: Create virtual mapping for the buffer object into kernel address space. Same restrictions as for vmap and friends apply.

Parameters

struct dma_buf * dmabuf: [in] buffer to vmap

Description

This call may fail due to lack of virtual mapping address space. These calls are optional in drivers. The intended use for them is for mapping objects linear in kernel space for high use objects. Please attempt to use kmap/kunmap before thinking about these interfaces.

Returns NULL on error.

void dma_buf_vunmap(struct dma_buf * dmabuf, void * vaddr)¶: Unmap a vmap obtained by dma_buf_vmap.

Parameters

struct dma_buf * dmabuf: [in] buffer to vunmap
void * vaddr: [in] vmap to vunmap

struct dma_buf_ops¶: operations possible on struct dma_buf

Definition

struct dma_buf_ops {
  int (* attach) (struct dma_buf *, struct device *, struct dma_buf_attachment *);
  void (* detach) (struct dma_buf *, struct dma_buf_attachment *);
  struct sg_table * (* map_dma_buf) (struct dma_buf_attachment *, enum dma_data_direction);
  void (* unmap_dma_buf) (struct dma_buf_attachment *,struct sg_table *, enum dma_data_direction);
  void (* release) (struct dma_buf *);
  int (* begin_cpu_access) (struct dma_buf *, enum dma_data_direction);
  int (* end_cpu_access) (struct dma_buf *, enum dma_data_direction);
  void *(* map_atomic) (struct dma_buf *, unsigned long);
  void (* unmap_atomic) (struct dma_buf *, unsigned long, void *);
  void *(* map) (struct dma_buf *, unsigned long);
  void (* unmap) (struct dma_buf *, unsigned long, void *);
  int (* mmap) (struct dma_buf *, struct vm_area_struct *vma);
  void *(* vmap) (struct dma_buf *);
  void (* vunmap) (struct dma_buf *, void *vaddr);
};

Members

attach

This is called from dma_buf_attach() to make sure that a given device can access the provided dma_buf. Exporters which support buffer objects in special locations like VRAM or device-specific carveout areas should check whether the buffer could be move to system memory (or directly accessed by the provided device), and otherwise need to fail the attach operation.

The exporter should also in general check whether the current allocation fullfills the DMA constraints of the new device. If this is not the case, and the allocation cannot be moved, it should also fail the attach operation.

Any exporter-private housekeeping data can be stored in the dma_buf_attachment.priv pointer.

This callback is optional.

Returns:

0 on success, negative error code on failure. It might return -EBUSY to signal that backing storage is already allocated and incompatible with the requirements of requesting device.

detach

This is called by dma_buf_detach() to release a dma_buf_attachment. Provided so that exporters can clean up any housekeeping for an dma_buf_attachment.

This callback is optional.

map_dma_buf

This is called by dma_buf_map_attachment() and is used to map a shared dma_buf into device address space, and it is mandatory. It can only be called if attach has been called successfully. This essentially pins the DMA buffer into place, and it cannot be moved any more

This call may sleep, e.g. when the backing storage first needs to be allocated, or moved to a location suitable for all currently attached devices.

Note that any specific buffer attributes required for this function should get added to device_dma_parameters accessible via device.dma_params from the dma_buf_attachment. The attach callback should also check these constraints.

If this is being called for the first time, the exporter can now choose to scan through the list of attachments for this buffer, collate the requirements of the attached devices, and choose an appropriate backing storage for the buffer.

Based on enum dma_data_direction, it might be possible to have multiple users accessing at the same time (for reading, maybe), or any other kind of sharing that the exporter might wish to make available to buffer-users.

Returns:

A sg_table scatter list of or the backing storage of the DMA buffer, already mapped into the device address space of the device attached with the provided dma_buf_attachment.

On failure, returns a negative error value wrapped into a pointer. May also return -EINTR when a signal was received while being blocked.

unmap_dma_buf

This is called by dma_buf_unmap_attachment() and should unmap and release the sg_table allocated in map_dma_buf, and it is mandatory. It should also unpin the backing storage if this is the last mapping of the DMA buffer, it the exporter supports backing storage migration.

release

Called after the last dma_buf_put to release the dma_buf, and mandatory.

begin_cpu_access

This is called from dma_buf_begin_cpu_access() and allows the exporter to ensure that the memory is actually available for cpu access - the exporter might need to allocate or swap-in and pin the backing storage. The exporter also needs to ensure that cpu access is coherent for the access direction. The direction can be used by the exporter to optimize the cache flushing, i.e. access with a different direction (read instead of write) might return stale or even bogus data (e.g. when the exporter needs to copy the data to temporary storage).

This callback is optional.

FIXME: This is both called through the DMA_BUF_IOCTL_SYNC command from userspace (where storage shouldn’t be pinned to avoid handing de-factor mlock rights to userspace) and for the kernel-internal users of the various kmap interfaces, where the backing storage must be pinned to guarantee that the atomic kmap calls can succeed. Since there’s no in-kernel users of the kmap interfaces yet this isn’t a real problem.

Returns:

0 on success or a negative error code on failure. This can for example fail when the backing storage can’t be allocated. Can also return -ERESTARTSYS or -EINTR when the call has been interrupted and needs to be restarted.

end_cpu_access

This is called from dma_buf_end_cpu_access() when the importer is done accessing the CPU. The exporter can use this to flush caches and unpin any resources pinned in begin_cpu_access. The result of any dma_buf kmap calls after end_cpu_access is undefined.

This callback is optional.

Returns:

0 on success or a negative error code on failure. Can return -ERESTARTSYS or -EINTR when the call has been interrupted and needs to be restarted.

map_atomic

maps a page from the buffer into kernel address space, users may not block until the subsequent unmap call. This callback must not sleep.

unmap_atomic

[optional] unmaps a atomically mapped page from the buffer. This Callback must not sleep.

map

maps a page from the buffer into kernel address space.

unmap

[optional] unmaps a page from the buffer.

mmap

This callback is used by the dma_buf_mmap() function

Note that the mapping needs to be incoherent, userspace is expected to braket CPU access using the DMA_BUF_IOCTL_SYNC interface.

Because dma-buf buffers have invariant size over their lifetime, the dma-buf core checks whether a vma is too large and rejects such mappings. The exporter hence does not need to duplicate this check. Drivers do not need to check this themselves.

If an exporter needs to manually flush caches and hence needs to fake coherency for mmap support, it needs to be able to zap all the ptes pointing at the backing storage. Now linux mm needs a struct address_space associated with the struct file stored in vma->vm_file to do that with the function unmap_mapping_range. But the dma_buf framework only backs every dma_buf fd with the anon_file struct file, i.e. all dma_bufs share the same file.

Hence exporters need to setup their own file (and address_space) association by setting vma->vm_file and adjusting vma->vm_pgoff in the dma_buf mmap callback. In the specific case of a gem driver the exporter could use the shmem file already provided by gem (and set vm_pgoff = 0). Exporters can then zap ptes by unmapping the corresponding range of the struct address_space associated with their own file.

This callback is optional.

Returns:

0 on success or a negative error code on failure.

vmap

[optional] creates a virtual mapping for the buffer into kernel address space. Same restrictions as for vmap and friends apply.

vunmap

[optional] unmaps a vmap from the buffer

struct dma_buf¶: shared buffer object

Definition

struct dma_buf {
  size_t size;
  struct file * file;
  struct list_head attachments;
  const struct dma_buf_ops * ops;
  struct mutex lock;
  unsigned vmapping_counter;
  void * vmap_ptr;
  const char * exp_name;
  struct module * owner;
  struct list_head list_node;
  void * priv;
  struct reservation_object * resv;
  wait_queue_head_t poll;
  struct dma_buf_poll_cb_t cb_excl;
  struct dma_buf_poll_cb_t cb_shared;
};

Members

size: size of the buffer
file: file pointer used for sharing buffers across, and for refcounting.
attachments: list of dma_buf_attachment that denotes all devices attached.
ops: dma_buf_ops associated with this buffer object.
lock: used internally to serialize list manipulation, attach/detach and vmap/unmap
vmapping_counter: used internally to refcnt the vmaps
vmap_ptr: the current vmap ptr if vmapping_counter > 0
exp_name: name of the exporter; useful for debugging.
owner: pointer to exporter module; used for refcounting when exporter is a kernel module.
list_node: node for dma_buf accounting and debugging.
priv: exporter specific private data for this buffer object.
resv: reservation object linked to this dma-buf
poll: for userspace poll support
cb_excl: for userspace poll support
cb_shared: for userspace poll support

Description

This represents a shared buffer, created by calling dma_buf_export(). The userspace representation is a normal file descriptor, which can be created by calling dma_buf_fd().

Shared dma buffers are reference counted using dma_buf_put() and get_dma_buf().

Device DMA access is handled by the separate struct dma_buf_attachment.

struct dma_buf_attachment¶: holds device-buffer attachment data

Definition

struct dma_buf_attachment {
  struct dma_buf * dmabuf;
  struct device * dev;
  struct list_head node;
  void * priv;
};

Members

dmabuf: buffer for this attachment.
dev: device attached to the buffer.
node: list of dma_buf_attachment.
priv: exporter specific attachment data.

Description

This structure holds the attachment information between the dma_buf buffer and its user device(s). The list contains one attachment struct per device attached to the buffer.

An attachment is created by calling dma_buf_attach(), and released again by calling dma_buf_detach(). The DMA mapping itself needed to initiate a transfer is created by dma_buf_map_attachment() and freed again by calling dma_buf_unmap_attachment().

struct dma_buf_export_info¶: holds information needed to export a dma_buf

Definition

struct dma_buf_export_info {
  const char * exp_name;
  struct module * owner;
  const struct dma_buf_ops * ops;
  size_t size;
  int flags;
  struct reservation_object * resv;
  void * priv;
};

Members

exp_name: name of the exporter - useful for debugging.
owner: pointer to exporter module - used for refcounting kernel module
ops: Attach allocator-defined dma buf ops to the new buffer
size: Size of the buffer
flags: mode flags for the file
resv: reservation-object, NULL to allocate default one
priv: Attach private data of allocator to this buffer

Description

This structure holds the information required to export the buffer. Used with dma_buf_export() only.

DEFINE_DMA_BUF_EXPORT_INFO(name)¶: helper macro for exporters

Parameters

name: export-info name

Description

DEFINE_DMA_BUF_EXPORT_INFO macro defines the struct dma_buf_export_info, zeroes it out and pre-populates exp_name in it.

void get_dma_buf(struct dma_buf * dmabuf)¶: convenience wrapper for get_file.

Parameters

struct dma_buf * dmabuf: [in] pointer to dma_buf

Description

Increments the reference count on the dma-buf, needed in case of drivers that either need to create additional references to the dmabuf on the kernel side. For example, an exporter that needs to keep a dmabuf ptr so that subsequent exports don’t create a new dmabuf.

Reservation Objects¶

The reservation object provides a mechanism to manage shared and exclusive fences associated with a buffer. A reservation object can have attached one exclusive fence (normally associated with write operations) or N shared fences (read operations). The RCU mechanism is used to protect read access to fences from locked write-side updates.

int reservation_object_reserve_shared(struct reservation_object * obj)¶: Reserve space to add a shared fence to a reservation_object.

Parameters

struct reservation_object * obj: reservation object

Description

Should be called before reservation_object_add_shared_fence(). Must be called with obj->lock held.

RETURNS Zero for success, or -errno

void reservation_object_add_shared_fence(struct reservation_object * obj, struct dma_fence * fence)¶: Add a fence to a shared slot

Parameters

struct reservation_object * obj: the reservation object
struct dma_fence * fence: the shared fence to add

Description

Add a fence to a shared slot, obj->lock must be held, and reservation_object_reserve_shared() has been called.

void reservation_object_add_excl_fence(struct reservation_object * obj, struct dma_fence * fence)¶: Add an exclusive fence.

Parameters

struct reservation_object * obj: the reservation object
struct dma_fence * fence: the shared fence to add

Description

Add a fence to the exclusive slot. The obj->lock must be held.

int reservation_object_copy_fences(struct reservation_object * dst, struct reservation_object * src)¶: Copy all fences from src to dst.

Parameters

struct reservation_object * dst: the destination reservation object
struct reservation_object * src: the source reservation object

Description

Copy all fences from src to dst. Both src->lock as well as dst-lock must be held.

int reservation_object_get_fences_rcu(struct reservation_object * obj, struct dma_fence ** pfence_excl, unsigned * pshared_count, struct dma_fence *** pshared)¶: Get an object’s shared and exclusive fences without update side lock held

Parameters

struct reservation_object * obj: the reservation object
struct dma_fence ** pfence_excl: the returned exclusive fence (or NULL)
unsigned * pshared_count: the number of shared fences returned
struct dma_fence *** pshared: the array of shared fence ptrs returned (array is krealloc’d to the required size, and must be freed by caller)

Description

RETURNS Zero or -errno

long reservation_object_wait_timeout_rcu(struct reservation_object * obj, bool wait_all, bool intr, unsigned long timeout)¶: Wait on reservation’s objects shared and/or exclusive fences.

Parameters

struct reservation_object * obj: the reservation object
bool wait_all: if true, wait on all fences, else wait on just exclusive fence
bool intr: if true, do interruptible wait
unsigned long timeout: timeout value in jiffies or zero to return immediately

Description

RETURNS Returns -ERESTARTSYS if interrupted, 0 if the wait timed out, or greater than zer on success.

bool reservation_object_test_signaled_rcu(struct reservation_object * obj, bool test_all)¶: Test if a reservation object’s fences have been signaled.

Parameters

struct reservation_object * obj: the reservation object
bool test_all: if true, test all fences, otherwise only test the exclusive fence

Description

RETURNS true if all fences signaled, else false

struct reservation_object_list¶: a list of shared fences

Definition

struct reservation_object_list {
  struct rcu_head rcu;
  u32 shared_count;
  u32 shared_max;
  struct dma_fence __rcu * shared;
};

Members

rcu: for internal use
shared_count: table of shared fences
shared_max: for growing shared fence table
shared: shared fence table

struct reservation_object¶: a reservation object manages fences for a buffer

Definition

struct reservation_object {
  struct ww_mutex lock;
  seqcount_t seq;
  struct dma_fence __rcu * fence_excl;
  struct reservation_object_list __rcu * fence;
  struct reservation_object_list * staged;
};

Members

lock: update side lock
seq: sequence count for managing RCU read-side synchronization
fence_excl: the exclusive fence, if there is one currently
fence: list of current shared fences
staged: staged copy of shared fences for RCU updates

void reservation_object_init(struct reservation_object * obj)¶: initialize a reservation object

Parameters

struct reservation_object * obj: the reservation object

void reservation_object_fini(struct reservation_object * obj)¶: destroys a reservation object

Parameters

struct reservation_object * obj: the reservation object

struct reservation_object_list * reservation_object_get_list(struct reservation_object * obj)¶: get the reservation object’s shared fence list, with update-side lock held

Parameters

struct reservation_object * obj: the reservation object

Description

Returns the shared fence list. Does NOT take references to the fence. The obj->lock must be held.

int reservation_object_lock(struct reservation_object * obj, struct ww_acquire_ctx * ctx)¶: lock the reservation object

Parameters

struct reservation_object * obj: the reservation object
struct ww_acquire_ctx * ctx: the locking context

Description

Locks the reservation object for exclusive access and modification. Note, that the lock is only against other writers, readers will run concurrently with a writer under RCU. The seqlock is used to notify readers if they overlap with a writer.

As the reservation object may be locked by multiple parties in an undefined order, a #ww_acquire_ctx is passed to unwind if a cycle is detected. See ww_mutex_lock() and ww_acquire_init(). A reservation object may be locked by itself by passing NULL as ctx.

bool reservation_object_trylock(struct reservation_object * obj)¶: trylock the reservation object

Parameters

struct reservation_object * obj: the reservation object

Description

Tries to lock the reservation object for exclusive access and modification. Note, that the lock is only against other writers, readers will run concurrently with a writer under RCU. The seqlock is used to notify readers if they overlap with a writer.

Also note that since no context is provided, no deadlock protection is possible.

Returns true if the lock was acquired, false otherwise.

void reservation_object_unlock(struct reservation_object * obj)¶: unlock the reservation object

Parameters

struct reservation_object * obj: the reservation object

Description

Unlocks the reservation object following exclusive access.

struct dma_fence * reservation_object_get_excl(struct reservation_object * obj)¶: get the reservation object’s exclusive fence, with update-side lock held

Parameters

struct reservation_object * obj: the reservation object

Description

Returns the exclusive fence (if any). Does NOT take a reference. The obj->lock must be held.

RETURNS The exclusive fence or NULL

struct dma_fence * reservation_object_get_excl_rcu(struct reservation_object * obj)¶: get the reservation object’s exclusive fence, without lock held.

Parameters

struct reservation_object * obj: the reservation object

Description

If there is an exclusive fence, this atomically increments it’s reference count and returns it.

RETURNS The exclusive fence or NULL if none

DMA Fences¶

u64 dma_fence_context_alloc(unsigned num)¶: allocate an array of fence contexts

Parameters

unsigned num: [in] amount of contexts to allocate

Description

This function will return the first index of the number of fences allocated. The fence context is used for setting fence->context to a unique number.

int dma_fence_signal_locked(struct dma_fence * fence)¶: signal completion of a fence

Parameters

struct dma_fence * fence: the fence to signal

Description

Signal completion for software callbacks on a fence, this will unblock dma_fence_wait() calls and run all the callbacks added with dma_fence_add_callback(). Can be called multiple times, but since a fence can only go from unsignaled to signaled state, it will only be effective the first time.

Unlike dma_fence_signal, this function must be called with fence->lock held.

int dma_fence_signal(struct dma_fence * fence)¶: signal completion of a fence

Parameters

struct dma_fence * fence: the fence to signal

Description

Signal completion for software callbacks on a fence, this will unblock dma_fence_wait() calls and run all the callbacks added with dma_fence_add_callback(). Can be called multiple times, but since a fence can only go from unsignaled to signaled state, it will only be effective the first time.

signed long dma_fence_wait_timeout(struct dma_fence * fence, bool intr, signed long timeout)¶: sleep until the fence gets signaled or until timeout elapses

Parameters

struct dma_fence * fence: [in] the fence to wait on
bool intr: [in] if true, do an interruptible wait
signed long timeout: [in] timeout value in jiffies, or MAX_SCHEDULE_TIMEOUT

Description

Returns -ERESTARTSYS if interrupted, 0 if the wait timed out, or the remaining timeout in jiffies on success. Other error values may be returned on custom implementations.

Performs a synchronous wait on this fence. It is assumed the caller directly or indirectly (buf-mgr between reservation and committing) holds a reference to the fence, otherwise the fence might be freed before return, resulting in undefined behavior.

void dma_fence_enable_sw_signaling(struct dma_fence * fence)¶: enable signaling on fence

Parameters

struct dma_fence * fence: [in] the fence to enable

Description

this will request for sw signaling to be enabled, to make the fence complete as soon as possible

int dma_fence_add_callback(struct dma_fence * fence, struct dma_fence_cb * cb, dma_fence_func_t func)¶: add a callback to be called when the fence is signaled

Parameters

struct dma_fence * fence: [in] the fence to wait on
struct dma_fence_cb * cb: [in] the callback to register
dma_fence_func_t func: [in] the function to call

Description

cb will be initialized by dma_fence_add_callback, no initialization by the caller is required. Any number of callbacks can be registered to a fence, but a callback can only be registered to one fence at a time.

Note that the callback can be called from an atomic context. If fence is already signaled, this function will return -ENOENT (and not call the callback)

Add a software callback to the fence. Same restrictions apply to refcount as it does to dma_fence_wait, however the caller doesn’t need to keep a refcount to fence afterwards: when software access is enabled, the creator of the fence is required to keep the fence alive until after it signals with dma_fence_signal. The callback itself can be called from irq context.

Returns 0 in case of success, -ENOENT if the fence is already signaled and -EINVAL in case of error.

int dma_fence_get_status(struct dma_fence * fence)¶: returns the status upon completion

Parameters

struct dma_fence * fence: [in] the dma_fence to query

Description

This wraps dma_fence_get_status_locked() to return the error status condition on a signaled fence. See dma_fence_get_status_locked() for more details.

Returns 0 if the fence has not yet been signaled, 1 if the fence has been signaled without an error condition, or a negative error code if the fence has been completed in err.

bool dma_fence_remove_callback(struct dma_fence * fence, struct dma_fence_cb * cb)¶: remove a callback from the signaling list

Parameters

struct dma_fence * fence: [in] the fence to wait on
struct dma_fence_cb * cb: [in] the callback to remove

Description

Remove a previously queued callback from the fence. This function returns true if the callback is successfully removed, or false if the fence has already been signaled.

WARNING: Cancelling a callback should only be done if you really know what you’re doing, since deadlocks and race conditions could occur all too easily. For this reason, it should only ever be done on hardware lockup recovery, with a reference held to the fence.

signed long dma_fence_default_wait(struct dma_fence * fence, bool intr, signed long timeout)¶: default sleep until the fence gets signaled or until timeout elapses

Parameters

struct dma_fence * fence: [in] the fence to wait on
bool intr: [in] if true, do an interruptible wait
signed long timeout: [in] timeout value in jiffies, or MAX_SCHEDULE_TIMEOUT

Description

Returns -ERESTARTSYS if interrupted, 0 if the wait timed out, or the remaining timeout in jiffies on success. If timeout is zero the value one is returned if the fence is already signaled for consistency with other functions taking a jiffies timeout.

signed long dma_fence_wait_any_timeout(struct dma_fence ** fences, uint32_t count, bool intr, signed long timeout, uint32_t * idx)¶: sleep until any fence gets signaled or until timeout elapses

Parameters

struct dma_fence ** fences: [in] array of fences to wait on
uint32_t count: [in] number of fences to wait on
bool intr: [in] if true, do an interruptible wait
signed long timeout: [in] timeout value in jiffies, or MAX_SCHEDULE_TIMEOUT
uint32_t * idx: [out] the first signaled fence index, meaningful only on positive return

Description

Returns -EINVAL on custom fence wait implementation, -ERESTARTSYS if interrupted, 0 if the wait timed out, or the remaining timeout in jiffies on success.

Synchronous waits for the first fence in the array to be signaled. The caller needs to hold a reference to all fences in the array, otherwise a fence might be freed before return, resulting in undefined behavior.

void dma_fence_init(struct dma_fence * fence, const struct dma_fence_ops * ops, spinlock_t * lock, u64 context, unsigned seqno)¶: Initialize a custom fence.

Parameters

struct dma_fence * fence: [in] the fence to initialize
const struct dma_fence_ops * ops: [in] the dma_fence_ops for operations on this fence
spinlock_t * lock: [in] the irqsafe spinlock to use for locking this fence
u64 context: [in] the execution context this fence is run on
unsigned seqno: [in] a linear increasing sequence number for this context

Description

Initializes an allocated fence, the caller doesn’t have to keep its refcount after committing with this fence, but it will need to hold a refcount again if dma_fence_ops.enable_signaling gets called. This can be used for other implementing other types of fence.

context and seqno are used for easy comparison between fences, allowing to check which fence is later by simply using dma_fence_later.

struct dma_fence¶: software synchronization primitive

Definition

struct dma_fence {
  struct kref refcount;
  const struct dma_fence_ops * ops;
  struct rcu_head rcu;
  struct list_head cb_list;
  spinlock_t * lock;
  u64 context;
  unsigned seqno;
  unsigned long flags;
  ktime_t timestamp;
  int error;
};

Members

refcount: refcount for this fence
ops: dma_fence_ops associated with this fence
rcu: used for releasing fence with kfree_rcu
cb_list: list of all callbacks to call
lock: spin_lock_irqsave used for locking
context: execution context this fence belongs to, returned by dma_fence_context_alloc()
seqno: the sequence number of this fence inside the execution context, can be compared to decide which fence would be signaled later.
flags: A mask of DMA_FENCE_FLAG_* defined below
timestamp: Timestamp when the fence was signaled.
error: Optional, only valid if < 0, must be set before calling dma_fence_signal, indicates that the fence has completed with an error.

Description

the flags member must be manipulated and read using the appropriate atomic ops (bit_*), so taking the spinlock will not be needed most of the time.

DMA_FENCE_FLAG_SIGNALED_BIT - fence is already signaled DMA_FENCE_FLAG_TIMESTAMP_BIT - timestamp recorded for fence signaling DMA_FENCE_FLAG_ENABLE_SIGNAL_BIT - enable_signaling might have been called DMA_FENCE_FLAG_USER_BITS - start of the unused bits, can be used by the implementer of the fence for its own purposes. Can be used in different ways by different fence implementers, so do not rely on this.

Since atomic bitops are used, this is not guaranteed to be the case. Particularly, if the bit was set, but dma_fence_signal was called right before this bit was set, it would have been able to set the DMA_FENCE_FLAG_SIGNALED_BIT, before enable_signaling was called. Adding a check for DMA_FENCE_FLAG_SIGNALED_BIT after setting DMA_FENCE_FLAG_ENABLE_SIGNAL_BIT closes this race, and makes sure that after dma_fence_signal was called, any enable_signaling call will have either been completed, or never called at all.

struct dma_fence_cb¶: callback for dma_fence_add_callback

Definition

struct dma_fence_cb {
  struct list_head node;
  dma_fence_func_t func;
};

Members

node: used by dma_fence_add_callback to append this struct to fence::cb_list
func: dma_fence_func_t to call

Description

This struct will be initialized by dma_fence_add_callback, additional data can be passed along by embedding dma_fence_cb in another struct.

struct dma_fence_ops¶: operations implemented for fence

Definition

struct dma_fence_ops {
  const char * (* get_driver_name) (struct dma_fence *fence);
  const char * (* get_timeline_name) (struct dma_fence *fence);
  bool (* enable_signaling) (struct dma_fence *fence);
  bool (* signaled) (struct dma_fence *fence);
  signed long (* wait) (struct dma_fence *fence, bool intr, signed long timeout);
  void (* release) (struct dma_fence *fence);
  int (* fill_driver_data) (struct dma_fence *fence, void *data, int size);
  void (* fence_value_str) (struct dma_fence *fence, char *str, int size);
  void (* timeline_value_str) (struct dma_fence *fence, char *str, int size);
};

Members

get_driver_name: returns the driver name.
get_timeline_name: return the name of the context this fence belongs to.
enable_signaling: enable software signaling of fence.
signaled: [optional] peek whether the fence is signaled, can be null.
wait: custom wait implementation, or dma_fence_default_wait.
release: [optional] called on destruction of fence, can be null
fill_driver_data: [optional] callback to fill in free-form debug info Returns amount of bytes filled, or -errno.
fence_value_str: [optional] fills in the value of the fence as a string
timeline_value_str: [optional] fills in the current value of the timeline as a string

Description

Notes on enable_signaling: For fence implementations that have the capability for hw->hw signaling, they can implement this op to enable the necessary irqs, or insert commands into cmdstream, etc. This is called in the first wait() or add_callback() path to let the fence implementation know that there is another driver waiting on the signal (ie. hw->sw case).

This function can be called called from atomic context, but not from irq context, so normal spinlocks can be used.

A return value of false indicates the fence already passed, or some failure occurred that made it impossible to enable signaling. True indicates successful enabling.

fence->error may be set in enable_signaling, but only when false is returned.

Calling dma_fence_signal before enable_signaling is called allows for a tiny race window in which enable_signaling is called during, before, or after dma_fence_signal. To fight this, it is recommended that before enable_signaling returns true an extra reference is taken on the fence, to be released when the fence is signaled. This will mean dma_fence_signal will still be called twice, but the second time will be a noop since it was already signaled.

Notes on signaled: May set fence->error if returning true.

Notes on wait: Must not be NULL, set to dma_fence_default_wait for default implementation. the dma_fence_default_wait implementation should work for any fence, as long as enable_signaling works correctly.

Must return -ERESTARTSYS if the wait is intr = true and the wait was interrupted, and remaining jiffies if fence has signaled, or 0 if wait timed out. Can also return other error values on custom implementations, which should be treated as if the fence is signaled. For example a hardware lockup could be reported like that.

Notes on release: Can be NULL, this function allows additional commands to run on destruction of the fence. Can be called from irq context. If pointer is set to NULL, kfree will get called instead.

void dma_fence_put(struct dma_fence * fence)¶: decreases refcount of the fence

Parameters

struct dma_fence * fence: [in] fence to reduce refcount of

struct dma_fence * dma_fence_get(struct dma_fence * fence)¶: increases refcount of the fence

Parameters

struct dma_fence * fence: [in] fence to increase refcount of

Description

Returns the same fence, with refcount increased by 1.

struct dma_fence * dma_fence_get_rcu(struct dma_fence * fence)¶: get a fence from a reservation_object_list with rcu read lock

Parameters

struct dma_fence * fence: [in] fence to increase refcount of

Description

Function returns NULL if no refcount could be obtained, or the fence.

struct dma_fence * dma_fence_get_rcu_safe(struct dma_fence *__rcu * fencep)¶: acquire a reference to an RCU tracked fence

Parameters

struct dma_fence *__rcu * fencep: [in] pointer to fence to increase refcount of

Description

Function returns NULL if no refcount could be obtained, or the fence. This function handles acquiring a reference to a fence that may be reallocated within the RCU grace period (such as with SLAB_TYPESAFE_BY_RCU), so long as the caller is using RCU on the pointer to the fence.

An alternative mechanism is to employ a seqlock to protect a bunch of fences, such as used by struct reservation_object. When using a seqlock, the seqlock must be taken before and checked after a reference to the fence is acquired (as shown here).

The caller is required to hold the RCU read lock.

bool dma_fence_is_signaled_locked(struct dma_fence * fence)¶: Return an indication if the fence is signaled yet.

Parameters

struct dma_fence * fence: [in] the fence to check

Description

Returns true if the fence was already signaled, false if not. Since this function doesn’t enable signaling, it is not guaranteed to ever return true if dma_fence_add_callback, dma_fence_wait or dma_fence_enable_sw_signaling haven’t been called before.

This function requires fence->lock to be held.

bool dma_fence_is_signaled(struct dma_fence * fence)¶: Return an indication if the fence is signaled yet.

Parameters

struct dma_fence * fence: [in] the fence to check

Description

Returns true if the fence was already signaled, false if not. Since this function doesn’t enable signaling, it is not guaranteed to ever return true if dma_fence_add_callback, dma_fence_wait or dma_fence_enable_sw_signaling haven’t been called before.

It’s recommended for seqno fences to call dma_fence_signal when the operation is complete, it makes it possible to prevent issues from wraparound between time of issue and time of use by checking the return value of this function before calling hardware-specific wait instructions.

bool __dma_fence_is_later(u32 f1, u32 f2)¶: return if f1 is chronologically later than f2

Parameters

u32 f1: [in] the first fence’s seqno
u32 f2: [in] the second fence’s seqno from the same context

Description

Returns true if f1 is chronologically later than f2. Both fences must be from the same context, since a seqno is not common across contexts.

bool dma_fence_is_later(struct dma_fence * f1, struct dma_fence * f2)¶: return if f1 is chronologically later than f2

Parameters

struct dma_fence * f1: [in] the first fence from the same context
struct dma_fence * f2: [in] the second fence from the same context

Description

Returns true if f1 is chronologically later than f2. Both fences must be from the same context, since a seqno is not re-used across contexts.

struct dma_fence * dma_fence_later(struct dma_fence * f1, struct dma_fence * f2)¶: return the chronologically later fence

Parameters

struct dma_fence * f1: [in] the first fence from the same context
struct dma_fence * f2: [in] the second fence from the same context

Description

Returns NULL if both fences are signaled, otherwise the fence that would be signaled last. Both fences must be from the same context, since a seqno is not re-used across contexts.

int dma_fence_get_status_locked(struct dma_fence * fence)¶: returns the status upon completion

Parameters

struct dma_fence * fence: [in] the dma_fence to query

Description

Drivers can supply an optional error status condition before they signal the fence (to indicate whether the fence was completed due to an error rather than success). The value of the status condition is only valid if the fence has been signaled, dma_fence_get_status_locked() first checks the signal state before reporting the error status.

Returns 0 if the fence has not yet been signaled, 1 if the fence has been signaled without an error condition, or a negative error code if the fence has been completed in err.

void dma_fence_set_error(struct dma_fence * fence, int error)¶: flag an error condition on the fence

Parameters

struct dma_fence * fence: [in] the dma_fence
int error: [in] the error to store

Description

Drivers can supply an optional error status condition before they signal the fence, to indicate that the fence was completed due to an error rather than success. This must be set before signaling (so that the value is visible before any waiters on the signal callback are woken). This helper exists to help catching erroneous setting of #dma_fence.error.

signed long dma_fence_wait(struct dma_fence * fence, bool intr)¶: sleep until the fence gets signaled

Parameters

struct dma_fence * fence: [in] the fence to wait on
bool intr: [in] if true, do an interruptible wait

Description

This function will return -ERESTARTSYS if interrupted by a signal, or 0 if the fence was signaled. Other error values may be returned on custom implementations.

Performs a synchronous wait on this fence. It is assumed the caller directly or indirectly holds a reference to the fence, otherwise the fence might be freed before return, resulting in undefined behavior.

Seqno Hardware Fences¶

struct seqno_fence * to_seqno_fence(struct dma_fence * fence)¶: cast a fence to a seqno_fence

Parameters

struct dma_fence * fence: fence to cast to a seqno_fence

Description

Returns NULL if the fence is not a seqno_fence, or the seqno_fence otherwise.

void seqno_fence_init(struct seqno_fence * fence, spinlock_t * lock, struct dma_buf * sync_buf, uint32_t context, uint32_t seqno_ofs, uint32_t seqno, enum seqno_fence_condition cond, const struct dma_fence_ops * ops)¶: initialize a seqno fence

Parameters

struct seqno_fence * fence: seqno_fence to initialize
spinlock_t * lock: pointer to spinlock to use for fence
struct dma_buf * sync_buf: buffer containing the memory location to signal on
uint32_t context: the execution context this fence is a part of
uint32_t seqno_ofs: the offset within sync_buf
uint32_t seqno: the sequence # to signal on
enum seqno_fence_condition cond: fence wait condition
const struct dma_fence_ops * ops: the fence_ops for operations on this seqno fence

Description

This function initializes a struct seqno_fence with passed parameters, and takes a reference on sync_buf which is released on fence destruction.

A seqno_fence is a dma_fence which can complete in software when enable_signaling is called, but it also completes when (s32)((sync_buf)[seqno_ofs] - seqno) >= 0 is true

The seqno_fence will take a refcount on the sync_buf until it’s destroyed, but actual lifetime of sync_buf may be longer if one of the callers take a reference to it.

Certain hardware have instructions to insert this type of wait condition in the command stream, so no intervention from software would be needed. This type of fence can be destroyed before completed, however a reference on the sync_buf dma-buf can be taken. It is encouraged to re-use the same dma-buf for sync_buf, since mapping or unmapping the sync_buf to the device’s vm can be expensive.

It is recommended for creators of seqno_fence to call dma_fence_signal() before destruction. This will prevent possible issues from wraparound at time of issue vs time of check, since users can check dma_fence_is_signaled() before submitting instructions for the hardware to wait on the fence. However, when ops.enable_signaling is not called, it doesn’t have to be done as soon as possible, just before there’s any real danger of seqno wraparound.

DMA Fence Array¶

struct dma_fence_array * dma_fence_array_create(int num_fences, struct dma_fence ** fences, u64 context, unsigned seqno, bool signal_on_any)¶: Create a custom fence array

Parameters

int num_fences: [in] number of fences to add in the array
struct dma_fence ** fences: [in] array containing the fences
u64 context: [in] fence context to use
unsigned seqno: [in] sequence number to use
bool signal_on_any: [in] signal on any fence in the array

Description

Allocate a dma_fence_array object and initialize the base fence with dma_fence_init(). In case of error it returns NULL.

The caller should allocate the fences array with num_fences size and fill it with the fences it wants to add to the object. Ownership of this array is taken and dma_fence_put() is used on each fence on release.

If signal_on_any is true the fence array signals if any fence in the array signals, otherwise it signals when all fences in the array signal.

bool dma_fence_match_context(struct dma_fence * fence, u64 context)¶: Check if all fences are from the given context

Parameters

struct dma_fence * fence: [in] fence or fence array
u64 context: [in] fence context to check all fences against

Description

Checks the provided fence or, for a fence array, all fences in the array against the given context. Returns false if any fence is from a different context.

struct dma_fence_array_cb¶: callback helper for fence array

Definition

struct dma_fence_array_cb {
  struct dma_fence_cb cb;
  struct dma_fence_array * array;
};

Members

cb: fence callback structure for signaling
array: reference to the parent fence array object

struct dma_fence_array¶: fence to represent an array of fences

Definition

struct dma_fence_array {
  struct dma_fence base;
  spinlock_t lock;
  unsigned num_fences;
  atomic_t num_pending;
  struct dma_fence ** fences;
};

Members

base: fence base class
lock: spinlock for fence handling
num_fences: number of fences in the array
num_pending: fences in the array still pending
fences: array of the fences

bool dma_fence_is_array(struct dma_fence * fence)¶: check if a fence is from the array subsclass

Parameters

struct dma_fence * fence: fence to test

Description

Return true if it is a dma_fence_array and false otherwise.

struct dma_fence_array * to_dma_fence_array(struct dma_fence * fence)¶: cast a fence to a dma_fence_array

Parameters

struct dma_fence * fence: fence to cast to a dma_fence_array

Description

Returns NULL if the fence is not a dma_fence_array, or the dma_fence_array otherwise.

DMA Fence uABI/Sync File¶

struct sync_file * sync_file_create(struct dma_fence * fence)¶: creates a sync file

Parameters

struct dma_fence * fence: fence to add to the sync_fence

Description

Creates a sync_file containg fence. This function acquires and additional reference of fence for the newly-created sync_file, if it succeeds. The sync_file can be released with fput(sync_file->file). Returns the sync_file or NULL in case of error.

struct dma_fence * sync_file_get_fence(int fd)¶: get the fence related to the sync_file fd

Parameters

int fd: sync_file fd to get the fence from

Description

Ensures fd references a valid sync_file and returns a fence that represents all fence in the sync_file. On error NULL is returned.

struct sync_file¶: sync file to export to the userspace

Definition

struct sync_file {
  struct file * file;
  char user_name;
#ifdef CONFIG_DEBUG_FS
  struct list_head sync_file_list;
#endif
  wait_queue_head_t wq;
  struct dma_fence * fence;
  struct dma_fence_cb cb;
};

Members

file: file representing this fence
user_name: Name of the sync file provided by userspace, for merged fences. Otherwise generated through driver callbacks (in which case the entire array is 0).
sync_file_list: membership in global file list
wq: wait queue for fence signaling
fence: fence with the fences in the sync_file
cb: fence callback information