fi\_efa(7) #VERSION# fi\_efa(7) # [NAME](#NAME) fi\_efa - The Amazon Elastic Fabric Adapter (EFA) Provider # [OVERVIEW](#OVERVIEW) The EFA provider supports the Elastic Fabric Adapter (EFA) device on Amazon EC2. EFA provides reliable and unreliable datagram send/receive with direct hardware access from userspace (OS bypass). For reliable datagram (RDM) EP type, it supports two fabric names: efa and efa-direct. The efa fabric implements a set of wire protocols to support more capabilities and features beyond the EFA device capabilities. The efa-direct fabric, on the contrary, offloads all libfabric data plane calls to the device directly without wire protocols. Compared to the efa fabric, the efa-direct fabric supports fewer capabilities and has more mode requirements for applications. But it provides a fast path to hand off application requests to the device. To use efa-direct, set the name field in fi\_fabric\_attr to efa-direct. More details and difference between the two fabrics will be presented below. # [SUPPORTED FEATURES](#SUPPORTED_FEATURES) The following features are supported: [*Endpoint types*](#Endpoint) The provider supports endpoint type *FI\_EP\_DGRAM*, and *FI\_EP\_RDM* on a new Scalable (unordered) Reliable Datagram protocol (SRD). SRD provides support for reliable datagrams and more complete error handling than typically seen with other Reliable Datagram (RD) implementations. [*RDM Endpoint capabilities*](#RDM) For the efa fabric, the following data transfer interfaces are supported: *FI\_MSG*, *FI\_TAGGED*, *FI\_SEND*, *FI\_RECV*, *FI\_RMA*, *FI\_WRITE*, *FI\_READ*, *FI\_ATOMIC*, *FI\_DIRECTED\_RECV*, *FI\_MULTI\_RECV*, and *FI\_SOURCE*. It provides SAS guarantees for data operations, and does not have a maximum message size (for all operations). For the efa-direct fabric, it supports *FI\_MSG*, *FI\_SEND*, *FI\_RECV*, *FI\_RMA*, *FI\_WRITE*, *FI\_READ*, and *FI\_SOURCE*. As mentioned earlier, it doesn’t provide SAS guarantees, and has different maximum message sizes for different operations. For MSG operations, the maximum message size is the MTU size of the efa device (approximately 8KiB). For RMA operations, the maximum message size is the maximum RDMA size of the EFA device. The exact values of these sizes can be queried by the fi\_getopt API with option names FI\_OPT\_MAX\_MSG\_SIZE and FI\_OPT\_MAX\_RMA\_SIZE [*DGRAM Endpoint capabilities*](#DGRAM) The DGRAM endpoint only supports *FI\_MSG* capability with a maximum message size of the MTU of the underlying hardware (approximately 8 KiB). [*Address vectors*](#Address) The provider supports *FI\_AV\_TABLE*. *FI\_AV\_MAP* was deprecated in Libfabric 2.x. Applications can still use *FI\_AV\_MAP* to create an address vector. But the EFA provider implementation will print a warning and switch to *FI\_AV\_TABLE*. *FI\_EVENT* is unsupported. [*Completion events*](#Completion) The provider supports *FI\_CQ\_FORMAT\_CONTEXT*, *FI\_CQ\_FORMAT\_MSG*, and *FI\_CQ\_FORMAT\_DATA*. *FI\_CQ\_FORMAT\_TAGGED* is supported on the efa fabric of RDM endpoint. The efa and efa-direct fabrics for RDM endpoints support *FI\_WAIT\_UNSPEC* and *FI\_WAIT\_FD* wait objects for blocking CQ operations (*fi\_cq\_sread*). DGRAM endpoints do not support wait objects. [*Modes*](#Modes) The provider requires the use of *FI\_MSG\_PREFIX* when running over the DGRAM endpoint. And it requires the use of *FI\_CONTEXT2* mode for DGRAM endpoint and the efa-direct fabric of RDM endpoint. The efa fabric of RDM endpoint doesn’t have these requirements. [*Memory registration modes*](#Memory) The efa fabric of RDM endpoint does not require memory registration for send and receive operations, i.e. it does not require *FI\_MR\_LOCAL*. Applications may specify *FI\_MR\_LOCAL* in the MR mode flags in order to use descriptors provided by the application. The efa-direct fabric of *FI\_EP\_RDM* endpint and the *FI\_EP\_DGRAM* endpoint only supports *FI\_MR\_LOCAL*. [*Progress*](#Progress) The *FI\_EP\_RDM* endpoint in the efa fabric supports *FI\_PROGRESS\_MANUAL*. The *FI\_EP\_RDM* and *FI\_EP\_DGRAM* endpoints in the efa-direct fabric support *FI\_PROGRESS\_AUTO*. See efa\_fabric\_comparison for a detailed comparison of efa vs efa-direct fabrics. [*Threading*](#Threading) Both RDM and DGRAM endpoints supports *FI\_THREAD\_SAFE*, *FI\_THREAD\_COMPLETION*, *FI\_THREAD\_DOMAIN*. [*Completion counters*](#Completion~2) The efa-direct fabric supports hardware completion counters backed by MSI-X(Message Signaled Interrupts Extended) hardware counters on the EFA device. Hardware counters are created via the cntr\_open\_ext GDA domain op (see below). When a hardware counter is bound to an endpoint, it is automatically attached to the QP when the endpoint is enabled. The counter is implicitly detached when the EP it is attached to is destroyed. The counter cannot be closed while it is still attached to any EP; the EP must be closed first. *fi\_cntr\_read*, *fi\_cntr\_readerr*, and *fi\_cntr\_wait* are not supported on the host for GPU memory-backed counters. Hardware counters do not internally poll the completion queue. When an operation returns *-FI\_EAGAIN*, the application must call *fi\_cq\_read* to make progress and free CQ space before retrying the operation, regardless of whether FI\_SELECTIVE\_COMPLETION is set. Failure to do so will result in CQ overrun. # [MR ABORTING](#MR_ABORTING) On the efa fabric of an *FI\_EP\_RDM* endpoint, an application can abort an in-flight transfer by calling fi\_close() on the source memory region (MR) backing it – the memory region associated with the local buffer for a send/write/read transmits operation, or the remote memory region associated with a read/write. Closing the MR signals the provider to cancel any in-flight operation that still references it. This does not apply to memory regions that are referenced by posted receive operations. The provider will fail to close a memory region that has a corresponding posted recv buffer leading to undefined behavior. The only way to abort receive operations, and/or TX operations who’s memory region is associated with posted recv’s is by closing the corresponding endpoint. If the user closes an MR on an outstanding TX operation, and its peer connection is functioning normally, the operation is guaranteed to eventually provide a single TX completion. The operation may succeed or fail, depending on timing. The exact error completion provided depends on underlying protocol used for the transfer. If a receive operation has been matched to an aborted TX operation, and its peer connection is functioning normally, the receiver side is guaranteed to write a CQ error for the matched receive. If the peer endpoint is closed, or if the peer dies, the corresponding matched TX/RX completions may never complete. Peer notification and remote-side cleanup on MR abort apply to send and tagged operations only. Device RMA operations that reference a closed MR need no peer cleanup: the initiator receives a local error completion and no provider state exists on the peer. Emulated RMA and atomic operations are not supported: closing an MR referenced by an in-flight emulated operation leads to undefined behavior. MR abort behavior is undefined unless both participants in the transaction are using libfabric version 2.7.0 or later. An older peer cannot distinguish a control message belonging to the current transfer from one belonging to an earlier, aborted transfer, so aborting transfers when either side runs an older version risks corrupting the communication state between the two endpoints. # [LIMITATIONS](#LIMITATIONS) ## [Completion events](#Completion_events) • DGRAM endpoints do not support synchronous CQ reads (*fi\_cq\_sread*) or wait objects. • *FI\_WAIT\_FD* is not supported for RDM endpoints with SHM transfers enabled. Valid wait objects are *FI\_WAIT\_NONE* or *FI\_WAIT\_UNSPEC*. Blocking read via *fi\_cq\_sread()* is supported and will wait on SHM completions. When SHM transfers are disabled, *FI\_WAIT\_FD* wait objects are supported. ## [RMA operations](#RMA_operations) • Completion events for RMA targets (*FI\_RMA\_EVENT*) is not supported. • For the efa-direct fabric, the target side of RMA operation must insert the initiator side’s address into AV before the RMA operation is kicked off, due to a current device limitation. The same limitation applies to the efa fabric when the FI\_OPT\_EFA\_HOMOGENEOUS\_PEERS option is set as true. ## fi\_cancel support • fi\_cancel is only supported in the non-zero-copy-receive mode of the efa fabric. It’s not supported in efa-direct, DGRAM endpoint, and the zero-copy receive mode of the efa fabric in RDM endpoint. ## [(Deprecated) Zero-copy receive mode](#%28Deprecated%29_Zero-copy_receive_mode) • Support for the zero-copy mode was deprecated in Libfabric v2.6 When using FI\_HMEM for AWS Neuron or Habana SynapseAI buffers, the provider requires peer to peer transaction support between the EFA and the FI\_HMEM device. Therefore, the FI\_HMEM\_P2P\_DISABLED option is not supported by the EFA provider for AWS Neuron or Habana SynapseAI. # [PROVIDER SPECIFIC ENDPOINT LEVEL OPTION](#PROVIDER_SPECIFIC_ENDPOINT_LEVEL_OPTION) [*FI\_OPT\_EFA\_RNR\_RETRY*](#FI_OPT_EFA_RNR_RETRY) Defines the number of RNR retry. The application can use it to reset RNR retry counter via the call to fi\_setopt. Note that this option must be set before the endpoint is enabled. Otherwise, the call will fail. Also note that this option only applies to RDM endpoint. [*FI\_OPT\_EFA\_EMULATED\_READ, FI\_OPT\_EFA\_EMULATED\_WRITE, FI\_OPT\_EFA\_EMULATED\_ATOMICS - bool*](#FI_OPT_EFA_EMULATED_READ,) These options only apply to the fi\_getopt() call. They are used to query the EFA provider to determine if the endpoint is emulating Read, Write, and Atomic operations (return value is true), or if these operations are assisted by hardware support (return value is false). [*FI\_OPT\_EFA\_USE\_DEVICE\_RDMA - bool*](#FI_OPT_EFA_USE_DEVICE_RDMA) This option only applies to the fi\_setopt() call. Only available if the application selects a libfabric API version >= 1.18. This option allows an application to change libfabric’s behavior with respect to RDMA transfers. Note that there is also an environment variable FI\_EFA\_USE\_DEVICE\_RDMA which the user may set as well. If the environment variable and the argument provided with this variable are in conflict, then fi\_setopt will return -FI\_EINVAL, and the environment variable will be respected. If the hardware does not support RDMA and the argument is true, then fi\_setopt will return -FI\_EOPNOTSUPP. If the application uses API version < 1.18, the argument is ignored and fi\_setopt returns -FI\_ENOPROTOOPT. The default behavior for RDMA transfers depends on API version. For API >= 1.18 RDMA is enabled by default on any hardware which supports it. For API<1.18, RDMA is enabled by default only on certain newer hardware revisions. [*FI\_OPT\_EFA\_SENDRECV\_IN\_ORDER\_ALIGNED\_128\_BYTES - bool*](#FI_OPT_EFA_SENDRECV_IN_ORDER_ALIGNED_128_BYTES) This option only applies to the fi\_setopt() call. It is used to force the endpoint to use in-order send/recv operation for each 128 bytes aligned block. Enabling the option will guarantee data inside each 128 bytes aligned block being sent and received in order, it will also guarantee data to be delivered to the receive buffer only once. If endpoint is not able to support this feature, it will return -FI\_EOPNOTSUPP for the call to fi\_setopt(). [*FI\_OPT\_EFA\_WRITE\_IN\_ORDER\_ALIGNED\_128\_BYTES - bool*](#FI_OPT_EFA_WRITE_IN_ORDER_ALIGNED_128_BYTES) This option only applies to the fi\_setopt() call. It is used to set the endpoint to use in-order RDMA write operation for each 128 bytes aligned block. Enabling the option will guarantee data inside each 128 bytes aligned block being written in order, it will also guarantee data to be delivered to the target buffer only once. If endpoint is not able to support this feature, it will return -FI\_EOPNOTSUPP for the call to fi\_setopt(). [*FI\_OPT\_EFA\_HOMOGENEOUS\_PEERS - bool*](#FI_OPT_EFA_HOMOGENEOUS_PEERS) This option only applies to the fi\_setopt() call for RDM endpoints on efa fabric. RDM endpoints on efa-direct fabric are unaffected by this option. When set to true, it indicates all peers are homogeneous, meaning they run on the same platform, use the same software versions, and share identical capabilities. It accelerates the initial communication setup as interoperability between peers is guaranteed. When set to true, the target side of a RMA operation must insert the initiator side’s address into AV before the RMA operation is kicked off, due to a current device limitation. The default value is false. [*FI\_OPT\_EFA\_USE\_UNSOLICITED\_WRITE\_RECV - bool*](#FI_OPT_EFA_USE_UNSOLICITED_WRITE_RECV) This option only applies to the fi\_setopt() call. It is used to disable unsolicited write recv for this endpoint, which can reduce the likelihood of CQ overflow. The default value is true. For efa-direct, FI\_RX\_CQ\_DATA is required when FI\_OPT\_EFA\_USE\_UNSOLICITED\_WRITE\_RECV is false, or it will return -FI\_EOPNOTSUPP for the call to fi\_setopt(). # [PROVIDER SPECIFIC OPERATION FLAGS](#PROVIDER_SPECIFIC_OPERATION_FLAGS) The EFA provider defines provider-specific operation flags that can be passed in the flags argument of data transfer calls such as fi\_writemsg(). [*FI\_EFA\_WR\_HIGH\_PPS*](#FI_EFA_WR_HIGH_PPS) This flag can be passed in the flags argument of RDMA write operations (e.g., fi\_writemsg()) to hint the device to optimize for higher message rate. # [PROVIDER SPECIFIC MR FLAGS](#PROVIDER_SPECIFIC_MR_FLAGS) The EFA provider defines provider-specific flags that can be passed in the flags argument of fi\_mr\_reg\*(). [*FI\_EFA\_MR\_RELAXED\_ORDERING*](#FI_EFA_MR_RELAXED_ORDERING) Request relaxed ordering for the memory region. By default, the EFA device issues all data transactions to a memory region (RDMA reads, RDMA writes, and receives) with strict ordering, so that the TX completion of an operation guarantees that subsequent operations to the same endpoint appear at the target after it. When FI\_EFA\_MR\_RELAXED\_ORDERING is specified, that ordering guarantee is lost. # [PROVIDER SPECIFIC OPERATION EXTENSIONS](#PROVIDER_SPECIFIC_OPERATION_EXTENSIONS) The efa provider exports extensions for operations that are not provided by the standard libfabric interface. These extensions are available via the “fi\_ext\_efa.h” header file and accessed through fi\_open\_ops, applied to either the domain or fabric fid depending on the extension. ## [Domain Operation Extension](#Domain_Operation_Extension) Domain operation extension is obtained by calling fi\_open\_ops (see [fi\_domain(3)](/man/enterprise-packages/libfabric/fi_domain.3)) ``` int fi_open_ops(struct fid *domain, const char *name, uint64_t flags, void **ops, void *context); ``` ## [FI\_EFA\_DOMAIN\_OPS](#FI_EFA_DOMAIN_OPS) Requesting FI\_EFA\_DOMAIN\_OPS in name returns ops as the pointer to the function table fi\_efa\_ops\_domain defined as follows: ``` struct fi_efa_ops_domain { int (*query_mr)(struct fid_mr *mr, struct fi_efa_mr_attr *mr_attr); }; ``` ## [query\_mr](#query_mr) This op queries an existing memory registration as input, and outputs the efa specific mr attribute which is defined as follows ``` struct fi_efa_mr_attr { uint16_t ic_id_validity; uint16_t recv_ic_id; uint16_t rdma_read_ic_id; uint16_t rdma_recv_ic_id; }; ``` [*ic\_id\_validity*](#ic_id_validity) Validity mask of interconnect id fields. Currently the following bits are supported in the mask: FI\_EFA\_MR\_ATTR\_RECV\_IC\_ID: recv\_ic\_id has a valid value. FI\_EFA\_MR\_ATTR\_RDMA\_READ\_IC\_ID: rdma\_read\_ic\_id has a valid value. FI\_EFA\_MR\_ATTR\_RDMA\_RECV\_IC\_ID: rdma\_recv\_ic\_id has a valid value. [*recv\_ic\_id*](#recv_ic_id) Physical interconnect used by the device to reach the MR for receive operation. It is only valid when ic\_id\_validity has the FI\_EFA\_MR\_ATTR\_RECV\_IC\_ID bit. [*rdma\_read\_ic\_id*](#rdma_read_ic_id) Physical interconnect used by the device to reach the MR for RDMA read operation. It is only valid when ic\_id\_validity has the FI\_EFA\_MR\_ATTR\_RDMA\_READ\_IC\_ID bit. [*rdma\_recv\_ic\_id*](#rdma_recv_ic_id) Physical interconnect used by the device to reach the MR for RDMA write receive. It is only valid when ic\_id\_validity has the FI\_EFA\_MR\_ATTR\_RDMA\_RECV\_IC\_ID bit. ## Return value **query\_mr()** returns 0 on success, or the value of errno on failure (which indicates the failure reason). ## [FI\_EFA\_GDA\_OPS](#FI_EFA_GDA_OPS) To enable GPU Direct Async (GDA), which allows the GPU to interact directly with the NIC, request FI\_EFA\_GDA\_OPS in the name parameter with efa-direct fabric. This returns ops as a pointer to the function table fi\_efa\_ops\_gda defined as follows: ``` struct fi_efa_ops_gda { int (*query_addr)(struct fid_ep *ep_fid, fi_addr_t addr, uint16_t *ahn, uint16_t *remote_qpn, uint32_t *remote_qkey); int (*query_qp_wqs)(struct fid_ep *ep_fid, struct fi_efa_wq_attr *sq_attr, struct fi_efa_wq_attr *rq_attr); int (*query_cq)(struct fid_cq *cq_fid, struct fi_efa_cq_attr *cq_attr); int (*cq_open_ext)(struct fid_domain *domain_fid, struct fi_cq_attr *attr, struct fi_efa_cq_init_attr *efa_cq_init_attr, struct fid_cq **cq_fid, void *context); uint64_t (*get_mr_lkey)(struct fid_mr *mr); int (*cntr_open_ext)(struct fid_domain *domain, struct fi_cntr_attr *attr, struct fid_cntr **cntr, void *context, struct fi_efa_comp_cntr_init_attr *efa_attr); }; ``` ## [query\_addr](#query_addr) This op queries the following address information for a given endpoint and destination address. [*ahn*](#ahn) Address handle number. [*remote\_qpn*](#remote_qpn) Remote queue pair Number. [*remote\_qkey*](#remote_qkey) qkey for the remote queue pair. ## Return value **query\_addr()** returns FI\_SUCCESS on success, or -FI\_EINVAL on failure. ## [query\_qp\_wqs](#query_qp_wqs) This op queries EFA specific Queue Pair work queue attributes for a given endpoint. It retrieves the send queue attributes in sq\_attr and receive queue attributes in rq\_attr, which is defined as follows. ``` struct fi_efa_wq_attr { uint8_t *buffer; uint32_t entry_size; uint32_t num_entries; uint32_t *doorbell; uint32_t max_batch; uint16_t caps; }; ``` [*buffer*](#buffer) Queue buffer. [*entry\_size*](#entry_size) Size of each entry in the queue. [*num\_entries*](#num_entries) Maximal number of entries in the queue. [*doorbell*](#doorbell) Queue doorbell. [*max\_batch*](#max_batch) Maximum batch size for queue submissions. [*caps*](#caps) Work queue capabilities: FI\_EFA\_WQ\_CAPS\_64\_BIT\_REQ\_ID Work queue supports posting requests with 64-bit IDs. When set, the associated completion queue is guaranteed to return 64-bit request IDs as well. ## Return value **query\_qp\_wqs()** returns 0 on success, or the value of errno on failure (which indicates the failure reason). ## [query\_cq](#query_cq) This op queries EFA specific Completion Queue attributes for a given cq. ``` struct fi_efa_cq_attr { uint8_t *buffer; uint32_t entry_size; uint32_t num_entries; }; ``` [*buffer*](#buffer~2) Completion queue buffer. [*entry\_size*](#entry_size~2) Size of each completion queue entry. [*num\_entries*](#num_entries~2) Maximal number of entries in the completion queue. ## Return value **query\_cq()** returns 0 on success, or the value of errno on failure (which indicates the failure reason). ## [cq\_open\_ext](#cq_open_ext) This op creates a completion queue with external memory provided via dmabuf. The memory can be passed by supplying the following struct. ``` struct fi_efa_cq_init_attr { uint64_t flags; struct { uint8_t *buffer; uint64_t length; uint64_t offset; uint32_t fd; } ext_mem_dmabuf; }; ``` [*flags*](#flags) A bitwise OR of the various values described below. FI\_EFA\_CQ\_INIT\_FLAGS\_EXT\_MEM\_DMABUF: create CQ with external memory provided via dmabuf. [*ext\_mem\_dmabuf*](#ext_mem_dmabuf) Structure containing information about external memory when using FI\_EFA\_CQ\_INIT\_FLAGS\_EXT\_MEM\_DMABUF flag. [*buffer*](#buffer~3) Pointer to the memory mapped in the process’s virtual address space. The field is optional, but if not provided, the use of CQ poll interfaces should be avoided. [*length*](#length) Length of the memory region to use. [*offset*](#offset) Offset within the dmabuf. [*fd*](#fd) File descriptor of the dmabuf. ## Return value **cq\_open\_ext()** returns 0 on success, or the value of errno on failure (which indicates the failure reason). ## [get\_mr\_lkey](#get_mr_lkey) Returns the local memory translation key associated with a MR. The memory registration must have completed successfully before invoking this. [*lkey*](#lkey) local memory translation key used by TX/RX buffer descriptor. ## Return value **get\_mr\_lkey()** returns lkey on success, or FI\_KEY\_NOTAVAIL if the registration has not completed. ## [cntr\_open\_ext](#cntr_open_ext) This op creates a completion counter backed by a MSI-X hardware counter. Applications must check FI\_VERSION\_GE(fi\_version(), FI\_VERSION(2, 5)) before calling cntr\_open\_ext, as older libfabric versions do not include this function pointer in the domain ops structure. When the *FI\_EFA\_COMP\_CNTR\_INIT\_WITH\_COMP\_EXTERNAL\_MEM* or *FI\_EFA\_COMP\_CNTR\_INIT\_WITH\_ERR\_EXTERNAL\_MEM* flag is set in fi\_efa\_comp\_cntr\_init\_attr.flags, the application provides its own memory for the completion or error count via the comp\_cntr\_ext\_mem or err\_cntr\_ext\_mem fields respectively. The external memory is described by a fi\_efa\_memory\_location structure which supports two modes: a virtual address (*FI\_EFA\_MEMORY\_LOCATION\_VA*), where the application supplies a direct pointer, or a DMA-BUF reference (*FI\_EFA\_MEMORY\_LOCATION\_DMABUF*), where the application supplies a file descriptor and offset into an exported DMA-BUF. When using DMA-BUF, the ptr field may also be set to provide a process-accessible mapping of the memory, which may enable more efficient counter reads. Using external memory allows the counter values to reside in application-managed buffers or in memory exported through DMA-BUF, enabling zero-copy observation of completion progress by co-located processes or devices. The default counter events type is FI\_CNTR\_EVENTS\_COMP. Counter memory supplied by the application must not be modified directly, or the resulting counter value will be non-deterministic. Counter values must only be updated using fi\_cntr* APIs. Only *FI\_WAIT\_NONE* and *FI\_WAIT\_UNSPEC* are supported as wait objects in fi\_cntr\_attr. ``` enum fi_efa_memory_location_type { FI_EFA_MEMORY_LOCATION_VA, FI_EFA_MEMORY_LOCATION_DMABUF, }; struct fi_efa_memory_location { uint8_t *ptr; struct { uint64_t offset; int32_t fd; uint32_t reserved; } dmabuf; uint8_t type; uint8_t reserved[7]; }; struct fi_efa_comp_cntr_init_attr { uint64_t comp_mask; uint32_t flags; uint32_t reserved; struct fi_efa_memory_location comp_cntr_ext_mem; struct fi_efa_memory_location err_cntr_ext_mem; }; ``` [*comp\_mask*](#comp_mask) Compatibility mask. [*flags*](#flags~2) A bitwise OR of the following values: FI\_EFA\_COMP\_CNTR\_INIT\_WITH\_COMP\_EXTERNAL\_MEM: Use application-provided memory for the completion count, as described by comp\_cntr\_ext\_mem. FI\_EFA\_COMP\_CNTR\_INIT\_WITH\_ERR\_EXTERNAL\_MEM: Use application-provided memory for the error count, as described by err\_cntr\_ext\_mem. [*comp\_cntr\_ext\_mem*](#comp_cntr_ext_mem) Memory location for the completion count when using external memory. [*err\_cntr\_ext\_mem*](#err_cntr_ext_mem) Memory location for the error count when using external memory. [*type*](#type) The type of memory location. FI\_EFA\_MEMORY\_LOCATION\_VA for a virtual address, or FI\_EFA\_MEMORY\_LOCATION\_DMABUF for a DMA-BUF reference. [*ptr*](#ptr) Virtual address pointer. Required when type is FI\_EFA\_MEMORY\_LOCATION\_VA. When type is FI\_EFA\_MEMORY\_LOCATION\_DMABUF, may optionally be set to provide a process-accessible mapping of the DMA-BUF memory. Otherwise should be NULL. [*dmabuf.fd*](#dmabuf.fd) DMA-BUF file descriptor (used when type is FI\_EFA\_MEMORY\_LOCATION\_DMABUF). [*dmabuf.offset*](#dmabuf.offset) Offset within the DMA-BUF. ## Return value **cntr\_open\_ext()** returns 0 on success, or a negative value on failure. ## [FI\_EFA\_MODIFY\_EP\_OPS](#FI_EFA_MODIFY_EP_OPS) To modify endpoint attributes at runtime, request FI\_EFA\_MODIFY\_EP\_OPS in the name parameter. This is only supported on the efa-direct fabric; on any other fabric fi\_open\_ops returns -FI\_EOPNOTSUPP. This returns ops as a pointer to the function table fi\_efa\_ops\_modify\_ep defined as follows: ``` struct fi_efa_ops_modify_ep { int (*modify_ep)(struct fid_ep *ep, struct fi_efa_ep_attr *ep_attr, uint64_t attr_mask); }; ``` ## [modify\_ep](#modify_ep) This op modifies attributes on the endpoint. The attr\_mask parameter specifies which attributes to modify. The endpoint must already be enabled. ``` enum { FI_EFA_EP_ATTR_QKEY = 1 << 0, }; struct fi_efa_ep_attr { uint32_t qkey; }; ``` [*attr\_mask*](#attr_mask) A bitwise OR of the attributes to modify. An attr\_mask of 0 is a successful no-op. Currently supported: FI\_EFA\_EP\_ATTR\_QKEY: Modify the queue key. The new value is taken from ep\_attr->qkey and must be less than 0x80000000, since 0x80000000 and above is the privileged queue key range. On success, fi\_getname reports the new queue key. The application is responsible for re-exchanging addresses with its peers after a successful call. Only supported by the efa-direct fabric. ``` A concurrent call to fi_getname may return an incorrect adresss. Applications are responsible for serializing any fi_getname calls with the modify_ep call. ``` Unsupported flags in attr\_mask will cause the call to return -FI\_EOPNOTSUPP. ## Return value **modify\_ep()** returns 0 on success, or a negative error code on failure: • -FI\_EINVAL: NULL endpoint or ep\_attr, fid is not an endpoint, endpoint not enabled, or a queue key in the privileged range. • -FI\_EOPNOTSUPP: Unsupported flags in attr\_mask, or the endpoint does not belong to an efa-direct domain. On failure, no attribute is modified. ## [Fabric Operation Extension](#Fabric_Operation_Extension) Fabric operation extension is obtained by calling fi\_open\_ops (see [fi\_fabric(3)](/man/enterprise-packages/libfabric/fi_fabric.3)) ``` int fi_open_ops(struct fid *fabric, const char *name, uint64_t flags, void **ops, void *context); ``` Requesting FI\_EFA\_FEATURE\_OPS in name returns ops as a pointer to the function table fi\_efa\_feature\_ops defined as follows: ``` struct fi_efa_feature_ops { bool (*query)(const char *feature); }; ``` Features are runtime-discoverable flags advertised by the provider, letting consumers detect the presence of a given behavior or bug fix independently of the libfabric API version (which cannot encode patch releases). The ops are exposed at fabric scope so consumers can probe features immediately after fi\_fabric(), before any domain is opened. Feature answers may differ between the efa-direct and efa (RDM) fabrics because the two exercise different code paths. Older provider builds do not expose FI\_EFA\_FEATURE\_OPS at all, so fi\_open\_ops() returns -FI\_EINVAL for the key; callers can treat that as “no features advertised”. Currently defined feature strings: *“mixed\_hmem\_iov”* (efa-direct only) The provider correctly inspects every descriptor in a multi-iov request for HMEM/iface, rather than only the first descriptor. Not currently advertised on the efa fabric because the RDM receive-copy path still dispatches on desc\[0] alone. ## [query](#query) Return true if the provider build advertises *feature*, otherwise false. Passing an unknown string (including NULL) returns false. # [Traffic Class (tclass) in EFA](#Traffic_Class_%28tclass%29_in_EFA) To prioritize the messages from a given endpoint, user can specify fi\_info->tx\_attr->tclass = FI\_TC\_LOW\_LATENCY in the fi\_endpoint() call to set the service level in rdma-core. All other tclass values will be ignored. # [RUNTIME PARAMETERS](#RUNTIME_PARAMETERS) [*FI\_EFA\_IFACE*](#FI_EFA_IFACE) A comma-delimited list of EFA device, i.e. NIC, names that should be visible to the application. This paramater can be used to include/exclude NICs to enforce process affinity based on the hardware topology. The default value is “all” which allows all available NICs to be discovered. [*FI\_EFA\_TX\_SIZE*](#FI_EFA_TX_SIZE) Maximum number of transmit operations before the provider returns -FI\_EAGAIN. For only the RDM endpoint, this parameter will cause transmit operations to be queued when this value is set higher than the default and the transmit queue is full. [*FI\_EFA\_RX\_SIZE*](#FI_EFA_RX_SIZE) Maximum number of receive operations before the provider returns -FI\_EAGAIN. # [RUNTIME PARAMETERS SPECIFIC TO RDM ENDPOINT](#RUNTIME_PARAMETERS_SPECIFIC_TO_RDM_ENDPOINT) These OFI runtime parameters apply only to the RDM endpoint. [*FI\_EFA\_RX\_WINDOW\_SIZE*](#FI_EFA_RX_WINDOW_SIZE) Maximum number of MTU-sized messages that can be in flight from any single endpoint as part of long message data transfer. [*FI\_EFA\_TX\_QUEUE\_SIZE*](#FI_EFA_TX_QUEUE_SIZE) Depth of transmit queue opened with the NIC. This may not be set to a value greater than what the NIC supports. [*FI\_EFA\_RECVWIN\_SIZE*](#FI_EFA_RECVWIN_SIZE) Size of out of order reorder buffer (in messages). Messages received out of this window will result in an error. [*FI\_EFA\_CQ\_SIZE*](#FI_EFA_CQ_SIZE) Size of any cq created, in number of entries. [*FI\_EFA\_MR\_CACHE\_ENABLE*](#FI_EFA_MR_CACHE_ENABLE) Enables using the mr cache and in-line registration instead of a bounce buffer for iov’s larger than max\_memcpy\_size. Defaults to true. When disabled, only uses a bounce buffer [*FI\_EFA\_MR\_MAX\_CACHED\_COUNT*](#FI_EFA_MR_MAX_CACHED_COUNT) Sets the maximum number of memory registrations that can be cached at any time. [*FI\_EFA\_MR\_MAX\_CACHED\_SIZE*](#FI_EFA_MR_MAX_CACHED_SIZE) Sets the maximum amount of memory that cached memory registrations can hold onto at any time. [*FI\_EFA\_MAX\_MEMCPY\_SIZE*](#FI_EFA_MAX_MEMCPY_SIZE) Threshold size switch between using memory copy into a pre-registered bounce buffer and memory registration on the user buffer. [*FI\_EFA\_MTU\_SIZE*](#FI_EFA_MTU_SIZE) Overrides the default MTU size of the device. [*FI\_EFA\_RX\_COPY\_UNEXP*](#FI_EFA_RX_COPY_UNEXP) Enables the use of a separate pool of bounce-buffers to copy unexpected messages out of the pre-posted receive buffers. [*FI\_EFA\_RX\_COPY\_OOO*](#FI_EFA_RX_COPY_OOO) Enables the use of a separate pool of bounce-buffers to copy out-of-order RTS packets out of the pre-posted receive buffers. [*FI\_EFA\_MAX\_TIMEOUT*](#FI_EFA_MAX_TIMEOUT) Maximum timeout (us) for backoff to a peer after a receiver not ready error. [*FI\_EFA\_TIMEOUT\_INTERVAL*](#FI_EFA_TIMEOUT_INTERVAL) Time interval (us) for the base timeout to use for exponential backoff to a peer after a receiver not ready error. [*FI\_EFA\_ENABLE\_SHM\_TRANSFER*](#FI_EFA_ENABLE_SHM_TRANSFER) Enable SHM provider to provide the communication across all intra-node processes. SHM transfer will be disabled in the case where ptrace protection is turned on. You can turn it off to enable shm transfer. FI\_EFA\_ENABLE\_SHM\_TRANSFER is parsed during the fi\_domain call and is related to the FI\_OPT\_SHARED\_MEMORY\_PERMITTED endpoint option. If FI\_EFA\_ENABLE\_SHM\_TRANSFER is set to true, the FI\_OPT\_SHARED\_MEMORY\_PERMITTED endpoint option overrides FI\_EFA\_ENABLE\_SHM\_TRANSFER. If FI\_EFA\_ENABLE\_SHM\_TRANSFER is set to false, but the FI\_OPT\_SHARED\_MEMORY\_PERMITTED is set to true, the FI\_OPT\_SHARED\_MEMORY\_PERMITTED setopt call will fail with -FI\_EINVAL. [*FI\_EFA\_SHM\_AV\_SIZE*](#FI_EFA_SHM_AV_SIZE) Defines the maximum number of entries in SHM provider’s address vector. [*FI\_EFA\_SHM\_MAX\_MEDIUM\_SIZE*](#FI_EFA_SHM_MAX_MEDIUM_SIZE) Defines the switch point between small/medium message and large message. The message larger than this switch point will be transferred with large message protocol. NOTE: This parameter is now deprecated. [*FI\_EFA\_INTER\_MAX\_MEDIUM\_MESSAGE\_SIZE*](#FI_EFA_INTER_MAX_MEDIUM_MESSAGE_SIZE) The maximum size for inter EFA messages to be sent by using medium message protocol. Messages which can fit in one packet will be sent as eager message. Messages whose sizes are smaller than this value will be sent using medium message protocol. Other messages will be sent using CTS based long message protocol. [*FI\_EFA\_FORK\_SAFE*](#FI_EFA_FORK_SAFE) Enable fork() support. This may have a small performance impact and should only be set when required. Applications that require to register regions backed by huge pages and also require fork support are not supported. [*FI\_EFA\_RUNT\_SIZE*](#FI_EFA_RUNT_SIZE) The maximum number of bytes that will be eagerly sent by inflight messages uses runting read message protocol (Default 307200). [*FI\_EFA\_INTER\_MIN\_READ\_MESSAGE\_SIZE*](#FI_EFA_INTER_MIN_READ_MESSAGE_SIZE) The minimum message size in bytes for inter EFA read message protocol. If instance support RDMA read, messages whose size is larger than this value will be sent by read message protocol. (Default 1048576). [*FI\_EFA\_INTER\_MIN\_READ\_WRITE\_SIZE*](#FI_EFA_INTER_MIN_READ_WRITE_SIZE) The mimimum message size for emulated inter EFA write to use read write protocol. If firmware support RDMA read, and FI\_EFA\_USE\_DEVICE\_RDMA is 1, write requests whose size is larger than this value will use the read write protocol (Default 65536). If the firmware supports RDMA write, device RDMA write will always be used. [*FI\_EFA\_USE\_DEVICE\_RDMA*](#FI_EFA_USE_DEVICE_RDMA) Specify whether to require or ignore RDMA features of the EFA device. - When set to 1/true/yes/on, all RDMA features of the EFA device are used. But if EFA device does not support RDMA and FI\_EFA\_USE\_DEVICE\_RDMA is set to 1/true/yes/on, user’s application is aborted and a warning message is printed. - When set to 0/false/no/off, libfabric will emulate all fi\_rma operations instead of offloading them to the EFA network device. Libfabric will not use device RDMA to implement send/receive operations. - If not set, RDMA operations will occur when available based on RDMA device ID/version. [*FI\_EFA\_USE\_HUGE\_PAGE*](#FI_EFA_USE_HUGE_PAGE) Specify Whether EFA provider can use huge page memory for internal buffer. Using huge page memory has a small performance advantage, but can cause system to run out of huge page memory. By default, EFA provider will use huge page unless FI\_EFA\_FORK\_SAFE is set to 1/on/true. [*FI\_EFA\_USE\_ZCPY\_RX*](#FI_EFA_USE_ZCPY_RX) Enables the use of application’s receive buffers in place of bounce-buffers when feasible. (Default: 1). Setting this environment variable to 0 can disable this feature. Explicitly setting this variable to 1 does not guarantee this feature due to other requirements. See https://github.com/ofiwg/libfabric/blob/main/prov/efa/docs/efa\_rdm\_protocol\_v4.md for details. [*FI\_EFA\_USE\_UNSOLICITED\_WRITE\_RECV*](#FI_EFA_USE_UNSOLICITED_WRITE_RECV) Use device’s unsolicited write recv functionality when it’s available. (Default: 1). Setting this environment variable to 0 can disable this feature. [*FI\_EFA\_INTERNAL\_RX\_REFILL\_THRESHOLD*](#FI_EFA_INTERNAL_RX_REFILL_THRESHOLD) The threshold that EFA provider will refill the internal rx pkt pool. (Default: 8). When the number of internal rx pkts to post is lower than this threshold, the refill will be skipped. [*FI\_EFA\_USE\_DATA\_PATH\_DIRECT*](#FI_EFA_USE_DATA_PATH_DIRECT) Use the direct data path implementation that bypasses rdma-core on data path, including the CQ polling and TX/RX submissions, when it’s available. Setting this variable as 0 will disable this feature (Default: true). [*FI\_EFA\_TRACK\_MR*](#FI_EFA_TRACK_MR) Enable tracking of memory registrations to detect if any outstanding operations still reference an MR when it is closed. When enabled, the provider will print a warning message if an MR is closed while TX or RX operations still reference it. This is useful for debugging memory registration issues. (Default: false). # [SEE ALSO](#SEE_ALSO) [fabric(7)](/man/enterprise-packages/libfabric/fabric.7), [fi\_provider(7)](/man/enterprise-packages/libfabric/fi_provider.7), [fi\_getinfo(3)](/man/enterprise-packages/libfabric/fi_getinfo.3) # [AUTHORS](#AUTHORS) OpenFabrics. 2026-09-08 Libfabric Programmer’s Manual --- Document: fi_efa(7) from libfabric 2.7.0-r3 (Chainguard OS) Source: https://github.com/ofiwg/libfabric