/usr/bin/ucx_perftest: unrecognized option '--help' Usage: ucx_perftest [ server-address ] [ options ] Supported server-address: with default socket-based test setup with default socket-based test setup lid: with MAD-based test setup (-K) guid: with MAD-based test setup (-K) Common options: -t test to run: am_lat - UCT active message latency put_lat - UCT put latency add_lat - UCT atomic add latency get - UCT get latency / bandwidth / message rate fadd - UCT atomic fetch-and-add latency / rate swap - UCT atomic swap latency / rate cswap - UCT atomic compare-and-swap latency / rate am_bw - UCT active message bandwidth / message rate put_bw - UCT put bandwidth / message rate get_bw - UCT get bandwidth / message rate add_mr - UCT atomic add message rate tag_lat - UCP tag match latency tag_bw - UCP tag match bandwidth tag_sync_lat - UCP tag sync match latency tag_sync_bw - UCP tag sync match bandwidth ucp_put_lat - UCP put latency ucp_put_bw - UCP put bandwidth ucp_get - UCP get latency / bandwidth / message rate ucp_add - UCP atomic add bandwidth / message rate ucp_fadd - UCP atomic fetch-and-add latency / bandwidth / rate ucp_swap - UCP atomic swap latency / bandwidth / rate ucp_cswap - UCP atomic compare-and-swap latency / bandwidth / rate stream_bw - UCP stream bandwidth stream_lat - UCP stream latency ucp_am_lat - UCP am latency ucp_am_bw - UCP am bandwidth / message rate -a [,] Accelerator device type and device id to use for running the test. device id is optional, it corresponds to the index of the device in the list of available devices -L device cooperation level for gdaki (thread) thread - thread level warp - warp level block - block level grid - grid level -F flow control window size for device tests (4). This option defines the number of iterations per which a single flow control request is sent. -Y channel selection mode for device tests (single) single - use a single fixed channel (channel 0, default) random[:] - use random channel per operation with optional random seed per-thread - use global thread ID -s list of scatter-gather sizes for single message (8) for example: "-s 16,48,8192,8192,14" compact form example: "-s 1024:16 expands to [1024, ..., 1024] with 16 elements -m [,] memory type of message for sender and receiver (host) host - System memory cuda - NVIDIA GPU memory cuda-managed - NVIDIA GPU managed/unified memory rocm - AMD/ROCm GPU memory rocm-managed - AMD/ROCm GPU managed memory rdma - RDMA device memory ze-host - Intel/Ze USM host memory ze-device - Intel/Ze GPU memory ze-managed - Intel/Ze GPU managed memory gaudi - HabanaLabs Gaudi memory -n number of iterations to run (1000000) -w number of warm-up iterations (10000) -c set affinity to this CPU list (separated by comma) (off) -O maximal number of uncompleted outstanding sends (0) -i distance between consecutive scatter-gather entries (0) -l use loopback connection in this case, the process will communicate with itself, so passing server hostname is not allowed -o do not progress the responder in one-sided tests -B register memory with NONBLOCK flag -b read and execute tests from a batch file: every line in the file is a test to run. The first word is a user-defined test name, followed by command-line arguments, for example: test_tag_bandwidth_64k -t tag_bw -s 65536 -n 10000 -R percentile rank of the percentile data in latency tests (50.0) -p TCP port to use for data exchange (13337) -6 Use IPv6 address for in data exchange -K use MAD for test setup and synchronization -h show this help message Output format: -N use numeric formatting (thousands separator) -f print only final numbers -v print CSV-formatted output -X print extra information about the operation -q do not print error messages UCT only: -d device to use for testing -x transport to use for testing -D data layout for sender side: short - short messages (default, cannot be used for get) shortiov - short io-vector messages (only for active messages) bcopy - copy-out (cannot be used for atomics) zcopy - zero-copy (cannot be used for atomics) -W flow control window size, for active messages (127) -H active message header size (8), included in message size -A asynchronous progress mode (thread_spinlock) thread_spinlock - separate progress thread with spin locking thread_mutex - separate progress thread with mutex locking signal - signal-based timer UCP only: -T [:] number of threads in the test (1). blocks is optional, it corresponds to the number of device blocks if blocks is specified, then threads value corresponds to the number of device threads in each block -M thread support level for progress engine (single) single - only the master thread can access serialized - one thread can access at a time multi - multiple threads can access -D [,] data layout for sender and receiver side (contig) contig - Continuous datatype iov - Scatter-gather list -C use wild-card tag for tag tests -U force unexpected flow by using tag probe -r receive mode for stream tests (recv) recv : Use ucp_stream_recv_nb recv_data : Use ucp_stream_recv_data_nb -I create context with wakeup feature enabled -e create endpoints with error handling support -E wait mode for tests poll : repeatedly call worker_progress sleep : go to sleep after posting requests -H active message header size (0), not included in message size -y do additional memcopy to the user memory in active message receive handler -z pass pre-registered memory handle -g [:], --daemon-local [:] IP address and port of the local daemon to offload UCP operations to Port is optional, by default daemon port is (1338) -G [:], --daemon-remote [:] IP address and port of the remote daemon to offload UCP operations to Port is optional, by default daemon port is (1338) NOTE: When running UCP tests, transport and device should be specified by environment variables: UCX_TLS and UCX_[SELF|SHM|NET]_DEVICES.