usage: torchrun [-h] [--nnodes NNODES] [--nproc-per-node NPROC_PER_NODE] [--rdzv-backend RDZV_BACKEND] [--rdzv-endpoint RDZV_ENDPOINT] [--rdzv-id RDZV_ID] [--rdzv-conf RDZV_CONF] [--standalone] [--max-restarts MAX_RESTARTS] [--monitor-interval MONITOR_INTERVAL] [--start-method {spawn,fork,forkserver}] [--event-log-handler EVENT_LOG_HANDLER] [--role ROLE] [-m] [--no-python] [--run-path] [--log-dir LOG_DIR] [-r REDIRECTS] [-t TEE] [--local-ranks-filter LOCAL_RANKS_FILTER] [--duplicate-stdout-filters DUPLICATE_STDOUT_FILTERS] [--duplicate-stderr-filters DUPLICATE_STDERR_FILTERS] [--node-rank NODE_RANK] [--master-addr MASTER_ADDR] [--master-port MASTER_PORT] [--local-addr LOCAL_ADDR] [--logs-specs LOGS_SPECS] [--numa-binding {node,socket,exclusive,core-complex}] [--signals-to-handle SIGNALS_TO_HANDLE] [--virtual-local-rank] training_script ... Torch Distributed Elastic Training Launcher positional arguments: training_script Full path to the (single GPU) training program/script to be launched in parallel, followed by all the arguments for the training script. training_script_args options: -h, --help show this help message and exit --nnodes NNODES Number of nodes, or the range of nodes in form :. --nproc-per-node NPROC_PER_NODE, --nproc_per_node NPROC_PER_NODE Number of workers per node; supported values: [auto, cpu, gpu, xpu, int]. --rdzv-backend RDZV_BACKEND, --rdzv_backend RDZV_BACKEND Rendezvous backend. --rdzv-endpoint RDZV_ENDPOINT, --rdzv_endpoint RDZV_ENDPOINT Rendezvous backend endpoint; usually in form :. --rdzv-id RDZV_ID, --rdzv_id RDZV_ID User-defined group id. --rdzv-conf RDZV_CONF, --rdzv_conf RDZV_CONF Additional rendezvous configuration (=,=,...). --standalone Start a local standalone rendezvous backend that is represented by a C10d TCP store on a free port. Useful when launching single-node, multi-worker job. If specified --rdzv-backend, --rdzv-endpoint, --rdzv-id are auto- assigned and any explicitly set values are ignored. --max-restarts MAX_RESTARTS, --max_restarts MAX_RESTARTS Maximum number of worker group restarts before failing. --monitor-interval MONITOR_INTERVAL, --monitor_interval MONITOR_INTERVAL Interval, in seconds, to monitor the state of workers. --start-method {spawn,fork,forkserver}, --start_method {spawn,fork,forkserver} Multiprocessing start method to use when creating workers. --event-log-handler EVENT_LOG_HANDLER, --event_log_handler EVENT_LOG_HANDLER name of a registered event logging handler (see: https://docs.pytorch.org/docs/stable/elastic/events.html) --role ROLE User-defined role for the workers. -m, --module Change each process to interpret the launch script as a Python module, executing with the same behavior as 'python -m'. --no-python, --no_python Skip prepending the training script with 'python' - just execute it directly. Useful when the script is not a Python script. --run-path, --run_path Run the training script with runpy.run_path in the same interpreter. Script must be provided as an abs path (e.g. /abs/path/script.py). Takes precedence over --no-python. --log-dir LOG_DIR, --log_dir LOG_DIR Base directory to use for log files (e.g. /var/log/torch/elastic). The same directory is reused for multiple runs (a unique job-level sub- directory is created with rdzv_id as the prefix). -r REDIRECTS, --redirects REDIRECTS Redirect std streams into a log file in the log directory (e.g. [-r 3] redirects both stdout+stderr for all workers, [-r 0:1,1:2] redirects stdout for local rank 0 and stderr for local rank 1). -t TEE, --tee TEE Tee std streams into a log file and also to console (see --redirects for format). --local-ranks-filter LOCAL_RANKS_FILTER, --local_ranks_filter LOCAL_RANKS_FILTER Only show logs from specified ranks in console (e.g. [--local_ranks_filter=0,1,2] will only show logs from rank 0, 1 and 2). This will only apply to stdout and stderr, not tolog files saved via --redirect or --tee --duplicate-stdout-filters DUPLICATE_STDOUT_FILTERS, --duplicate_stdout_filters DUPLICATE_STDOUT_FILTERS Duplicates logs streamed to stdout to another specified file with a list of filters (e.g. [--duplicate_stdout_filters 'apple,orange'] will duplicate log lines matching 'apple' OR 'orange'. An empty filters list won't duplicate any lines. Use double comma to escape a comma) --duplicate-stderr-filters DUPLICATE_STDERR_FILTERS, --duplicate_stderr_filters DUPLICATE_STDERR_FILTERS Duplicates logs streamed to stderr to another specified file with a list of filters (e.g. [--duplicate_stdout_filters 'apple,orange'] will duplicate log lines matching 'apple' OR 'orange'. An empty filters list won't duplicate any lines. Use double comma to escape a comma) --node-rank NODE_RANK, --node_rank NODE_RANK Rank of the node for multi-node distributed training. --master-addr MASTER_ADDR, --master_addr MASTER_ADDR Address of the master node (rank 0) that only used for static rendezvous. It should be either the IP address or the hostname of rank 0. For single node multi-proc training the --master-addr can simply be 127.0.0.1; IPv6 should have the pattern `[0:0:0:0:0:0:0:1]`. --master-port MASTER_PORT, --master_port MASTER_PORT Port on the master node (rank 0) to be used for communication during distributed training. It is only used for static rendezvous. --local-addr LOCAL_ADDR, --local_addr LOCAL_ADDR Address of the local node. If specified, will use the given address for connection. Else, will look up the local node address instead. Else, it will be default to local machine's FQDN. --logs-specs LOGS_SPECS, --logs_specs LOGS_SPECS torchrun.logs_specs group entrypoint name, value must be type of LogsSpecs. Can be used to override custom logging behavior. --numa-binding {node,socket,exclusive,core-complex}, --numa_binding {node,socket,exclusive,core-complex} If provided, we will affinitize the worker processes based on NUMA nodes for better performance. (E.g., preferring to allocate memory locally and run on CPUs on the same NUMA node.) NOTE: This is currently only supported for GPUs, and we assume that the LOCAL_RANK process corresponds to the GPU with index LOCAL_RANK. If this is not accurate for your workload, this feature may be a pessimization. Available options are: - node: Processes are bound to cpu cores within a NUMA node. This is a good starting point, but other options may perform even slightly better in some cases. - socket: Processes are bound to cpu cores within a socket. - exclusive: Processes are bound to exclusive sets of cpu cores within a NUMA node. - core-complex: Processes are bound to cpu cores in a core-complex. NOTE: The core-complex option might not achieve optimal performance on architectures featuring a single L3 cache per socket. --signals-to-handle SIGNALS_TO_HANDLE, --signals_to_handle SIGNALS_TO_HANDLE Comma-separated list of signals to handle and forward to subprocesses. Default: SIGTERM,SIGINT,SIGHUP,SIGQUIT. Common additional signals: SIGUSR1,SIGUSR2 (used in SLURM environments). --virtual-local-rank, --virtual_local_rank Enable virtual local rank mode for workers. When enabled, LOCAL_RANK is set to 0 for all workers and CUDA_VISIBLE_DEVICES is adjusted so each worker accesses its assigned GPU at device index 0.