usage: torchrun [-h] [--nnodes NNODES] [--nproc-per-node NPROC_PER_NODE] [--rdzv-backend RDZV_BACKEND] [--rdzv-endpoint RDZV_ENDPOINT] [--rdzv-id RDZV_ID] [--rdzv-conf RDZV_CONF] [--standalone] [--max-restarts MAX_RESTARTS] [--monitor-interval MONITOR_INTERVAL] [--start-method {spawn,fork,forkserver}] [--event-log-handler EVENT_LOG_HANDLER] [--role ROLE] [-m] [--no-python] [--run-path] [--log-dir LOG_DIR] [-r REDIRECTS] [-t TEE] [--local-ranks-filter LOCAL_RANKS_FILTER] [--duplicate-stdout-filters DUPLICATE_STDOUT_FILTERS] [--duplicate-stderr-filters DUPLICATE_STDERR_FILTERS] [--node-rank NODE_RANK] [--master-addr MASTER_ADDR] [--master-port MASTER_PORT] [--local-addr LOCAL_ADDR] [--logs-specs LOGS_SPECS] [--numa-binding {node,socket,exclusive,core-complex}] [--signals-to-handle SIGNALS_TO_HANDLE] [--shutdown-timeout SHUTDOWN_TIMEOUT] [--virtual-local-rank] training_script ... Torch Distributed Elastic Training Launcher positional arguments: training_script Full path to the (single GPU) training program/script to be launched in parallel, followed by all the arguments for the training script. training_script_args options: -h, --help show this help message and exit --nnodes NNODES Number of nodes, or the range of nodes in form :. --nproc-per-node NPROC_PER_NODE, --nproc_per_node NPROC_PER_NODE Number of workers per node; supported values: [auto, cpu, gpu, xpu, int]. --rdzv-backend RDZV_BACKEND, --rdzv_backend RDZV_BACKEND Rendezvous backend. --rdzv-endpoint RDZV_ENDPOINT, --rdzv_endpoint RDZV_ENDPOINT Rendezvous backend endpoint; usually in form :. --rdzv-id RDZV_ID, --rdzv_id RDZV_ID User-defined group id. --rdzv-conf RDZV_CONF, --rdzv_conf RDZV_CONF Additional rendezvous configuration (=,=,...). --standalone Start a local standalone rendezvous backend that is represented by a C10d TCP store on a free port. Useful when launching single-node, multi-worker job. If specified --rdzv-backend, --rdzv-endpoint, --rdzv-id are auto- assigned and any explicitly set values are ignored. --max-restarts MAX_RESTARTS, --max_restarts MAX_RESTARTS Maximum number of worker group restarts before failing. --monitor-interval MONITOR_INTERVAL, --monitor_interval MONITOR_INTERVAL Interval, in seconds, to monitor the state of workers. --start-method {spawn,fork,forkserver}, --start_method {spawn,fork,forkserver} Multiprocessing start method to use when creating workers. --event-log-handler EVENT_LOG_HANDLER, --event_log_handler EVENT_LOG_HANDLER name of a registered event logging handler (see: https://docs.pytorch.org/docs/stable/elastic/events.html) --role ROLE User-defined role for the workers. -m, --module Change each process to interpret the launch script as a Python module, executing with the same behavior as 'python -m'. --no-python, --no_python Skip prepending the training script with 'python' - just execute it directly. Useful when the script is not a Python script. --run-path, --run_path Run the training script with runpy.run_path in the same interpreter. Script must be provided as an abs path (e.g. /abs/path/script.py). Takes precedence over --no-python. --log-dir LOG_DIR, --log_dir LOG_DIR Base directory to use for log files (e.g. /var/log/torch/elastic). The same directory is reused for multiple runs (a unique job-level sub- directory is created with rdzv_id as the prefix). -r REDIRECTS, --redirects REDIRECTS Redirect std streams into a log file in the log directory (e.g. [-r 3] redirects both stdout+stderr for all workers, [-r 0:1,1:2] redirects stdout for local rank 0 and stderr for local rank 1). -t TEE, --tee TEE Tee std streams into a log file and also to console (see --redirects for format). --local-ranks-filter LOCAL_RANKS_FILTER, --local_ranks_filter LOCAL_RANKS_FILTER Only show logs from specified ranks in console (e.g. [--local_ranks_filter=0,1,2] will only show logs from rank 0, 1 and 2). This will only apply to stdout and stderr, not tolog files saved via --redirect or --tee --duplicate-stdout-filters DUPLICATE_STDOUT_FILTERS, --duplicate_stdout_filters DUPLICATE_STDOUT_FILTERS Duplicates logs streamed to stdout to another specified file with a list of filters (e.g. [--duplicate_stdout_filters 'apple,orange'] will duplicate log lines matching 'apple' OR 'orange'. An empty filters list won't duplicate any lines. Use double comma to escape a comma) --duplicate-stderr-filters DUPLICATE_STDERR_FILTERS, --duplicate_stderr_filters DUPLICATE_STDERR_FILTERS Duplicates logs streamed to stderr to another specified file with a list of filters (e.g. [--duplicate_stdout_filters 'apple,orange'] will duplicate log lines matching 'apple' OR 'orange'. An empty filters list won't duplicate any lines. Use double comma to escape a comma) --node-rank NODE_RANK, --node_rank NODE_RANK Rank of the node for multi-node distributed training. --master-addr MASTER_ADDR, --master_addr MASTER_ADDR Address of the master node (rank 0) that only used for static rendezvous. It should be either the IP address or the hostname of rank 0. For single node multi-proc training the --master-addr can simply be 127.0.0.1; IPv6 should have the pattern `[0:0:0:0:0:0:0:1]`. --master-port MASTER_PORT, --master_port MASTER_PORT Port on the master node (rank 0) to be used for communication during distributed training. It is only used for static rendezvous. Defaults to 29500. --local-addr LOCAL_ADDR, --local_addr LOCAL_ADDR Address of the local node. If specified, will use the given address for connection. Else, will look up the local node address instead. Else, it will be default to local machine's FQDN. --logs-specs LOGS_SPECS, --logs_specs LOGS_SPECS torchrun.logs_specs group entrypoint name, value must be type of LogsSpecs. Can be used to override custom logging behavior. --numa-binding {node,socket,exclusive,core-complex}, --numa_binding {node,socket,exclusive,core-complex} Bind worker processes to CPUs near their assigned GPUs for better performance. See torch/numa/binding.py for available modes and details. --signals-to-handle SIGNALS_TO_HANDLE, --signals_to_handle SIGNALS_TO_HANDLE Comma-separated list of signals to handle and forward to subprocesses. Default: SIGTERM,SIGINT,SIGHUP,SIGQUIT. Common additional signals: SIGUSR1,SIGUSR2 (used in SLURM environments). --shutdown-timeout SHUTDOWN_TIMEOUT, --shutdown_timeout SHUTDOWN_TIMEOUT Time in seconds to wait for graceful shutdown of worker processes before sending SIGKILL. If not specified, uses TORCH_ELASTIC_SHUTDOWN_TIMEOUT environment variable or defaults to 30 seconds. --virtual-local-rank, --virtual_local_rank Enable virtual local rank mode for workers. When enabled, LOCAL_RANK is set to 0 for all workers and CUDA_VISIBLE_DEVICES is adjusted so each worker accesses its assigned GPU at device index 0.