usage: torchrun [-h] [--nnodes NNODES] [--nproc-per-node NPROC_PER_NODE]
                [--rdzv-backend RDZV_BACKEND] [--rdzv-endpoint RDZV_ENDPOINT] [--rdzv-id RDZV_ID]
                [--rdzv-conf RDZV_CONF] [--standalone] [--max-restarts MAX_RESTARTS]
                [--monitor-interval MONITOR_INTERVAL] [--start-method {spawn,fork,forkserver}]
                [--role ROLE] [-m] [--no-python] [--run-path] [--log-dir LOG_DIR] [-r REDIRECTS]
                [-t TEE] [--local-ranks-filter LOCAL_RANKS_FILTER] [--node-rank NODE_RANK]
                [--master-addr MASTER_ADDR] [--master-port MASTER_PORT] [--local-addr LOCAL_ADDR]
                [--logs-specs LOGS_SPECS]
                training_script ...

Torch Distributed Elastic Training Launcher

positional arguments:
  training_script       Full path to the (single GPU) training program/script to be launched in
                        parallel, followed by all the arguments for the training script.
  training_script_args

options:
  -h, --help            show this help message and exit
  --nnodes NNODES       Number of nodes, or the range of nodes in form
                        <minimum_nodes>:<maximum_nodes>.
  --nproc-per-node NPROC_PER_NODE, --nproc_per_node NPROC_PER_NODE
                        Number of workers per node; supported values: [auto, cpu, gpu, int].
  --rdzv-backend RDZV_BACKEND, --rdzv_backend RDZV_BACKEND
                        Rendezvous backend.
  --rdzv-endpoint RDZV_ENDPOINT, --rdzv_endpoint RDZV_ENDPOINT
                        Rendezvous backend endpoint; usually in form <host>:<port>.
  --rdzv-id RDZV_ID, --rdzv_id RDZV_ID
                        User-defined group id.
  --rdzv-conf RDZV_CONF, --rdzv_conf RDZV_CONF
                        Additional rendezvous configuration (<key1>=<value1>,<key2>=<value2>,...).
  --standalone          Start a local standalone rendezvous backend that is represented by a C10d
                        TCP store on a free port. Useful when launching single-node, multi-worker
                        job. If specified --rdzv-backend, --rdzv-endpoint, --rdzv-id are auto-
                        assigned and any explicitly set values are ignored.
  --max-restarts MAX_RESTARTS, --max_restarts MAX_RESTARTS
                        Maximum number of worker group restarts before failing.
  --monitor-interval MONITOR_INTERVAL, --monitor_interval MONITOR_INTERVAL
                        Interval, in seconds, to monitor the state of workers.
  --start-method {spawn,fork,forkserver}, --start_method {spawn,fork,forkserver}
                        Multiprocessing start method to use when creating workers.
  --role ROLE           User-defined role for the workers.
  -m, --module          Change each process to interpret the launch script as a Python module,
                        executing with the same behavior as 'python -m'.
  --no-python, --no_python
                        Skip prepending the training script with 'python' - just execute it
                        directly. Useful when the script is not a Python script.
  --run-path, --run_path
                        Run the training script with runpy.run_path in the same interpreter.
                        Script must be provided as an abs path (e.g. /abs/path/script.py). Takes
                        precedence over --no-python.
  --log-dir LOG_DIR, --log_dir LOG_DIR
                        Base directory to use for log files (e.g. /var/log/torch/elastic). The
                        same directory is re-used for multiple runs (a unique job-level sub-
                        directory is created with rdzv_id as the prefix).
  -r REDIRECTS, --redirects REDIRECTS
                        Redirect std streams into a log file in the log directory (e.g. [-r 3]
                        redirects both stdout+stderr for all workers, [-r 0:1,1:2] redirects
                        stdout for local rank 0 and stderr for local rank 1).
  -t TEE, --tee TEE     Tee std streams into a log file and also to console (see --redirects for
                        format).
  --local-ranks-filter LOCAL_RANKS_FILTER, --local_ranks_filter LOCAL_RANKS_FILTER
                        Only show logs from specified ranks in console (e.g.
                        [--local_ranks_filter=0,1,2] will only show logs from rank 0, 1 and 2).
                        This will only apply to stdout and stderr, not tolog files saved via
                        --redirect or --tee
  --node-rank NODE_RANK, --node_rank NODE_RANK
                        Rank of the node for multi-node distributed training.
  --master-addr MASTER_ADDR, --master_addr MASTER_ADDR
                        Address of the master node (rank 0) that only used for static rendezvous.
                        It should be either the IP address or the hostname of rank 0. For single
                        node multi-proc training the --master-addr can simply be 127.0.0.1; IPv6
                        should have the pattern `[0:0:0:0:0:0:0:1]`.
  --master-port MASTER_PORT, --master_port MASTER_PORT
                        Port on the master node (rank 0) to be used for communication during
                        distributed training. It is only used for static rendezvous.
  --local-addr LOCAL_ADDR, --local_addr LOCAL_ADDR
                        Address of the local node. If specified, will use the given address for
                        connection. Else, will look up the local node address instead. Else, it
                        will be default to local machine's FQDN.
  --logs-specs LOGS_SPECS, --logs_specs LOGS_SPECS
                        torchrun.logs_specs group entrypoint name, value must be type of
                        LogsSpecs. Can be used to override custom logging behavior.