/opt/chainguard/nemo-rl/venvs/mcore/bin/torchrun

usage: torchrun [-h] [--nnodes NNODES] [--nproc-per-node NPROC_PER_NODE]
                [--rdzv-backend RDZV_BACKEND] [--rdzv-endpoint RDZV_ENDPOINT] [--rdzv-id RDZV_ID]
                [--rdzv-conf RDZV_CONF] [--standalone] [--max-restarts MAX_RESTARTS]
                [--monitor-interval MONITOR_INTERVAL] [--start-method {spawn,fork,forkserver}]
                [--event-log-handler EVENT_LOG_HANDLER] [--role ROLE] [-m] [--no-python]
                [--run-path] [--log-dir LOG_DIR] [-r REDIRECTS] [-t TEE]
                [--local-ranks-filter LOCAL_RANKS_FILTER]
                [--duplicate-stdout-filters DUPLICATE_STDOUT_FILTERS]
                [--duplicate-stderr-filters DUPLICATE_STDERR_FILTERS] [--node-rank NODE_RANK]
                [--master-addr MASTER_ADDR] [--master-port MASTER_PORT] [--local-addr LOCAL_ADDR]
                [--logs-specs LOGS_SPECS] [--numa-binding {node,socket,exclusive,core-complex}]
                [--signals-to-handle SIGNALS_TO_HANDLE] [--virtual-local-rank]
                training_script ...

Torch Distributed Elastic Training Launcher

positional arguments:
  training_script       Full path to the (single GPU) training program/script to be launched in
                        parallel, followed by all the arguments for the training script.
  training_script_args

options:
  -h, --help            show this help message and exit
  --nnodes NNODES       Number of nodes, or the range of nodes in form
                        <minimum_nodes>:<maximum_nodes>.
  --nproc-per-node, --nproc_per_node NPROC_PER_NODE
                        Number of workers per node; supported values: [auto, cpu, gpu, xpu, int].
  --rdzv-backend, --rdzv_backend RDZV_BACKEND
                        Rendezvous backend.
  --rdzv-endpoint, --rdzv_endpoint RDZV_ENDPOINT
                        Rendezvous backend endpoint; usually in form <host>:<port>.
  --rdzv-id, --rdzv_id RDZV_ID
                        User-defined group id.
  --rdzv-conf, --rdzv_conf RDZV_CONF
                        Additional rendezvous configuration (<key1>=<value1>,<key2>=<value2>,...).
  --standalone          Start a local standalone rendezvous backend that is represented by a C10d
                        TCP store on a free port. Useful when launching single-node, multi-worker
                        job. If specified --rdzv-backend, --rdzv-endpoint, --rdzv-id are auto-
                        assigned and any explicitly set values are ignored.
  --max-restarts, --max_restarts MAX_RESTARTS
                        Maximum number of worker group restarts before failing.
  --monitor-interval, --monitor_interval MONITOR_INTERVAL
                        Interval, in seconds, to monitor the state of workers.
  --start-method, --start_method {spawn,fork,forkserver}
                        Multiprocessing start method to use when creating workers.
  --event-log-handler, --event_log_handler EVENT_LOG_HANDLER
                        name of a registered event logging handler (see:
                        https://docs.pytorch.org/docs/stable/elastic/events.html)
  --role ROLE           User-defined role for the workers.
  -m, --module          Change each process to interpret the launch script as a Python module,
                        executing with the same behavior as 'python -m'.
  --no-python, --no_python
                        Skip prepending the training script with 'python' - just execute it
                        directly. Useful when the script is not a Python script.
  --run-path, --run_path
                        Run the training script with runpy.run_path in the same interpreter.
                        Script must be provided as an abs path (e.g. /abs/path/script.py). Takes
                        precedence over --no-python.
  --log-dir, --log_dir LOG_DIR
                        Base directory to use for log files (e.g. /var/log/torch/elastic). The
                        same directory is reused for multiple runs (a unique job-level sub-
                        directory is created with rdzv_id as the prefix).
  -r, --redirects REDIRECTS
                        Redirect std streams into a log file in the log directory (e.g. [-r 3]
                        redirects both stdout+stderr for all workers, [-r 0:1,1:2] redirects
                        stdout for local rank 0 and stderr for local rank 1).
  -t, --tee TEE         Tee std streams into a log file and also to console (see --redirects for
                        format).
  --local-ranks-filter, --local_ranks_filter LOCAL_RANKS_FILTER
                        Only show logs from specified ranks in console (e.g.
                        [--local_ranks_filter=0,1,2] will only show logs from rank 0, 1 and 2).
                        This will only apply to stdout and stderr, not tolog files saved via
                        --redirect or --tee
  --duplicate-stdout-filters, --duplicate_stdout_filters DUPLICATE_STDOUT_FILTERS
                        Duplicates logs streamed to stdout to another specified file with a list
                        of filters (e.g. [--duplicate_stdout_filters 'apple,orange'] will
                        duplicate log lines matching 'apple' OR 'orange'. An empty filters list
                        won't duplicate any lines. Use double comma to escape a comma)
  --duplicate-stderr-filters, --duplicate_stderr_filters DUPLICATE_STDERR_FILTERS
                        Duplicates logs streamed to stderr to another specified file with a list
                        of filters (e.g. [--duplicate_stdout_filters 'apple,orange'] will
                        duplicate log lines matching 'apple' OR 'orange'. An empty filters list
                        won't duplicate any lines. Use double comma to escape a comma)
  --node-rank, --node_rank NODE_RANK
                        Rank of the node for multi-node distributed training.
  --master-addr, --master_addr MASTER_ADDR
                        Address of the master node (rank 0) that only used for static rendezvous.
                        It should be either the IP address or the hostname of rank 0. For single
                        node multi-proc training the --master-addr can simply be 127.0.0.1; IPv6
                        should have the pattern `[0:0:0:0:0:0:0:1]`.
  --master-port, --master_port MASTER_PORT
                        Port on the master node (rank 0) to be used for communication during
                        distributed training. It is only used for static rendezvous.
  --local-addr, --local_addr LOCAL_ADDR
                        Address of the local node. If specified, will use the given address for
                        connection. Else, will look up the local node address instead. Else, it
                        will be default to local machine's FQDN.
  --logs-specs, --logs_specs LOGS_SPECS
                        torchrun.logs_specs group entrypoint name, value must be type of
                        LogsSpecs. Can be used to override custom logging behavior.
  --numa-binding, --numa_binding {node,socket,exclusive,core-complex}
                        Bind worker processes to CPUs near their assigned GPUs for better
                        performance. See torch/numa/binding.py for available modes and details.
  --signals-to-handle, --signals_to_handle SIGNALS_TO_HANDLE
                        Comma-separated list of signals to handle and forward to subprocesses.
                        Default: SIGTERM,SIGINT,SIGHUP,SIGQUIT. Common additional signals:
                        SIGUSR1,SIGUSR2 (used in SLURM environments).
  --virtual-local-rank, --virtual_local_rank
                        Enable virtual local rank mode for workers. When enabled, LOCAL_RANK is
                        set to 0 for all workers and CUDA_VISIBLE_DEVICES is adjusted so each
                        worker accesses its assigned GPU at device index 0.