/opt/chainguard/nemo-rl/venvs/mcore/bin/nvrx-control

usage: nvrx-control [-h] --nnodes NNODES [--rdzv-backend RDZV_BACKEND]
                    --rdzv-endpoint RDZV_ENDPOINT [--rdzv-id RDZV_ID] [--rdzv-conf RDZV_CONF]
                    [--local-addr LOCAL_ADDR] [--ft-segment FT_SEGMENT]
                    [--ft-cfg-path FT_CFG_PATH]
                    [--ft-per-cycle-applog-prefix FT_PER_CYCLE_APPLOG_PREFIX]
                    [--ft-nvrx-logfile FT_NVRX_LOGFILE]
                    [--ft-enable-log-server FT_ENABLE_LOG_SERVER]
                    [--ft-log-server-port FT_LOG_SERVER_PORT]
                    [--ft-log-aggregator-count FT_LOG_AGGREGATOR_COUNT]
                    [--ft-log-leaf-max-queue-chunks FT_LOG_LEAF_MAX_QUEUE_CHUNKS]
                    [--ft-log-server-log-prefix FT_LOG_SERVER_LOG_PREFIX]
                    [--ft-log-server-graceful-shutdown-timeout FT_LOG_SERVER_GRACEFUL_SHUTDOWN_TIMEOUT]
                    [--ft-attribution-endpoint FT_ATTRIBUTION_ENDPOINT]
                    [--ft-attribution-startup-timeout FT_ATTRIBUTION_STARTUP_TIMEOUT]
                    [--ft-attribution-llm-api-key-file FT_ATTRIBUTION_LLM_API_KEY_FILE]
                    [--ft-attribution-llm-base-url FT_ATTRIBUTION_LLM_BASE_URL]
                    [--ft-attribution-llm-model FT_ATTRIBUTION_LLM_MODEL]
                    [--ft-attribution-analysis-backend {mcp,lib}]
                    [--ft-attribution-compute-timeout FT_ATTRIBUTION_COMPUTE_TIMEOUT]
                    [--ft-attribution-log-level {DEBUG,INFO,WARNING}]
                    [--ft-cycle-info-dir FT_CYCLE_INFO_DIR]
                    [--ft-cycle-info-job-id FT_CYCLE_INFO_JOB_ID]

Run the external NVRx control-node process.

options:
  -h, --help            show this help message and exit
  --nnodes NNODES       Training node range MIN[:MAX].
  --rdzv-backend, --rdzv_backend RDZV_BACKEND
                        Rendezvous backend. Currently only c10d is supported.
  --rdzv-endpoint, --rdzv_endpoint RDZV_ENDPOINT
                        Rendezvous backend endpoint; usually in form <host>:<port>.
  --rdzv-id, --rdzv_id RDZV_ID
                        User-defined group id.
  --rdzv-conf, --rdzv_conf RDZV_CONF
                        Additional rendezvous configuration (<key1>=<value1>,<key2>=<value2>,...).
  --local-addr, --local_addr LOCAL_ADDR
                        Address of the local node. If specified, will use the given address for
                        connection. Else, will look up the local node address instead. Else, it
                        will be default to local machine's FQDN.
  --ft-segment, --ft_segment FT_SEGMENT
                        Controls hot spare node behavior and segment-aware rank assignment.
                        Default: None (simple hot spare mode for H100 and non-NVSwitch systems, no
                        ClusterUUID required). When set to N: Enables segment-aware mode for
                        NVSwitch systems (DGX H200, HGX B200). Specifies minimum nodes per NVLink
                        domain (identified by GPU ClusterUUID via nvidia-smi). Domains with fewer
                        than N nodes are excluded. From valid domains, complete segments are
                        selected. min_nodes must be divisible by segment. Note: segment=None
                        (default) is suitable for H100; segment=1 is similar but requires
                        ClusterUUID.
  --ft-cfg-path, --ft-cfg_path FT_CFG_PATH
                        Path to a YAML file that contains Fault Tolerance pkg config
                        (`fault_tolerance` section). NOTE: config items from the file can be
                        overwritten by `--ft-param-*` args.
  --ft-per-cycle-applog-prefix, --ft_per_cycle_applog_prefix FT_PER_CYCLE_APPLOG_PREFIX
                        Prefix for per-cycle application log files (must be absolute path, e.g.
                        /lustre/logs/job_12345.log). Creates training worker logs per cycle:
                        /lustre/logs/job_12345_cycle0.log, job_12345_cycle1.log, etc. All
                        ranks/nodes capture logs with automatic rank prefixes (like 'srun -l').
                        Without --ft-enable-log-server: Each node writes directly to Lustre with
                        O_APPEND (safe concurrent writes). With --ft-enable-log-server
                        (recommended): All nodes stream logs to gRPC server on rank 0, which
                        becomes the single Lustre writer. gRPC mode eliminates Lustre lock
                        contention and scales to 1000+ nodes. Note: NVRx launcher logs go to
                        stdout/stderr by default unless --ft-nvrx-logfile is specified.
  --ft-nvrx-logfile, --ft_nvrx_logfile FT_NVRX_LOGFILE
                        Optional: Path for NVRx process logs (must be absolute path if specified).
                        ft_launcher routes these logs through the same infrastructure as worker
                        logs. nvrx-control also uses this path as a fallback source for gRPC
                        server diagnostic logs. Example: --ft-nvrx-logfile
                        /lustre/logs/launcher.log
  --ft-enable-log-server, --ft_enable_log_server FT_ENABLE_LOG_SERVER
                        Enable gRPC-based log funneling to a single Lustre file. When enabled with
                        --ft-per-cycle-applog-prefix, NVRx automatically: (1) Starts gRPC log
                        server(s) on the TCPStore host, (2) Configures all nodes as gRPC clients
                        to a first-level aggregator port, (3) Root server writes to Lustre (single
                        writer). With --ft-log-aggregator-count=N>1, N leaf processes plus one
                        root use ports P..P+N (see --ft-log-server-port). Use --ft-log-aggregator-
                        count=1 for a single server on P only. Benefits: No Lustre lock
                        contention, scales to 1500+ nodes. Requires: grpcio. Default: True.
  --ft-log-server-port, --ft_log_server_port FT_LOG_SERVER_PORT
                        Port for gRPC log funnel server (only used if --ft-enable-log-server is
                        enabled). Default: 50051
  --ft-log-aggregator-count, --ft_log_aggregator_count FT_LOG_AGGREGATOR_COUNT
                        Number of first-level gRPC log aggregators. 0 (default): auto - single
                        root server for up to 1536 nodes, adds leaf servers beyond that (one per
                        1536 nodes). 1: single root server (legacy). N>1: N leaf servers on ports
                        [P, P+1, ..., P+N-1] and one root on P+N (P = --ft-log-server-port). Open
                        TCP ports P through P+N on the TCP store host. Each node picks a leaf from
                        SLURM array/procid when set, else from hostname.
  --ft-log-leaf-max-queue-chunks, --ft_log_leaf_max_queue_chunks FT_LOG_LEAF_MAX_QUEUE_CHUNKS
                        Max queued log chunks per leaf (only when N>1). Default -1: auto scale
                        with fan-in, max(16384, min(1000000, per_leaf*256)).
  --ft-log-server-log-prefix, --ft_log_server_log_prefix FT_LOG_SERVER_LOG_PREFIX
                        Prefix for gRPC log server process logs (only used if --ft-enable-log-
                        server is enabled). Root and leaf process logs are written to
                        '<prefix>_root.log' and '<prefix>_leaf_<N>.log'. If not specified, derives
                        from --ft-nvrx-logfile first, then --ft-per-cycle-applog-prefix by
                        stripping a trailing '.log' and appending '_grpc'.
  --ft-log-server-graceful-shutdown-timeout, --ft_log_server_graceful_shutdown_timeout FT_LOG_SERVER_GRACEFUL_SHUTDOWN_TIMEOUT
                        Maximum seconds to wait for clients during gRPC server graceful shutdown
                        (only used if --ft-enable-log-server is enabled). When store host exits,
                        server will continue accepting logs from other ranks for up to this
                        timeout or until all clients disconnect, whichever comes first. This
                        ensures other nodes can flush their final logs. When --ft-log-aggregator-
                        count>1, each leaf uses this timeout; the root log server uses 2x so
                        leaves can finish waiting for downstream clients and drain to root first.
                        Default: 60.0
  --ft-attribution-endpoint, --ft_attribution_endpoint FT_ATTRIBUTION_ENDPOINT
                        Endpoint for the attribution service. Default: disabled. Set to localhost
                        to let the TCPStore host process manage nvrx-attrsvc. Use an explicit
                        endpoint such as http://host:port, grpc://host:port, or
                        unix:///path/to/socket for an externally managed attribution service.
  --ft-attribution-startup-timeout, --ft_attribution_startup_timeout FT_ATTRIBUTION_STARTUP_TIMEOUT
                        Seconds to wait for launcher-managed attribution service /healthz
                        readiness. Default: 20.0.
  --ft-attribution-llm-api-key-file, --ft_attribution_llm_api_key_file FT_ATTRIBUTION_LLM_API_KEY_FILE
                        Path to the LLM API key file for launcher-managed attribution service. If
                        unset, LLM_API_KEY_FILE from the launcher environment is used.
  --ft-attribution-llm-base-url, --ft_attribution_llm_base_url FT_ATTRIBUTION_LLM_BASE_URL
                        LLM base URL for launcher-managed attribution service.
  --ft-attribution-llm-model, --ft_attribution_llm_model FT_ATTRIBUTION_LLM_MODEL
                        LLM model identifier for launcher-managed attribution service.
  --ft-attribution-analysis-backend, --ft_attribution_analysis_backend {mcp,lib}
                        Analysis backend for launcher-managed attribution service: mcp or lib.
  --ft-attribution-compute-timeout, --ft_attribution_compute_timeout FT_ATTRIBUTION_COMPUTE_TIMEOUT
                        Analysis compute timeout in seconds for launcher-managed attribution
                        service.
  --ft-attribution-log-level, --ft_attribution_log_level {DEBUG,INFO,WARNING}
                        Log level for launcher-managed attribution service.
  --ft-cycle-info-dir, --ft_cycle_info_dir FT_CYCLE_INFO_DIR
                        Full path to NVRx cycle info directory (e.g. <base>/nvrx/). The rendezvous
                        host writes cycle_info.<job_id>.<attempt>.<cycle> and symlink
                        cycle_info.<job_id>.current there. Workload receives current cycle file
                        path via NVRX_CURRENT_CYCLE_INFO env.
  --ft-cycle-info-job-id, --ft_cycle_info_job_id FT_CYCLE_INFO_JOB_ID
                        Job id used by nvrx-control when writing cycle_info files. Use the same
                        effective job id compute launchers use for NVRX_CURRENT_CYCLE_INFO
                        (SLURM_ARRAY_JOB_ID when present, otherwise SLURM_JOB_ID).