Usage of /usr/bin/pd-sidecar:
--cert-path string Directory with tls.crt and tls.key for secure serving. Empty generates a self-signed certificate, which is only suitable for testing.
--configuration --configuration={port: 8085, model-server-port: 8203}. Inline configuration and file configuration are mutually exclusive. Sidecar configuration in YAML provided as inline specification. Example --configuration={port: 8085, model-server-port: 8203}. Inline configuration and file configuration are mutually exclusive.
--configuration-file --configuration-file=/etc/config/sidecar-config.yaml Path to file which contains sidecar configuration in YAML. Example --configuration-file=/etc/config/sidecar-config.yaml. Inline configuration and file configuration are mutually exclusive.
--data-parallel-size int the model server's data-parallel size (default 1)
--decode-chunk-size int enables chunked decode mode when > 0; value is the token budget per chunk. For best performance should be a multiple of the block size.
--ec-connector string the EC protocol between encoder and prefiller (for EPD mode). Supported: ec-example, ec-nixl. Leave empty to skip encoder stage.
--enable-p2p-pull declare the OffloadingConnector P2P tier available for cached-prefix pulls when the PD connector is NIXL, i.e. engines run MultiConnector(NixlConnector + OffloadingConnector). Rejected with any other --kv-connector; offloading provides the tier natively without this flag.
--enable-prefiller-sampling if true, the target prefill instance will be selected randomly from among the provided prefill host values
--enable-ssrf-protection enable SSRF protection using InferencePool allowlisting
--enable-tls strings stages to enable TLS for. Supported: prefiller, decoder, encoder. Can be specified multiple times or as comma-separated values.
--inference-pool string InferencePool in namespace/name or name format (e.g., default/my-pool or my-pool). A single name implies the 'default' namespace. Can also use INFERENCE_POOL env var.
--kv-connector string the KV protocol between prefiller and decoder. Supported: nixlv2, shared-storage, sglang, mooncake, offloading (default "nixlv2")
--max-idle-conns-per-host int max idle keep-alive connections per host for reverse proxy transports; set to at least the expected concurrency (default 1024)
--metrics-cert-dir string Directory with tls.crt and tls.key for the metrics endpoint. Empty serves metrics over plain HTTP. Independent of --secure-serving and --cert-path, which apply to the serving listener.
--metrics-port int Port for the Prometheus /metrics endpoint (exposes the moriio_dns_* and llm_d_disagg_sidecar_* counters). 0 (the default) disables it. Takes precedence over the MORIIO_METRICS_ADDR env var.
--model-server-port string the port the model server is listening on (default 8200) (default "8200")
--mooncake-bootstrap-port int the port used to query the Mooncake bootstrap endpoint on prefill pods (only used with --kv-connector=mooncake) (default 8998)
--moriio-decode-handshake-port int Base MoRI-IO handshake port on the decode pod. (default 6301)
--moriio-decode-hosts strings Wide-EP: comma-separated decode-side pod hosts, emitted as the prefill request's remote_hosts. Kubernetes DNS names (e.g., 'pod-name.namespace.svc.cluster.local') are resolved to IPs at startup; literal IPs are used as-is. Pair with --moriio-dp-size-local.
--moriio-decode-notify-port int Base MoRI-IO notify port on the decode pod. (default 61005)
--moriio-dp-size int Data-parallel world size, emitted as kv_transfer_params[remote_dp_size] on both requests. Set to the engine DP size for Wide-EP (TP=1, DP>1); default 1 leaves the wire unchanged. (default 1)
--moriio-dp-size-local int Wide-EP: per-pod DP size used to map a global DP rank to a pod index. Must satisfy --moriio-dp-size = dp-size-local * len(hosts).
--moriio-local-pod-ip string Decode pod's routable address, used as the prefill request's remote_host. A Kubernetes DNS name (e.g., 'pod-name.namespace.svc.cluster.local') is resolved to an IP at startup; a literal IP is used as-is. Defaults to the POD_IP env var. Required with --moriio-write-mode.
--moriio-parallel-decode-wait-timeout duration Backstop timeout for parallel dispatch: how long decode may wait on the prefill outcome/KV before being cancelled so the request fails instead of hanging. Only used with --moriio-parallel-dispatch. (default 30s)
--moriio-parallel-dispatch Fire prefill and decode concurrently. OFF by default (serial dispatch via handleNIXLV2 is the SLA-safe path); opt-in only. Requires --moriio-write-mode.
--moriio-prefill-handshake-port int Prefill pod's base MoRI-IO handshake port, used to build the decode request in parallel-dispatch mode. (default 6301)
--moriio-prefill-notify-port int Prefill pod's base MoRI-IO notify port. (default 61005)
--moriio-remote-hosts strings Wide-EP: comma-separated remote (prefill-side) pod hosts for per-DP-rank fan-out. Kubernetes DNS names (e.g., 'pod-name.namespace.svc.cluster.local') are resolved to IPs at startup; literal IPs are used as-is. Pair with --moriio-dp-size-local.
--moriio-tp-size int Tensor-parallel size of the engines, echoed into kv_transfer_params[tp_size]. (default 1)
--moriio-write-mode Enable MoRI-IO WRITE-mode passthrough in kv_transfer_params. Requires --kv-connector=nixlv2.
--p2p-connector-port int the prefiller's OffloadingConnector P2P tier listening port, injected as remote_port on the decode request; with --data-parallel-size > 1 this is the rank-0 port and rank r uses port+r (used with --kv-connector=offloading or --enable-p2p-pull) (default 7777)
--pool-group inference.networking.k8s.io Kubernetes resource group of the InferencePool this sidecar is associated with. Only inference.networking.k8s.io is currently supported. (default "inference.networking.k8s.io")
--port string the port the sidecar is listening on (default "8000")
--prefill-max-retries int max retry attempts when a prefill request fails with a 5xx error; 0 means no retries (default)
--prefill-retry-backoff duration delay between prefill retry attempts (default 200ms)
--secure-serving Serve the listener over TLS. (default true)
--tls-cipher-suites strings Comma-separated list of TLS cipher suites for secure serving (Go crypto/tls names, e.g., TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256). Empty uses the crypto/tls default. Only effective for TLS 1.2 and below; TLS 1.3 cipher suites are not configurable.
--tls-insecure-skip-verify strings stages to skip TLS verification for. Supported: prefiller, decoder, encoder. Can be specified multiple times or as comma-separated values.
--tls-min-version string Minimum TLS version for secure serving (e.g., VersionTLS12, VersionTLS13). Empty uses VersionTLS12.
--tracing Enable OpenTelemetry tracing
--zap-devel Development Mode defaults(encoder=consoleEncoder,logLevel=Debug,stackTraceLevel=Warn). Production Mode defaults(encoder=jsonEncoder,logLevel=Info,stackTraceLevel=Error)
--zap-encoder encoder Zap log encoding (one of 'json' or 'console')
--zap-log-level level Zap Level to configure the verbosity of logging. Can be one of 'debug', 'info', 'error', 'panic' or any integer value > 0 which corresponds to custom debug levels of increasing verbosity
--zap-stacktrace-level level Zap Level at and above which stacktraces are captured (one of 'info', 'error', 'panic').
--zap-time-encoding time-encoding Zap time encoding (one of 'epoch', 'millis', 'nano', 'iso8601', 'rfc3339' or 'rfc3339nano'). Defaults to 'epoch'.
Options (45)
| Option | Description | Source |
|---|---|---|
--cert-path string | Directory with tls.crt and tls.key for secure serving. | help |
--configuration --configuration={port: | Sidecar configuration in YAML provided as inline specification. | help |
--configuration-file | Path to file which contains sidecar configuration in YAML. | help |
--data-parallel-size int | the model server's data-parallel size (default 1) | help |
--decode-chunk-size int | enables chunked decode mode when > 0; | help |
--ec-connector string | the EC protocol between encoder and prefiller (for EPD mode). | help |
--enable-p2p-pull | declare the OffloadingConnector P2P tier available for cached-prefix pulls when the PD connector is NIXL, i.e. engines run MultiConnector(NixlConnector + OffloadingConnector). | help |
--enable-prefiller-sampling | if true, the target prefill instance will be selected randomly from among the provided prefill host values | help |
--enable-ssrf-protection | enable SSRF protection using InferencePool allowlisting | help |
--enable-tls strings | stages to enable TLS for. | help |
--inference-pool string | InferencePool in namespace/name or name format (e.g., default/my-pool or my-pool). | help |
--kv-connector string | the KV protocol between prefiller and decoder. | help |
--max-idle-conns-per-host int | max idle keep-alive connections per host for reverse proxy transports; | help |
--metrics-cert-dir string | Directory with tls.crt and tls.key for the metrics endpoint. | help |
--metrics-port int | Port for the Prometheus /metrics endpoint (exposes the moriio_dns_* and llm_d_disagg_sidecar_* counters). | help |
--model-server-port string | the port the model server is listening on (default 8200) (default "8200") | help |
--mooncake-bootstrap-port int | the port used to query the Mooncake bootstrap endpoint on prefill pods (only used with --kv-connector=mooncake) (default 8998) | help |
--moriio-decode-handshake-port int | Base MoRI-IO handshake port on the decode pod. | help |
--moriio-decode-hosts strings | Wide-EP: comma-separated decode-side pod hosts, emitted as the prefill request's remote_hosts. | help |
--moriio-decode-notify-port int | Base MoRI-IO notify port on the decode pod. | help |
--moriio-dp-size int | Data-parallel world size, emitted as kv_transfer_params[remote_dp_size] on both requests. | help |
--moriio-dp-size-local int | Wide-EP: per-pod DP size used to map a global DP rank to a pod index. | help |
--moriio-local-pod-ip string | Decode pod's routable address, used as the prefill request's remote_host. | help |
--moriio-parallel-decode-wait-timeout duration | Backstop timeout for parallel dispatch: how long decode may wait on the prefill outcome/KV before being cancelled so the request fails instead of hanging. | help |
--moriio-parallel-dispatch | Fire prefill and decode concurrently. | help |
--moriio-prefill-handshake-port int | Prefill pod's base MoRI-IO handshake port, used to build the decode request in parallel-dispatch mode. | help |
--moriio-prefill-notify-port int | Prefill pod's base MoRI-IO notify port. | help |
--moriio-remote-hosts strings | Wide-EP: comma-separated remote (prefill-side) pod hosts for per-DP-rank fan-out. | help |
--moriio-tp-size int | Tensor-parallel size of the engines, echoed into kv_transfer_params[tp_size]. | help |
--moriio-write-mode | Enable MoRI-IO WRITE-mode passthrough in kv_transfer_params. | help |
--p2p-connector-port int | the prefiller's OffloadingConnector P2P tier listening port, injected as remote_port on the decode request; | help |
--pool-group inference.networking.k8s.io | Kubernetes resource group of the InferencePool this sidecar is associated with. | help |
--port string | the port the sidecar is listening on (default "8000") | help |
--prefill-max-retries int | max retry attempts when a prefill request fails with a 5xx error; | help |
--prefill-retry-backoff duration | delay between prefill retry attempts (default 200ms) | help |
--secure-serving | Serve the listener over TLS. | help |
--tls-cipher-suites strings | Comma-separated list of TLS cipher suites for secure serving (Go crypto/tls names, e.g., TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256). | help |
--tls-insecure-skip-verify strings | stages to skip TLS verification for. | help |
--tls-min-version string | Minimum TLS version for secure serving (e.g., VersionTLS12, VersionTLS13). | help |
--tracing | Enable OpenTelemetry tracing | help |
--zap-devel | Development Mode defaults(encoder=consoleEncoder,logLevel=Debug,stackTraceLevel=Warn). | help |
--zap-encoder encoder | Zap log encoding (one of 'json' or 'console') | help |
--zap-log-level level | Zap Level to configure the verbosity of logging. | help |
--zap-stacktrace-level level | Zap Level at and above which stacktraces are captured (one of 'info', 'error', 'panic'). | help |
--zap-time-encoding time-encoding | Zap time encoding (one of 'epoch', 'millis', 'nano', 'iso8601', 'rfc3339' or 'rfc3339nano'). | help |