SGLang Model Gateway - Rust-based inference gateway

Usage:
  smg launch [OPTIONS]     Launch router (short command)
  amg launch [OPTIONS]     Launch router (alternative)
  sglang-router [OPTIONS]  Launch router (full name)

Examples:
  # Regular mode
  smg launch --worker-urls http://worker1:8000 http://worker2:8000

  # PD disaggregated mode
  smg launch --pd-disaggregation \
    --prefill http://127.0.0.1:30001 9001 \
    --prefill http://127.0.0.2:30002 9002 \
    --decode http://127.0.0.3:30003 \
    --decode http://127.0.0.4:30004 \
    --policy cache_aware

  # With different policies
  smg launch --pd-disaggregation \
    --prefill http://127.0.0.1:30001 9001 \
    --prefill http://127.0.0.2:30002 \
    --decode http://127.0.0.3:30003 \
    --decode http://127.0.0.4:30004 \
    --prefill-policy cache_aware --decode-policy power_of_two



Usage: sgl-model-gateway [OPTIONS]
       sgl-model-gateway <COMMAND>

Commands:
  launch  Launch the router (same as running without subcommand) [alias: start]
  help    Print this message or the help of the given subcommand(s)

Options:
      --enable-mesh


      --mesh-server-name <MESH_SERVER_NAME>


      --mesh-host <MESH_HOST>
          [default: 0.0.0.0]

      --mesh-port <MESH_PORT>
          [default: 39527]

      --mesh-peer-urls [<MESH_PEER_URLS>...]


  -h, --help
          Print help (see a summary with '-h')

Worker Configuration:
      --host <HOST>
          Host address to bind the router server

          [default: 0.0.0.0]

      --port <PORT>
          Port number to bind the router server

          [default: 30000]

      --worker-urls [<WORKER_URLS>...]
          List of worker URLs (supports IPv4 and IPv6)

Routing Policy:
      --policy <POLICY>
          Load balancing policy to use

          [default: cache_aware]
          [possible values: random, round_robin, cache_aware, power_of_two, prefix_hash, manual]

      --cache-threshold <CACHE_THRESHOLD>
          Cache threshold (0.0-1.0) for cache-aware routing

          [default: 0.3]

      --balance-abs-threshold <BALANCE_ABS_THRESHOLD>
          Absolute threshold for load balancing trigger

          [default: 64]

      --balance-rel-threshold <BALANCE_REL_THRESHOLD>
          Relative threshold for load balancing trigger

          [default: 1.5]

      --eviction-interval <EVICTION_INTERVAL>
          Interval in seconds between cache eviction operations

          [default: 120]

      --max-tree-size <MAX_TREE_SIZE>
          Maximum size of the approximation tree for cache-aware routing

          [default: 67108864]

      --max-idle-secs <MAX_IDLE_SECS>
          Maximum idle time in seconds before eviction (for manual policy)

          [default: 14400]

      --assignment-mode <ASSIGNMENT_MODE>
          Assignment mode for manual policy when encountering a new routing key

          [default: random]
          [possible values: random, min_load, min_group]

      --prefix-token-count <PREFIX_TOKEN_COUNT>
          Number of prefix tokens to use for prefix_hash policy

          [default: 256]

      --prefix-hash-load-factor <PREFIX_HASH_LOAD_FACTOR>
          Load factor threshold for prefix_hash policy

          [default: 1.25]

      --dp-aware
          Enable data parallelism aware scheduling

      --enable-igw
          Enable IGW (Inference Gateway) mode for multi-model support

PD Disaggregation:
      --pd-disaggregation
          Enable PD (Prefill-Decode) disaggregated mode

      --decode <DECODE>
          Decode server URLs (can be specified multiple times)

      --prefill-policy <PREFILL_POLICY>
          Specific policy for prefill nodes in PD mode

          [possible values: random, round_robin, cache_aware, power_of_two, prefix_hash, manual]

      --decode-policy <DECODE_POLICY>
          Specific policy for decode nodes in PD mode

          [possible values: random, round_robin, cache_aware, power_of_two, prefix_hash, manual]

      --worker-startup-timeout-secs <WORKER_STARTUP_TIMEOUT_SECS>
          Timeout in seconds for worker startup and registration

          [default: 1800]

      --worker-startup-check-interval <WORKER_STARTUP_CHECK_INTERVAL>
          Interval in seconds between worker startup checks

          [default: 30]

Service Discovery (Kubernetes):
      --service-discovery
          Enable Kubernetes service discovery

      --selector [<SELECTOR>...]
          Label selector for Kubernetes service discovery (format: key=value)

      --service-discovery-port <SERVICE_DISCOVERY_PORT>
          Port to use for discovered worker pods

          [default: 80]

      --service-discovery-namespace <SERVICE_DISCOVERY_NAMESPACE>
          Kubernetes namespace to watch for pods

      --prefill-selector [<PREFILL_SELECTOR>...]
          Label selector for prefill server pods in PD mode

      --decode-selector [<DECODE_SELECTOR>...]
          Label selector for decode server pods in PD mode

Logging:
      --log-dir <LOG_DIR>
          Directory to store log files

      --log-level <LOG_LEVEL>
          Set the logging level

          [default: info]
          [possible values: debug, info, warn, error]

      --json-log
          Enable structured JSON log output instead of plain text

Prometheus Metrics:
      --prometheus-port <PROMETHEUS_PORT>
          Port to expose Prometheus metrics

          [default: 29000]

      --prometheus-host <PROMETHEUS_HOST>
          Host address to bind the Prometheus metrics server

          [default: 0.0.0.0]

      --prometheus-duration-buckets [<PROMETHEUS_DURATION_BUCKETS>...]
          Custom buckets for Prometheus duration metrics

Request Handling:
      --request-id-headers [<REQUEST_ID_HEADERS>...]
          Custom HTTP headers to check for request IDs

      --request-timeout-secs <REQUEST_TIMEOUT_SECS>
          Request timeout in seconds

          [default: 1800]

      --shutdown-grace-period-secs <SHUTDOWN_GRACE_PERIOD_SECS>
          Grace period in seconds to wait for in-flight requests during shutdown

          [default: 180]

      --max-payload-size <MAX_PAYLOAD_SIZE>
          Maximum payload size in bytes

          [default: 536870912]

      --cors-allowed-origins [<CORS_ALLOWED_ORIGINS>...]
          CORS allowed origins

HTTP Client:
      --pool-idle-timeout-secs <POOL_IDLE_TIMEOUT_SECS>
          Idle timeout in seconds for pooled upstream HTTP connections

          [env: SMG_POOL_IDLE_TIMEOUT_SECS=]
          [default: 50]

      --connect-timeout-secs <CONNECT_TIMEOUT_SECS>
          Timeout in seconds for new upstream HTTP connections

          [env: SMG_CONNECT_TIMEOUT_SECS=]
          [default: 10]

      --pool-max-idle-per-host <POOL_MAX_IDLE_PER_HOST>
          Maximum idle upstream HTTP connections to keep per host

          [env: SMG_POOL_MAX_IDLE_PER_HOST=]
          [default: 500]

      --tcp-keepalive-secs <TCP_KEEPALIVE_SECS>
          TCP keepalive idle time in seconds for upstream HTTP connections

          [env: SMG_TCP_KEEPALIVE_SECS=]
          [default: 30]

Rate Limiting:
      --max-concurrent-requests <MAX_CONCURRENT_REQUESTS>
          Maximum concurrent requests (-1 to disable)

          [default: -1]

      --queue-size <QUEUE_SIZE>
          Queue size for pending requests when limit reached

          [default: 100]

      --queue-timeout-secs <QUEUE_TIMEOUT_SECS>
          Maximum time in seconds a request can wait in queue

          [default: 60]

      --rate-limit-tokens-per-second <RATE_LIMIT_TOKENS_PER_SECOND>
          Token bucket refill rate (tokens per second)

Retry Configuration:
      --retry-max-retries <RETRY_MAX_RETRIES>
          Maximum number of retry attempts

          [default: 5]

      --retry-initial-backoff-ms <RETRY_INITIAL_BACKOFF_MS>
          Initial backoff delay in milliseconds

          [default: 50]

      --retry-max-backoff-ms <RETRY_MAX_BACKOFF_MS>
          Maximum backoff delay in milliseconds

          [default: 30000]

      --retry-backoff-multiplier <RETRY_BACKOFF_MULTIPLIER>
          Multiplier for exponential backoff

          [default: 1.5]

      --retry-jitter-factor <RETRY_JITTER_FACTOR>
          Jitter factor (0.0-1.0) for retry delays

          [default: 0.2]

      --disable-retries
          Disable automatic retries

Circuit Breaker:
      --cb-failure-threshold <CB_FAILURE_THRESHOLD>
          Number of failures before circuit opens

          [default: 10]

      --cb-success-threshold <CB_SUCCESS_THRESHOLD>
          Successes needed in half-open state to close

          [default: 3]

      --cb-timeout-duration-secs <CB_TIMEOUT_DURATION_SECS>
          Seconds before attempting to close open circuit

          [default: 60]

      --cb-window-duration-secs <CB_WINDOW_DURATION_SECS>
          Sliding window duration for tracking failures

          [default: 120]

      --disable-circuit-breaker
          Disable circuit breaker

Health Checks:
      --health-failure-threshold <HEALTH_FAILURE_THRESHOLD>
          Failures before marking worker unhealthy

          [default: 3]

      --health-success-threshold <HEALTH_SUCCESS_THRESHOLD>
          Successes before marking worker healthy

          [default: 2]

      --health-check-timeout-secs <HEALTH_CHECK_TIMEOUT_SECS>
          Timeout in seconds for health check requests

          [default: 5]

      --health-check-interval-secs <HEALTH_CHECK_INTERVAL_SECS>
          Interval in seconds between health checks

          [default: 60]

      --health-check-endpoint <HEALTH_CHECK_ENDPOINT>
          Health check endpoint path

          [default: /health]

      --disable-health-check
          Disable all worker health checks at startup

Tokenizer:
      --model-path <MODEL_PATH>
          Model path for loading tokenizer (HuggingFace ID or local path)

      --tokenizer-path <TOKENIZER_PATH>
          Explicit tokenizer path (overrides model_path)

      --chat-template <CHAT_TEMPLATE>
          Chat template path

      --tokenizer-cache-enable-l0
          Enable L0 (exact match) tokenizer cache

      --tokenizer-cache-l0-max-entries <TOKENIZER_CACHE_L0_MAX_ENTRIES>
          Maximum entries in L0 tokenizer cache

          [default: 10000]

      --tokenizer-cache-enable-l1
          Enable L1 (prefix matching) tokenizer cache

      --tokenizer-cache-l1-max-memory <TOKENIZER_CACHE_L1_MAX_MEMORY>
          Maximum memory for L1 tokenizer cache in bytes

          [default: 52428800]

Parsers:
      --reasoning-parser <REASONING_PARSER>
          Parser for reasoning models (e.g., deepseek-r1, qwen3)

      --tool-call-parser <TOOL_CALL_PARSER>
          Parser for tool-call interactions

      --mcp-config-path <MCP_CONFIG_PATH>
          Path to MCP server configuration file

Backend:
      --backend <BACKEND>
          Backend runtime to use

          [default: sglang]
          [possible values: sglang, vllm, trtllm, openai, anthropic]

      --history-backend <HISTORY_BACKEND>
          History storage backend

          [default: memory]
          [possible values: memory, none, oracle, postgres, redis]

      --enable-wasm
          Enable WebAssembly support

Oracle Database:
      --oracle-wallet-path <ORACLE_WALLET_PATH>
          Path to Oracle ATP wallet directory

          [env: ATP_WALLET_PATH=]

      --oracle-tns-alias <ORACLE_TNS_ALIAS>
          Oracle TNS alias from tnsnames.ora

          [env: ATP_TNS_ALIAS=]

      --oracle-dsn <ORACLE_DSN>
          Oracle connection descriptor/DSN

          [env: ATP_DSN=]

      --oracle-user <ORACLE_USER>
          Oracle database username

          [env: ATP_USER=]

      --oracle-password <ORACLE_PASSWORD>
          Oracle database password

          [env: ATP_PASSWORD=]

      --oracle-pool-min <ORACLE_POOL_MIN>
          Minimum Oracle connection pool size

          [env: ATP_POOL_MIN=]

      --oracle-pool-max <ORACLE_POOL_MAX>
          Maximum Oracle connection pool size

          [env: ATP_POOL_MAX=]

      --oracle-pool-timeout-secs <ORACLE_POOL_TIMEOUT_SECS>
          Oracle connection pool timeout in seconds

          [env: ATP_POOL_TIMEOUT_SECS=]

PostgreSQL Database:
      --postgres-db-url <POSTGRES_DB_URL>
          PostgreSQL database connection URL

      --postgres-pool-max-size <POSTGRES_POOL_MAX_SIZE>
          Maximum PostgreSQL connection pool size

Redis Database:
      --redis-url <REDIS_URL>
          Redis connection URL

      --redis-pool-max-size <REDIS_POOL_MAX_SIZE>
          Maximum Redis connection pool size

      --redis-retention-days <REDIS_RETENTION_DAYS>
          Redis data retention in days (-1 for persistent, default 30)

TLS/mTLS Security:
      --tls-cert-path <TLS_CERT_PATH>
          Path to server TLS certificate (PEM format)

      --tls-key-path <TLS_KEY_PATH>
          Path to server TLS private key (PEM format)

Tracing (OpenTelemetry):
      --enable-trace
          Enable OpenTelemetry tracing

      --otlp-traces-endpoint <OTLP_TRACES_ENDPOINT>
          OTLP collector endpoint (format: host:port)

          [default: localhost:4317]

Control Plane Authentication:
      --api-key <API_KEY>
          API key for worker authorization

      --jwt-issuer <JWT_ISSUER>
          JWT issuer URL for OIDC authentication

          [env: JWT_ISSUER=]

      --jwt-audience <JWT_AUDIENCE>
          Expected JWT audience claim

          [env: JWT_AUDIENCE=]

      --jwt-jwks-uri <JWT_JWKS_URI>
          Explicit JWKS URI (discovered from issuer if not set)

          [env: JWT_JWKS_URI=]

      --jwt-role-claim <JWT_ROLE_CLAIM>
          JWT claim name containing the role

          [default: roles]

      --jwt-role-mapping <JWT_ROLE_MAPPING>
          Role mapping from IDP to gateway role (format: idp_role=gateway_role)

      --control-plane-api-keys <CONTROL_PLANE_API_KEYS>
          API keys for control plane access (format: id:name:role:key)

          [env: CONTROL_PLANE_API_KEYS=]

      --disable-audit-logging
          Disable audit logging for control plane operations