/usr/lib/python3.10/site-packages/click/core.py:1369: UserWarning: The parameter --data-config-route is used more than once. Remove its duplicate as parameters should be unique. parser = self.make_parser(ctx) /usr/lib/python3.10/site-packages/click/core.py:1988: UserWarning: The parameter --data-config-route is used more than once. Remove its duplicate as parameters should be unique. rest = super().parse_args(ctx, args) /usr/lib/python3.10/site-packages/click/core.py:1948: UserWarning: The parameter --data-config-route is used more than once. Remove its duplicate as parameters should be unique. rv = super().collect_usage_pieces(ctx) /usr/lib/python3.10/site-packages/click/core.py:1273: UserWarning: The parameter --data-config-route is used more than once. Remove its duplicate as parameters should be unique. self.format_arguments(ctx, formatter) /usr/lib/python3.10/site-packages/click/core.py:1953: UserWarning: The parameter --data-config-route is used more than once. Remove its duplicate as parameters should be unique. super().format_options(ctx, formatter) Usage: opal-server [OPTIONS] [COMMAND] [ARGS]... Config top level options: - Use env-vars (same as cmd options) but uppercase and with "_" instead of "-"; all prefixed with "OPAL_" - Use command line options as detailed by '--help' - Use .env or .ini files Examples: - opal-server --help Detailed help on CLI - opal-server run --help Help on run command - opal-server run --engine-type gunicorn Run server with gunicorn Options: --scopes-store-read-timeout FLOAT Timeout for a scope-store read taken while holding a source's lock — the sibling check a delete/repoint purge runs before authorizing the fleet-wide cache purge. The Redis client is built without a socket timeout, so without this an unreachable store would pin that lock for the life of the process and block every later sync, purge and delete for the source. On expiry the sibling check fails open, and what that decides is the fleet-wide MEMORY purge only — the leader no longer touches the clone tree: a DELETE confirms it defensively (its record is already gone, so withholding would strand the fleet's cache entries, while over-purging self-heals on the surviving sibling's next sync), a REPOINT withholds it (the old source's record is still live, just moved). The clone dir is unaffected either way: on a store fault the delete floor KEEPS this worker's clone rather than risk deleting one a live sibling shares, leaving an orphan tracked by PER-15612 (0 or negative means no timeout). [default: 10.0] --policy-refresh-interval INTEGER Policy polling refresh interval --base-dir TEXT The base directory for OPAL [default: /tmp/.local/state/opal] --redis-url TEXT The URL for the Redis server [default: (redis://localhost)] --scopes-repo-clones-shards INTEGER The max number of local clones to use for the same repo (reused across scopes) [default: 1] --scopes BOOLEAN Enable scopes --debug-internal-stats BOOLEAN Expose GET /internal/git-fetcher-cache-stats with in-memory cache sizes and process RSS. For diagnostics/tests only; keep off in production. --server-bind-port INTEGER (if run via CLI) Port for the server to bind [default: 7002] --server-port TEXT Deprecated, use SERVER_BIND_PORT instead --server-host TEXT (if run via CLI) Address for the server to bind [default: (127.0.0.1)] --server-worker-count INTEGER (if run via CLI) Worker count for the server [Default calculated to CPU-cores] --no-rpc-logs BOOLEAN Disable RPC logs [default: True] --bundle-ignore LIST List of patterns to ignore in the bundle --filter-file-extensions LIST List of file extensions to filter. Example: ['.rego', '.json'] [default: .rego, .json] --policy-repo-polling-interval INTEGER The polling interval for the policy repository --policy-repo-webhook-params Parameters for processing the incoming webhook [default: secret_header_name='x-hub-signature-256' secret_type= secret_parsing_regex='sha256=(.*)' event_header_name='X-GitHub-Event' event_request_key=None push_event_value='push' match_sender_url=True] --policy-repo-webhook-enforce-branch BOOLEAN Enforce branch name in incoming webhook --policy-repo-webhook-secret TEXT The secret for the policy repository webhook --data-config-route TEXT URL to trigger data update events [default: (/data/update)] --data-config-sources Configuration of data sources by topics [default: config=DataSourceConfig(entries=[D ataSourceEntryWithPollingInterval(url='http: //localhost:7002/policy-data', config='', topics=['policy_data'], dst_path='', save_method='PUT', data='', periodic_update_interval=None)]) external_source_url=None] --data-callback-default-route TEXT Exists as a sane default in case the user did not set OPAL_DEFAULT_UPDATE_CALLBACKS [default: (/data/callback_report)] --data-config-route TEXT URL to fetch the full basic configuration of data [default: (/data/config)] --all-data-url TEXT URL for all data config [If you choose to have it all at one place] [default: http://localhost:7002/policy-data] --all-data-route TEXT The route for all policy data [default: (/policy-data)] --all-data-topic TEXT Top level topic for data [default: (policy_data)] --scopes-purge-channel TEXT Pub/sub channel (worker-to-worker, over the broadcaster) used to purge GitPolicyFetcher caches fleet-wide when a scope is deleted or repointed to a new source. Every worker subscribes and drops its own cache entries; the leader is the only actor that may authorize that, because only it can check whether a surviving scope still shares the source. It does not remove the clone dir. [default: (__opal_scope_purge__)] --statistics-server-keepalive-timeout TEXT Timeout for forgetting a server from which a keep-alive haven't been seen (keep-alive frequency would be half of this value) [default: 20] --statistics-server-keepalive-channel TEXT The topic workers use to signal they exist and are alive [default: (__opal_stats_server_keepalive)] --statistics-state-sync-channel TEXT The topic other servers with statistics provide their state to a waking-up server [default: (__opal_stats_state_sync)] --statistics-wakeup-channel TEXT The topic a waking-up OPAL server uses to notify others he needs their statistics data [default: (__opal_stats_wakeup)] --max-channels-per-client INTEGER max number of records per client, after this number it will not be added to statistics, relevant only if STATISTICS_ENABLED [default: 15] --broadcast-keepalive-topic TEXT the topic on which we should send broadcaster keepalive messages [default: (__broadcast_session_keepalive__)] --broadcast-keepalive-interval INTEGER the time to wait between sending two consecutive broadcaster keepalive messages [default: 3600] --publisher-enabled BOOLEAN Enable the publisher [default: True] --repo-watcher-enabled BOOLEAN Enable the repository watcher. In scopes mode (SCOPES=true) this same flag enables the scopes sync task on the leader worker: the periodic sync_scopes pass, the post- leadership sync of all scopes and the fleet purger. It does NOT gate the pre-fork preload, which still clones/fetches every registered scope at boot. With the flag off, scopes are registered, preloaded at boot and served, but never re-synced or purged afterwards. [default: True] --policy-bundle-git-add-pattern TEXT File pattern to add files to git default to all the files (*) [default: (*)] --policy-bundle-tmp-path TEXT Path for temp policy file, need to be writeable [default: (/tmp/bundle.tar.gz)] --policy-bundle-server-aws-region TEXT The AWS region of the S3 bucket [default: (us-east-1)] --policy-bundle-server-token-id TEXT The id of the secret token to be sent to API bundle server --policy-bundle-server-token TEXT Secret token to be sent to API bundle server --policy-bundle-server-type The type of bundle server e.g. basic HTTP , AWS S3. (affects how we authenticate with it) [default: (HTTP)] --leader-lock-file-path TEXT The path to the leader lock file [default: (/tmp/opal_server_leader.lock)] --scopes-policy-clone-wait-max-inflight INTEGER Maximum number of requests one worker process may hold at once inside the SCOPES_POLICY_CLONE_WAIT_SECONDS wait. Requests beyond the cap get the immediate 503 + Retry-After 30 they would have got before the wait existed, so the cap can never make things worse than not waiting. It exists because polling is cheap but RELEASING is not: when the clone lands, every held request builds a full bundle on the loop's DEFAULT executor: about min(32, cpu+4) threads shared by every off-loop call this process makes, in front of an unbounded queue. Measured throughput there falls from ~52 bundles/s at 32 concurrent builds to ~18/s at 1000. Scope git clone/fetch does NOT share that pool — each op runs on its own single-use daemon-thread executor, bounded by SCOPES_GIT_MAX_WORKERS through a semaphore — so this key is the only bound on how many bundle builds can pile up at once. Size it so the cap divided by the bundles- per-second that pod can really build fits inside the 60s ALB idle timeout MINUS the hold: above that, released requests queue past the timeout and 504 — the very failure the hold exists to prevent — and a rolling restart drains worse, because uvicorn waits for in-flight requests while gunicorn SIGKILLs the worker at 30s, dropping every websocket that worker still holds. The count is per process, not per pod: a pod running N workers holds up to N times this number. 0 or negative means no cap. [default: 64] --scopes-policy-clone-wait-seconds FLOAT How long GET /scopes/:scope_id/policy holds a request while that scope's clone is still being populated, before falling through to the 503 + Retry-After it answers today. The route re-checks once a second and returns the bundle the moment the clone is usable. It exists because the opal-client PDP ignores Retry-After: it makes five attempts with random-exponential backoff capped at 10s (~20-40s of coverage) and then stays quiet until the next pub/sub policy message or a reconnect, so a clone that outlives those attempts strands that PDP with no policy — and the update-all published when a clone completes names only the scope that was syncing, so siblings sharing the same clone are not woken. Holding the request converts that gap into latency the client already tolerates: five client attempts against a 20s hold cover about two minutes of clone time, so short and medium re-clones — meaning the download phase; the rmtree- and-init window before it answers 503 + Retry-After 5 and is not waited on — produce no client-visible gap. What this bounds is the WAIT plus at most one more bundle attempt: time queued behind other bundle builds on the shared executor is outside the deadline, which is what SCOPES_POLICY_CLONE_WAIT_MAX_INFLIGHT bounds instead. The budget matters in both directions — 20s is well under the 60s ALB idle timeout (a longer hold surfaces as a 504, which the client cannot tell apart from a dead server) and far under the client's 300s aiohttp total timeout. Readiness is derived from DISK (the clone still has no remote-tracking refs), never from an in- process marker, so every worker answers alike: the clone runs in the leader while this route is served by any worker. The hold is an awaited sleep loop, so it occupies no thread between polls and leaves the event loop and the gunicorn worker heartbeat unaffected; it is abandoned early if the client disconnects. 0 or negative disables the wait (answer 503 immediately). nan, inf and -inf are treated as disabled too: unlike a non-numeric value, which fails this process at startup when the environment is parsed, they parse cleanly — inf would otherwise mean the clamped maximum hold on every clone-in-progress request, and nan is not a budget at all. Values above 55s are clamped to 55s so the hold can never outlive the load balancer's idle timeout. [default: 20.0] --scopes-git-backoff-max-seconds FLOAT Optional cap on the per-source doubling backoff of SCOPES_GIT_BACKOFF_BASE_SECONDS. 0 (the default), negative, nan or inf means NO cap: a repository that has been unreachable for a day is checked again in two, then four, and before long only at the next restart or explicit refresh — a repository that keeps failing is, in all likelihood, dead. Set a positive value to bound instead how stale a repository that comes back on its own (without anyone touching its scope) can get: the periodic pass then re-attempts it at most that long after the previous attempt. A value below the base is floored at the base (one pass at a time), never inert. Lowering the cap at runtime is not retroactive for delays already armed. --scopes-git-backoff-base-seconds FLOAT First delay before the periodic sync pass re-attempts a SOURCE whose git clone or fetch just failed (unreachable host, revoked credentials, deleted repo); every further consecutive failure doubles it — 10s, 20s, 40s, ... minutes, hours, days — with no upper bound unless SCOPES_GIT_BACKOFF_MAX_SECONDS is set. A delay shorter than the gap to the next pass simply does not skip that pass, so the first few doublings cost one attempt per pass exactly as before and the schedule bites from roughly the fourth consecutive failure. It exists because nothing else records a failure: without it every pass re-attempts every dead repo, and so does every duplicate scope sharing that repo — a handful of dead repositories can account for thousands of clone attempts per hour. Only the periodic pass and the boot preload honour it (in both phases, and re-checked under the source lock, so duplicates of a source that fails in a pass cost one attempt, not one per scope): an explicit POST /scopes/:scope_id/refresh, POST /scopes/refresh or PUT /scopes attempts the source immediately, so an operator who has just repaired credentials recovers at once, and any successful clone or fetch clears both the delay and the consecutive-failure count. The state is in-memory and per process — it resets on restart. When a periodic pass runs (POLICY_REFRESH_INTERVAL > 0) a forked worker inherits whatever the pre-fork preload recorded, so the leader does not re-hammer repos that already failed at boot; without one, the boot sync is the only pass-originated sync, so it drops the inherited entries and attempts every source once. Watch opal_server.scopes.sources_in_backoff (gauge of sources currently being skipped, tagged by pid) and opal_server.scopes.git_op_skipped with reason:backoff (counter); the WARNING logged when a source enters backoff, and again when its delay first exceeds a day, names the repository. 0 or negative disables the backoff entirely — nothing is recorded and nothing is skipped; nan and inf are treated as disabled too, because they parse cleanly rather than failing the process at startup and neither is a duration. [default: 10.0] --scopes-git-max-zombies INTEGER Maximum number of in-flight scope git operations (live plus lingering timed-out) allowed to hold a daemon thread at once, counted GLOBALLY across all sources. It is a last-resort ceiling on thread growth when remotes hang, not a per-source guard — that is handled separately, by skipping a source that already has an operation in flight. Once the ceiling is reached, new git ops are refused (logged, and retried next cycle) for EVERY scope, healthy ones included, until enough threads drain; with remotes that never return, that state can persist. Set it well above SCOPES_GIT_MAX_WORKERS and alert on the refusal log (0 = no cap; a negative value is clamped to 0 and also means no cap, at the cost of unbounded thread growth during an outage). [default: 40] --scopes-git-preload-drain-timeout FLOAT Max seconds the pre-fork scope preload waits for in-flight git ops to finish before tearing down and forking workers. Ops still lingering past this bound are left running on their daemon threads and their cached handles are left unfreed by the pre-fork cache reset, avoiding a use-after-free (0 = don't wait). [default: 10.0] --scopes-git-max-workers INTEGER Maximum number of concurrent scope git operations. It bounds phase 1 (the network clone/fetch of each distinct repo) AND phase 2 (the local change-check of scopes that reuse an already-cloned repo), so it sets how many scopes are synced at once in either phase. A timed-out operation stops counting against this limit, so one hung remote does not hold a concurrency slot — its lingering daemon thread persists on its own (a black- holed remote's can persist for the life of the process). That tail is bounded by SCOPES_GIT_MAX_ZOMBIES, which is a GLOBAL ceiling: read its description, because at that ceiling new git ops are refused for every scope, healthy ones included. [default: 10] --scopes-git-fetch-timeout FLOAT Soft timeout in seconds for a single scope git clone/fetch: the awaiting operation is abandoned (the event loop and the sync pass move on) and the op is logged and skipped, retried next cycle. It is a SOFT timeout: the underlying git call keeps running on its own thread, and the pinned libgit2 sets no socket/server read timeout, so a black-holed remote can keep that thread — and the source's in-flight marker — alive for the life of the process (that source is then skipped until it recovers or the process restarts); SCOPES_GIT_MAX_ZOMBIES bounds how many such threads accumulate. Either way one unreachable repo never blocks boot or other scopes (0 = no timeout). [default: 120.0] --policy-repo-clone-timeout INTEGER The timeout for cloning the policy repository (0 means wait forever) --policy-repo-manifest-path TEXT Path of the directory holding the '.manifest' file (new fashion), or of the manifest file itself (old fashion). Repo's root is used by default [default: ()] --policy-repo-ssh-key TEXT The SSH key for the policy repository --policy-repo-main-branch TEXT The main branch of the policy repository [default: (master)] --policy-repo-reuse-clone-path BOOLEAN Set if OPAL server should use a fixed clone path (and reuse if it already exists) instead of randomizing its suffix on each run --policy-repo-clone-folder-prefix TEXT Prefix for the local git folder [default: (opal_repo_clone)] --policy-repo-clone-path TEXT Base path to create local git folder inside it that manage policy change [default: (/tmp/regoclone)] --policy-bundle-url TEXT Set your API bundle URL, relevant only on API source type --policy-repo-url TEXT Set your remote repo URL e.g:https://github.com/permitio/opal- example-policy-repo.git , relevant only on GIT source type --policy-source-type Set your policy source can be GIT / API [default: (GIT)] --auth-master-token TEXT The master token for authentication --auth-jwks-static-dir TEXT The directory for static JWKS files [default: (/tmp/jwks_dir)] --auth-jwks-url TEXT The URL for the JSON Web Key Set (JWKS) [default: (/.well-known/jwks.json)] --auth-private-key The private key for authentication --auth-private-key-passphrase TEXT The passphrase for the private key --auth-private-key-format The format of the private key for authentication [default: (pem)] --broadcast-freeze-on-disconnect BOOLEAN During a broadcaster backbone gap, freeze client-facing publishes on every worker instead of applying them locally — so a write that cannot fan out to the whole fleet is never served by just one worker (fleet consistency over freshness). Recovery is the reconnect resync: clients re-fetch their configured data sources and policy, so updates covered by those are reconciled; one-off updates outside them (inline data payloads, ad-hoc fetch URLs) are DROPPED by a freeze, not deferred. Internal coordination topics (statistics, keepalive, the git webhook trigger) are exempt and keep the deliver-locally + buffer-for-replay behavior. Engages only with BROADCAST_RECONNECT_ENABLED (without the reconnecting broadcaster there is no gap signal, so the flag is a no-op) and requires BROADCAST_RESYNC_ON_RECONNECT (if the resync is disabled, freezing is refused with a warning). Set to False for the previous behavior, where the receiving worker's own clients update immediately and peers only after reconnect. [default: True] --broadcast-healthcheck-enabled BOOLEAN Make /healthcheck reflect the broadcaster reader's health so a k8s readiness/liveness probe can route away from or restart a worker whose reader is wedged while clients depend on it. Set to False to revert /healthcheck to always returning ok. [default: True] --broadcast-resync-settle-seconds FLOAT Grace period after a broadcaster reconnect before replaying buffered broadcasts and resyncing clients, to let peer servers re- subscribe. [default: 2.0] --broadcast-resync-on-reconnect BOOLEAN After a backbone gap that may have lost updates, force this worker's connected clients to reconnect so they re-fetch full policy + data state (guarantees cross- instance consistency). [default: True] --broadcast-replay-buffer-size INTEGER Max number of outbound broadcasts buffered while the backbone is down and replayed on reconnect (0 disables buffering). On overflow the oldest buffered broadcasts are dropped; the resync on reconnect still reconciles clients. [default: 10000] --broadcast-reconnect-backoff-max-seconds FLOAT Maximum backoff in seconds between broadcaster reconnect attempts. [default: 30.0] --broadcast-reconnect-backoff-min-seconds FLOAT Minimum backoff in seconds between broadcaster reconnect attempts. [default: 0.5] --broadcast-reconnect-max-retries INTEGER Maximum consecutive broadcaster reconnect attempts before giving up and letting the worker restart (0 = retry forever). --broadcast-reconnect-enabled BOOLEAN Reconnect the broadcaster reader on a backbone disconnect instead of dropping all client connections. Set to False to revert to the legacy (non-reconnecting) broadcaster. [default: True] --broadcast-channel-name TEXT The name to be used for segmentation in the backbone pub/sub [default: (EventNotifier)] --broadcast-uri TEXT The URL for the backbone pub/sub server --client-load-limit-notation TEXT If supplied, rate limit would be enforced on server's websocket endpoint. Format is `limits`-style notation (e.g '10 per second'), see link: https://limits.readthedo cs.io/en/stable/quickstart.html#rate-limit- string-notation --ws-token TEXT The WebSocket token for OPAL [default: (THIS_IS_A_DEV_SECRET)] --ws-local-url TEXT The local WebSocket URL for OPAL [default: (ws://localhost:7002/ws)] --http-fetcher-timeout FLOAT The timeout for the httpx or aiohttp fetcher provider, in seconds. if provided different value, 5 seconds will be used. [default: 5] --http-fetcher-provider-client TEXT The client to use for fetching data, can be either aiohttp or httpx.if provided different value, aiohttp will be used. [default: (aiohttp)] --enable-datadog-apm BOOLEAN Set if OPAL server should enable tracing with datadog APM --enable-metrics BOOLEAN Enable metrics collection --policy-repo-policy-extensions LIST List of extensions to serve as policy modules [default: .rego] --auth-jwt-issuer TEXT Issuer for JWT authentication [default: (https://opal.ac/)] --auth-jwt-audience TEXT Audience for JWT authentication [default: (https://api.opal.ac/v1/)] --auth-jwt-algorithm jwt algorithm, possible values: see: https:/ /pyjwt.readthedocs.io/en/stable/algorithms.h tml [default: RS256] --auth-public-key Public key for authentication --auth-public-key-format Format of the public key for authentication [default: (ssh)] --client-ssl-context-trusted-ca-file TEXT A path to your own CA public certificate file (usually a .crt or a .pem file). Certificates signed by this issuer will be trusted by OPAL Client. DO NOT USE THIS IN PRODUCTION! --client-self-signed-certificates-allowed BOOLEAN Whether or not OPAL Client will trust HTTPs connections protected by self signed certificates. DO NOT USE THIS IN PRODUCTION! --git-ssh-key-file TEXT Path to the SSH key file for Git [default: (/tmp/.ssh/opal_repo_ssh_key)] --fetching-enqueue-timeout INTEGER Time in seconds to wait for queuing a new task (if the queue is full) [default: 10] --fetching-callback-timeout INTEGER Time in seconds to wait on the queued fetch task [default: 10] --fetching-worker-count INTEGER Max number of worker tasks handling fetch events concurrently [default: 6] --fetch-provider-modules LIST List of modules to load fetch providers from [default: opal_common.fetcher.providers] --statistics-remove-client-channel TEXT The topic to update about OPAL clients disconnection [default: (__opal_stats_rm)] --statistics-add-client-channel TEXT The topic to update about new OPAL clients connection [default: (__opal_stats_add)] --statistics-enabled BOOLEAN Set if OPAL server will collect statistics about OPAL clients may cause a small performance hit --log-file-level TEXT The log level to show in file [default: (INFO)] --log-file-serialize TEXT Serialize log messages in file [default: True] --log-file-compression TEXT Log file compression format --log-file-retention TEXT Log file retention time [default: (10 days)] --log-file-rotation TEXT Log file rotation size [default: (250 MB)] --log-file-path TEXT path to save log file [default: (opal_{time}.log)] --log-to-file BOOLEAN Should we log to a file --log-patch-uvicorn-logs BOOLEAN Should we takeover UVICORN's logs so they appear in the main logger [default: True] --log-module-include-list LIST List of modules to include in logging [default: uvicorn.protocols.http] --log-module-exclude-list LIST List of modules to exclude from logging [default: uvicorn] --log-level TEXT The log level to show [default: (INFO)] --log-show-code-line BOOLEAN Show code line in log messages [default: True] --log-pipe-to-stderr BOOLEAN Should we send logs to stderr? otherwise to stdout [default: True] --log-serialize BOOLEAN Serialize log messages --log-colorize BOOLEAN Colorize log messages [default: True] --log-diagnose BOOLEAN Include diagnosis (local variable values) in tracebacks. Off by default because loguru renders raw variable values, which can leak credentials (e.g. auth headers/tokens) into logs - only enable for local debugging. Also gates verbose git SSH protocol tracing (GIT_TRACE/GIT_CURL_VERBOSE on SSH clones), which adds noisy protocol/host disclosure to logs. --log-traceback BOOLEAN Include traceback in log messages [default: True] --log-format TEXT The format of the log messages [default: {time} | {name: <40}|{level:^6} | {message} {exception}] --log-format-include-pid BOOLEAN Include process ID in log format --allowed-origins LIST List of allowed origins for CORS [default: *] --help Show this message and exit. Commands: generate-secret obtain-token Obtain a secret JWT (JSON-Web-Token) from the... print-config To test config values, print the configuration... publish-data-update Publish a DataUpdate through an OPAL-server... run Run the server as a daemon. version Print the OPAL version.