/usr/bin/safekeeper

A fleet of safekeepers is responsible for reliably storing WAL received from
compute, passing it through consensus (mitigating potential computes brain
split), and serving the hardened part further downstream to pageserver(s).


Usage: safekeeper [OPTIONS]

Options:
  -D, --datadir <DATADIR>
          Path to the safekeeper data directory

          [default: ./]

      --id <ID>
          Safekeeper node id

      --init
          Initialize safekeeper with given id and exit

  -l, --listen-pg <LISTEN_PG>
          Listen endpoint for receiving/sending WAL in the form host:port

          [default: 127.0.0.1:5454]

      --listen-pg-tenant-only <LISTEN_PG_TENANT_ONLY>
          Listen endpoint for receiving/sending WAL in the form host:port allowing
          only tenant scoped auth tokens. Pointless if auth is disabled.

      --listen-http <LISTEN_HTTP>
          Listen http endpoint for management and metrics in the form host:port

          [default: 127.0.0.1:7676]

      --listen-https <LISTEN_HTTPS>
          Listen https endpoint for management and metrics in the form host:port

      --advertise-pg <ADVERTISE_PG>
          Advertised endpoint for receiving/sending WAL in the form host:port. If not specified, listen_pg is used to advertise instead

      --availability-zone <AVAILABILITY_ZONE>
          Availability zone of the safekeeper

  -n, --no-sync
          Do not wait for changes to be written safely to disk. Unsafe

      --dump-control-file <DUMP_CONTROL_FILE>
          Dump control file at path specified by this argument and exit

      --broker-endpoint <BROKER_ENDPOINT>
          Broker endpoint for storage nodes coordination in the form
          http[s]://host:port. In case of https schema TLS is connection is
          established; plaintext otherwise.

          [default: http://127.0.0.1:50051]

      --broker-keepalive-interval <BROKER_KEEPALIVE_INTERVAL>
          Broker keepalive interval

          [default: "5000 ms"]

      --heartbeat-timeout <HEARTBEAT_TIMEOUT>
          Peer safekeeper is considered dead after not receiving heartbeats from
          it during this period passed as a human readable duration.

          [default: 5000ms]

      --peer-recovery <PEER_RECOVERY>
          Enable/disable peer recovery

          [default: false]
          [possible values: true, false]

      --remote-storage <REMOTE_STORAGE>
          Remote storage configuration for WAL backup (offloading to s3) as TOML
          inline table, e.g.
            {max_concurrent_syncs = 17, max_sync_errors = 13, bucket_name = "<BUCKETNAME>", bucket_region = "<REGION>", concurrency_limit = 119}
          Safekeeper offloads WAL to
            [prefix_in_bucket/]<tenant_id>/<timeline_id>/<segment_file>, mirroring
          structure on the file system.

      --max-offloader-lag <MAX_OFFLOADER_LAG>
          Safekeeper won't be elected for WAL offloading if it is lagging for more than this value in bytes

          [default: 134217728]

      --max-reelect-offloader-lag-bytes <MAX_REELECT_OFFLOADER_LAG_BYTES>
          Safekeeper will re-elect a new offloader if the current backup lagging for more than this value in bytes

          [default: 0]

      --max-timeline-disk-usage-bytes <MAX_TIMELINE_DISK_USAGE_BYTES>
          Safekeeper will stop accepting new WALs if the timeline disk usage exceeds this value in bytes. Setting this value to 0 disables the limit

          [default: 0]

      --wal-backup-parallel-jobs <WAL_BACKUP_PARALLEL_JOBS>
          Number of max parallel WAL segments to be offloaded to remote storage

          [default: 5]

      --disable-wal-backup
          Disable WAL backup to s3. When disabled, safekeeper removes WAL ignoring WAL backup horizon

      --pg-auth-public-key-path <PG_AUTH_PUBLIC_KEY_PATH>
          If given, enables auth on incoming connections to WAL service endpoint
          (--listen-pg). Value specifies path to a .pem public key used for
          validations of JWT tokens. Empty string is allowed and means disabling
          auth.

      --pg-tenant-only-auth-public-key-path <PG_TENANT_ONLY_AUTH_PUBLIC_KEY_PATH>
          If given, enables auth on incoming connections to tenant only WAL
          service endpoint (--listen-pg-tenant-only). Value specifies path to a
          .pem public key used for validations of JWT tokens. Empty string is
          allowed and means disabling auth.

      --http-auth-public-key-path <HTTP_AUTH_PUBLIC_KEY_PATH>
          If given, enables auth on incoming connections to http management
          service endpoint (--listen-http). Value specifies path to a .pem public
          key used for validations of JWT tokens. Empty string is allowed and
          means disabling auth.

      --log-format <LOG_FORMAT>
          Format for logging, either 'plain' or 'json'

          [default: plain]

      --current-thread-runtime
          Run everything in single threaded current thread runtime, might be useful for debugging

      --walsenders-keep-horizon
          Keep horizon for walsenders, i.e. don't remove WAL segments that are still needed for existing replication connection

      --partial-backup-timeout <PARTIAL_BACKUP_TIMEOUT>
          Controls how long backup will wait until uploading the partial segment.

          [default: 15m]

      --disable-periodic-broker-push
          Disable task to push messages to broker every second. Supposed to be used in tests

      --enable-offload
          Enable automatic switching to offloaded state

      --delete-offloaded-wal
          Delete local WAL files after offloading. When disabled, they will be left on disk

      --control-file-save-interval <CONTROL_FILE_SAVE_INTERVAL>
          Pending updates to control file will be automatically saved after this interval

          [default: 300s]

      --partial-backup-concurrency <PARTIAL_BACKUP_CONCURRENCY>
          Number of allowed concurrent uploads of partial segments to remote storage

          [default: 5]

      --eviction-min-resident <EVICTION_MIN_RESIDENT>
          How long a timeline must be resident before it is eligible for eviction. Usually, timeline eviction has to wait for `partial_backup_timeout` before being eligible for eviction, but if a timeline is un-evicted and then _not_ written to, it would immediately flap to evicting again, if it weren't for `eviction_min_resident` preventing that.

          Also defines interval for eviction retries.

          [default: 15m]

      --wal-reader-fanout
          Enable fanning out WAL to different shards from the same reader

      --max-delta-for-fanout <MAX_DELTA_FOR_FANOUT>
          Only fan out the WAL reader if the absoulte delta between the new requested position and the current position of the reader is smaller than this value

      --ssl-key-file <SSL_KEY_FILE>
          Path to a file with certificate's private key for https API

          [default: server.key]

      --ssl-cert-file <SSL_CERT_FILE>
          Path to a file with a X509 certificate for https API

          [default: server.crt]

      --ssl-cert-reload-period <SSL_CERT_RELOAD_PERIOD>
          Period to reload certificate and private key from files

          [default: 60s]

      --ssl-ca-file <SSL_CA_FILE>
          Trusted root CA certificates to use in https APIs

      --use-https-safekeeper-api
          Flag to use https for requests to peer's safekeeper API

      --auth-token-path <AUTH_TOKEN_PATH>
          Path to the JWT auth token used to authenticate with other safekeepers

      --enable-tls-wal-service-api
          Enable TLS in WAL service API. Does not force TLS: the client negotiates TLS usage during the handshake. Uses key and certificate from ssl_key_file/ssl_cert_file

      --force-metric-collection-on-scrape
          Controls whether to collect all metrics on each scrape or to return potentially stale results

      --dev
          Run in development mode (disables security checks)

      --enable-pull-timeline-on-startup


      --global-disk-check-interval <GLOBAL_DISK_CHECK_INTERVAL>
          How often to scan entire data-dir for total disk usage

          [default: 60s]

      --max-global-disk-usage-ratio <MAX_GLOBAL_DISK_USAGE_RATIO>
          The portion of the filesystem capacity that can be used by all timelines. A circuit breaker will trip and reject all WAL writes if the total usage exceeds this ratio. Set to 0 to disable the global disk usage limit

          [default: 0]

  -h, --help
          Print help (see a summary with '-h')

  -V, --version
          Print version