/usr/bin/text-generation-benchmark

Text Generation Benchmarking tool

Usage: text-generation-benchmark [OPTIONS] --tokenizer-name <TOKENIZER_NAME>

Options:
  -t, --tokenizer-name <TOKENIZER_NAME>
          The name of the tokenizer (as in model_id on the huggingface hub, or local path)

          [env: TOKENIZER_NAME=]

      --revision <REVISION>
          The revision to use for the tokenizer if on the hub

          [env: REVISION=]
          [default: main]

  -b, --batch-size <BATCH_SIZE>
          The various batch sizes to benchmark for, the idea is to get enough batching to start seeing increased latency, this usually means you're moving from memory bound (usual as BS=1) to compute bound, and this is a sweet spot for the maximum batch size for the model under test

  -s, --sequence-length <SEQUENCE_LENGTH>
          This is the initial prompt sent to the text-generation-server length in token. Longer prompt will slow down the benchmark. Usually the latency grows somewhat linearly with this for the prefill step.

          Most importantly, the prefill step is usually not the one dominating your runtime, so it's ok to keep it short.

          [env: SEQUENCE_LENGTH=]
          [default: 10]

  -d, --decode-length <DECODE_LENGTH>
          This is how many tokens will be generated by the server and averaged out to give the `decode` latency. This is the *critical* number you want to optimize for LLM spend most of their time doing decoding.

          Decode latency is usually quite stable.

          [env: DECODE_LENGTH=]
          [default: 8]

  -r, --runs <RUNS>
          How many runs should we average from

          [env: RUNS=]
          [default: 10]

  -w, --warmups <WARMUPS>
          Number of warmup cycles

          [env: WARMUPS=]
          [default: 1]

  -m, --master-shard-uds-path <MASTER_SHARD_UDS_PATH>
          The location of the grpc socket. This benchmark tool bypasses the router completely and directly talks to the gRPC processes

          [env: MASTER_SHARD_UDS_PATH=]
          [default: /tmp/text-generation-server-0]

      --temperature <TEMPERATURE>
          Generation parameter in case you want to specifically test/debug particular decoding strategies, for full doc refer to the `text-generation-server`

          [env: TEMPERATURE=]

      --top-k <TOP_K>
          Generation parameter in case you want to specifically test/debug particular decoding strategies, for full doc refer to the `text-generation-server`

          [env: TOP_K=]

      --top-p <TOP_P>
          Generation parameter in case you want to specifically test/debug particular decoding strategies, for full doc refer to the `text-generation-server`

          [env: TOP_P=]

      --typical-p <TYPICAL_P>
          Generation parameter in case you want to specifically test/debug particular decoding strategies, for full doc refer to the `text-generation-server`

          [env: TYPICAL_P=]

      --repetition-penalty <REPETITION_PENALTY>
          Generation parameter in case you want to specifically test/debug particular decoding strategies, for full doc refer to the `text-generation-server`

          [env: REPETITION_PENALTY=]

      --frequency-penalty <FREQUENCY_PENALTY>
          Generation parameter in case you want to specifically test/debug particular decoding strategies, for full doc refer to the `text-generation-server`

          [env: FREQUENCY_PENALTY=]

      --watermark
          Generation parameter in case you want to specifically test/debug particular decoding strategies, for full doc refer to the `text-generation-server`

          [env: WATERMARK=]

      --do-sample
          Generation parameter in case you want to specifically test/debug particular decoding strategies, for full doc refer to the `text-generation-server`

          [env: DO_SAMPLE=]

      --top-n-tokens <TOP_N_TOKENS>
          Generation parameter in case you want to specifically test/debug particular decoding strategies, for full doc refer to the `text-generation-server`

          [env: TOP_N_TOKENS=]

  -h, --help
          Print help (see a summary with '-h')

  -V, --version
          Print version