You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
22 lines
799 B
22 lines
799 B
##
|
|
## vLLM engine configuration — smol135 (CPU smoke test)
|
|
##
|
|
## Consumed by `vllm serve --config` (see vllm-model@.container). Each key is a
|
|
## long CLI flag with the leading `--` stripped; store-true flags (e.g.
|
|
## --enforce-eager) become `true`.
|
|
##
|
|
## Installed to /etc/quadlets/vllm/models/smol135.yaml and mounted read-only into
|
|
## the container at /etc/vllm/config.yaml.
|
|
##
|
|
|
|
# Ultra-light model of a DIFFERENT architecture (Llama vs Qwen2) so the swap is
|
|
# genuinely exercised. No tool-call parser (SmolLM has no tool template).
|
|
model: HuggingFaceTB/SmolLM2-135M-Instruct
|
|
served-model-name: smol135
|
|
dtype: bfloat16
|
|
max-model-len: 4096
|
|
max-num-seqs: 4
|
|
|
|
# --enforce-eager: this CPU image's torch inductor JIT fails at warmup; eager
|
|
# mode skips it (fine for a smoke test).
|
|
enforce-eager: true
|
|
|