You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
50 lines
2.5 KiB
50 lines
2.5 KiB
##
|
|
## vLLM environment file
|
|
##
|
|
## Injected into every vLLM container launched by llmsnap (via --env-file in
|
|
## config.yaml). Installed with 0600 root:root permissions as it holds a secret.
|
|
##
|
|
|
|
# Hugging Face token, used to download gated / private model weights.
|
|
#HF_TOKEN=hf_replace_me
|
|
|
|
# Optional: speed up weight downloads (requires hf_transfer in the image).
|
|
#HF_HUB_ENABLE_HF_TRANSFER=1
|
|
|
|
# Keep vLLM's INTERNAL inter-worker traffic off the LAN. With Network=host, a
|
|
# multi-worker instance (tensor/pipeline parallel) otherwise exposes a fan of
|
|
# `VLLM::Worker` listeners on the primary routable IP. Two independent knobs are
|
|
# needed because vLLM binds these ports through two different backends, and ONLY
|
|
# the second one is what you actually see in `netstat`:
|
|
#
|
|
# 1. VLLM_HOST_IP -> vLLM's own ZMQ path (get_ip(), MessageQueue). In a
|
|
# single-node run its remote socket is not even bound (local readers use
|
|
# IPC), so this alone has NO visible effect — but it is the documented knob
|
|
# and correct to pin. https://docs.vllm.ai/en/stable/configuration/env_vars/
|
|
#
|
|
# 2. GLOO_SOCKET_IFNAME / NCCL_SOCKET_IFNAME -> the torch distributed process
|
|
# group (gloo on CPU, nccl on GPU). vLLM builds ProcessGroupGloo WITHOUT an
|
|
# interface (distributed/utils.py: init_gloo_process_group), so gloo falls
|
|
# back to resolving the hostname and binds the default-route interface (the
|
|
# LAN IP). These are torch/NCCL env vars, NOT routed through VLLM_HOST_IP —
|
|
# this is THE fix for the `VLLM::Worker` listeners. `lo` = loopback iface.
|
|
#
|
|
# All valid ONLY because this is a SINGLE-NODE deployment (every worker on this
|
|
# host, loopback reachable). Do NOT set these on a real multi-node cluster.
|
|
# Distinct from the API server bind, which is --host/--port in
|
|
# vllm-model@.container (COOKBOOK_VLLM_MODEL_PORT).
|
|
VLLM_HOST_IP=127.0.0.1
|
|
GLOO_SOCKET_IFNAME=lo
|
|
NCCL_SOCKET_IFNAME=lo
|
|
|
|
# VLLM container image
|
|
#
|
|
# CPU inference image: the OFFICIAL vLLM CI build (multi-arch amd64/arm64), used
|
|
# by the shipped config.yaml / model Quadlets to validate the whole chain WITHOUT
|
|
# a GPU. Its ENTRYPOINT is `vllm serve`, so the model Quadlets only append flags.
|
|
#
|
|
# For a real GPU deployment, swap this for the CUDA image, whose tag is
|
|
# architecture-specific (`docker.io/vllm/vllm-openai:v0.24.0` on amd64,
|
|
# `:v0.24.0-aarch64` on arm64) — Podman cannot template it at generation time.
|
|
# See SPECS.md for the production (GPU) reference.
|
|
COOKBOOK_VLLM_IMAGE=public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.24.0
|
|
|