bing
80eef4ce6a
Three always-running vLLM services (text TP=2 GPU0+1, ocr + embed on GPU2, sleep mode) behind a FastAPI router that auto-wakes models on request. Tiered idle (sleep 15 min / offload 3 h), depth-aware 503s with Retry-After, persisted wake-intent recovery, admin API on 127.0.0.1:8010. Routine control via vllmctl is pure HTTP — no docker on the request path. Verified: 91 router unit tests + 15-test E2E on real hardware (measurements in CALIBRATION.md; design record in .claude/memory/router-front-door-plan.md). Old nginx stack files removed before git init; design survives in .claude/memory/sleep-mode-implementation-plan.md. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
29 lines
1017 B
Docker
29 lines
1017 B
Docker
# vllm-router: single process, one event loop, two sockets (public :8000,
|
|
# admin :8010). No GPU, no model weights -- pure HTTP front door.
|
|
FROM python:3.12-slim
|
|
|
|
# Non-root runtime user.
|
|
RUN groupadd --system --gid 10001 router \
|
|
&& useradd --system --uid 10001 --gid router --home-dir /app router
|
|
|
|
WORKDIR /app
|
|
|
|
# Deps first so code changes don't bust the layer.
|
|
COPY requirements.txt ./
|
|
RUN pip install --no-cache-dir -r requirements.txt
|
|
|
|
COPY app.py config.py services.py routing.py admin_api.py ./
|
|
|
|
# Wake-intent state dir: a named volume seeded from this ownership, so the
|
|
# non-root runtime user can always write it (bind mounts inherit host uids).
|
|
RUN mkdir -p /state && chown router:router /state
|
|
|
|
USER router
|
|
EXPOSE 8000 8010
|
|
|
|
# Liveness only (a sleeping backend is normal, not an outage).
|
|
HEALTHCHECK --interval=30s --timeout=5s --start-period=15s --retries=3 \
|
|
CMD ["python", "-c", "import urllib.request;urllib.request.urlopen('http://127.0.0.1:8000/health', timeout=3)"]
|
|
|
|
CMD ["python", "app.py"]
|