Initial commit: router front-door vLLM stack
Three always-running vLLM services (text TP=2 GPU0+1, ocr + embed on GPU2, sleep mode) behind a FastAPI router that auto-wakes models on request. Tiered idle (sleep 15 min / offload 3 h), depth-aware 503s with Retry-After, persisted wake-intent recovery, admin API on 127.0.0.1:8010. Routine control via vllmctl is pure HTTP — no docker on the request path. Verified: 91 router unit tests + 15-test E2E on real hardware (measurements in CALIBRATION.md; design record in .claude/memory/router-front-door-plan.md). Old nginx stack files removed before git init; design survives in .claude/memory/sleep-mode-implementation-plan.md. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
28
router/Dockerfile
Normal file
28
router/Dockerfile
Normal file
@@ -0,0 +1,28 @@
|
||||
# vllm-router: single process, one event loop, two sockets (public :8000,
|
||||
# admin :8010). No GPU, no model weights -- pure HTTP front door.
|
||||
FROM python:3.12-slim
|
||||
|
||||
# Non-root runtime user.
|
||||
RUN groupadd --system --gid 10001 router \
|
||||
&& useradd --system --uid 10001 --gid router --home-dir /app router
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
# Deps first so code changes don't bust the layer.
|
||||
COPY requirements.txt ./
|
||||
RUN pip install --no-cache-dir -r requirements.txt
|
||||
|
||||
COPY app.py config.py services.py routing.py admin_api.py ./
|
||||
|
||||
# Wake-intent state dir: a named volume seeded from this ownership, so the
|
||||
# non-root runtime user can always write it (bind mounts inherit host uids).
|
||||
RUN mkdir -p /state && chown router:router /state
|
||||
|
||||
USER router
|
||||
EXPOSE 8000 8010
|
||||
|
||||
# Liveness only (a sleeping backend is normal, not an outage).
|
||||
HEALTHCHECK --interval=30s --timeout=5s --start-period=15s --retries=3 \
|
||||
CMD ["python", "-c", "import urllib.request;urllib.request.urlopen('http://127.0.0.1:8000/health', timeout=3)"]
|
||||
|
||||
CMD ["python", "app.py"]
|
||||
Reference in New Issue
Block a user