bing
80eef4ce6a
Three always-running vLLM services (text TP=2 GPU0+1, ocr + embed on GPU2, sleep mode) behind a FastAPI router that auto-wakes models on request. Tiered idle (sleep 15 min / offload 3 h), depth-aware 503s with Retry-After, persisted wake-intent recovery, admin API on 127.0.0.1:8010. Routine control via vllmctl is pure HTTP — no docker on the request path. Verified: 91 router unit tests + 15-test E2E on real hardware (measurements in CALIBRATION.md; design record in .claude/memory/router-front-door-plan.md). Old nginx stack files removed before git init; design survives in .claude/memory/sleep-mode-implementation-plan.md. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
11 lines
423 B
Plaintext
11 lines
423 B
Plaintext
# Pinned (plan section 6). Direct dependencies only; the transitive set
|
|
# (starlette, pydantic, anyio, ...) is resolved by pip at build time and
|
|
# recorded by the image build.
|
|
#
|
|
# fastapi == 0.141.1 (current stable, verified with starlette 1.6)
|
|
# uvicorn == 0.52.3 (two Server instances on one asyncio loop)
|
|
# httpx == 0.28.1 (AsyncClient streaming proxy)
|
|
fastapi==0.141.1
|
|
uvicorn==0.52.3
|
|
httpx==0.28.1
|