A deployment workflow for a Jev-compatible HTTP API backed by SGLang on Modal, with temporary inference checks, stable deployment, and persistent model and compilation caches.
Original by Eric ZhangIntegrationsGitHub READMESource reviewed
Before you dive in
What you’ll find in the original
Use the lightweight local environment for API tooling and tests while keeping GPU dependencies in the Modal image.
Run the temporary Modal command for real inference checks before publishing a stable endpoint.
Persist model files and SGLang or Triton caches in a Modal Volume, while expecting CUDA graph capture during later starts.
Worth knowing
Requires Modal credentials and paid GPU resources. Compatibility describes the HTTP shape; quality, cold-start time, and cost depend on the selected model and Modal configuration.