PostHog Shares How It Uses Jev to Gate Expensive LLM Calls
PostHog has published a thread on how it uses Jev decision models in production, from privacy scrubbing to chart picking, and where they fall short.
PostHog posted a thread on X on 19 November 2026 describing how it uses Jev, TypeSafe AI’s decision model, inside its own products. Nathan LeClaire of TypeSafe AI reshared it the next morning with the line “jev – 445x cheaper and 200x faster”. The thread summarises a longer newsletter piece by Jina Yoon, “Not everything needs an LLM”, published on 9 November.
“Whereas LLMs return free-form and overly confident prose, decision models make model uncertainty explicit.”
Jina Yoon, PostHog, “Not everything needs an LLM” (9 Nov 2026)
Four ways PostHog uses Jev
| Use | What it does | Reported result |
|---|---|---|
| Gate an LLM call | Replay Vision puts each session summary to Jev as two yes/no questions: does it name a real person, and does it show contact details or an ID. A score of 0.4 or higher sends the summary to Gemini for a rewrite and a recheck. | 98.7% of summaries skip the LLM call. Two narrow questions cut false flags to about a tenth of one broad question. |
| Route models | The PostHog Slack bot asks a decision model which model should do a task, using a list of models with a short note on what each is good for. | No figure given in the thread. |
| Rank results | An embedding search pulls session replays for a theme such as “onboarding friction”, then a decision model ranks that list. | PostHog says this is cheaper than asking an LLM to rank. |
| Pick charts | For opted-in organisations, a decision model reads the SQL and the first three result rows to choose a chart, replacing fixed rules. | No figure given in the thread. |
Where PostHog says decision models fall short
- Ranking review findings: on its internal pull request review agent, Jev scored an AUC of 0.52, which PostHog calls inconclusive.
- Large requests: default limits are 64k tokens per request and 32k per question. TypeSafe’s opt-in long-context mode raises both to 262k, but TypeSafe warns it is slower and less accurate.
- Multi-step judgements: asking whether a session is worth watching, and why, did not work well, so PostHog moved to embeddings.
- Open-ended work: PostHog’s advice is to keep an LLM as the default.
For jobs that need more context but must stay cheap, PostHog’s in-house research team built Jeeves, a decision model that reasons before deciding when it is unsure. JevMade has a listing for Jeeves and an earlier news post on its release.
How to read the 444.6x and 193.6x figures
| Source | Claim | Basis |
|---|---|---|
| TypeSafe AI | Up to 444.6x cheaper and 193.6x faster | Four workflows written by TypeSafe’s own model capabilities team, scored against an average of two frontier models. TypeSafe says these sit at the high end of real-world gains. |
| Fractional Research | Roughly 2 to 5x faster, 5 to 50x cheaper per decision | Independent tests, at small-LLM accuracy. |
| PostHog | 98.7% of LLM calls avoided in one pipeline | Production figure from Replay Vision. This is a saving from skipping the call, not a per-call speed or price comparison. |
Sources: PostHog’s thread on X, PostHog’s newsletter post “Not everything needs an LLM”, TypeSafe AI’s Jev launch post, Fractional Research’s Jev test, and Nathan LeClaire’s post on X.