JevMade

Sign in
← Back to Jev News Ecosystem & Tools

PostHog Shares How It Uses Jev to Gate Expensive LLM Calls

PostHog has published a thread on how it uses Jev decision models in production, from privacy scrubbing to chart picking, and where they fall short.

3 min read By JevMade Source: PostHog (Jina Yoon) ↗
Cut-out collage of a coral-orange sieve letting a few gold flecks fall toward a tall stack of coins while most scraps drop into a metal tray

PostHog posted a thread on X on 19 November 2026 describing how it uses Jev, TypeSafe AI’s decision model, inside its own products. Nathan LeClaire of TypeSafe AI reshared it the next morning with the line “jev – 445x cheaper and 200x faster”. The thread summarises a longer newsletter piece by Jina Yoon, “Not everything needs an LLM”, published on 9 November.

PostHog’s thread on using decision models like Jev, 19 November 2026.

“Whereas LLMs return free-form and overly confident prose, decision models make model uncertainty explicit.”

Jina Yoon, PostHog, “Not everything needs an LLM” (9 Nov 2026)

Four ways PostHog uses Jev

UseWhat it doesReported result
Gate an LLM callReplay Vision puts each session summary to Jev as two yes/no questions: does it name a real person, and does it show contact details or an ID. A score of 0.4 or higher sends the summary to Gemini for a rewrite and a recheck.98.7% of summaries skip the LLM call. Two narrow questions cut false flags to about a tenth of one broad question.
Route modelsThe PostHog Slack bot asks a decision model which model should do a task, using a list of models with a short note on what each is good for.No figure given in the thread.
Rank resultsAn embedding search pulls session replays for a theme such as “onboarding friction”, then a decision model ranks that list.PostHog says this is cheaper than asking an LLM to rank.
Pick chartsFor opted-in organisations, a decision model reads the SQL and the first three result rows to choose a chart, replacing fixed rules.No figure given in the thread.
Summarised from PostHog’s thread and newsletter post.

Where PostHog says decision models fall short

  • Ranking review findings: on its internal pull request review agent, Jev scored an AUC of 0.52, which PostHog calls inconclusive.
  • Large requests: default limits are 64k tokens per request and 32k per question. TypeSafe’s opt-in long-context mode raises both to 262k, but TypeSafe warns it is slower and less accurate.
  • Multi-step judgements: asking whether a session is worth watching, and why, did not work well, so PostHog moved to embeddings.
  • Open-ended work: PostHog’s advice is to keep an LLM as the default.

For jobs that need more context but must stay cheap, PostHog’s in-house research team built Jeeves, a decision model that reasons before deciding when it is unsure. JevMade has a listing for Jeeves and an earlier news post on its release.

How to read the 444.6x and 193.6x figures

SourceClaimBasis
TypeSafe AIUp to 444.6x cheaper and 193.6x fasterFour workflows written by TypeSafe’s own model capabilities team, scored against an average of two frontier models. TypeSafe says these sit at the high end of real-world gains.
Fractional ResearchRoughly 2 to 5x faster, 5 to 50x cheaper per decisionIndependent tests, at small-LLM accuracy.
PostHog98.7% of LLM calls avoided in one pipelineProduction figure from Replay Vision. This is a saving from skipping the call, not a per-call speed or price comparison.
PostHog’s footnote attributes the multipliers to TypeSafe AI; they are not PostHog’s own measurements.

Sources: PostHog’s thread on X, PostHog’s newsletter post “Not everything needs an LLM”, TypeSafe AI’s Jev launch post, Fractional Research’s Jev test, and Nathan LeClaire’s post on X.

Keep what you save

Sign in to keep bookmarks and private notes on every device. We email you a link; there is no password, and the same link creates your account.