LLM & Speech Inference Blog

LLM · Infrastructure01 /

Nearly 6× faster with a compile cache. B200 workers in 45 seconds.

An S3 compile cache made H200 startup nearly 6× faster. Dynamo Snapshot brought B200 workers online in about 45 seconds.

~45 sB200 startup · Dynamo Snapshot

Speech · TTS02 /

Reworking VITS from a model invocation into a compiled, batched pipeline for interactive voice calls.

120 msp50 first audio at 150 streams

LLM · Voice serving05 /

How I tuned graph coverage, speculative decoding, and precision for voice-agent traffic with a one-second LLM budget.

500 msE2E p95 · 40 concurrent user streams