Introducing Mercury, the first commercial-scale diffusion large language model
Takeaways
1
We are announcing the Mercury family of diffusion large language models (dLLMs), a new generation of LLMs that push the frontier of fast, high-quality text generation.
2
Mercury is up to 10x faster than frontier speed-optimized LLMs. Our models run at over 1000 tokens/sec on NVIDIA H100s, a speed previously possible only using custom chips.
3
A code generation model, Mercury Coder, is available to test in a playground. We offer enterprise clients access to code and generalist models via an API and on-premise deployments.
Our Vision — Next Generation LLMs Powered By Diffusion
Current large language models are autoregressive, meaning that they generate text left to right, one token at a time. Generation is inherently sequential—a token cannot be generated until all the text that comes before it has been generated—and generating each token requires evaluating a neural network with billions of parameters. Frontier LLM companies are betting on test-time computation to increase reasoning and error-correction capabilities, but generating long reasoning traces comes at the price of ballooning inference costs and unusable latency. A paradigm shift is needed to make high-quality AI solutions truly accessible.
Diffusion models provide such a paradigm shift. These models operate with a “coarse-to-fine” generation process, where the output is refined from pure noise over a few “denoising” steps, as illustrated in the video above.
Because diffusion models are not restricted to only considering previous output, they are better at reasoning and at structuring their responses. And because diffusion models can continually refine their outputs, they can correct mistakes and hallucinations. For these reasons, diffusion powers all of the most prominent AI solutions for video, image, and audio generation, including Sora, Midjourney, and Riffusion. However, applications of diffusion to discrete data such as text and code have never been successful. Until now.
Mercury Coder — Frontier Intelligence at 1000+ Tokens per Second
We are excited to announce Mercury Coder, our first publicly available dLLM.
Mercury Coder pushes the frontier of AI capabilities: it is 5-10x faster than the current generation of LLMs, providing high-quality responses at low costs. Our work builds on breakthrough research from our founders–who pioneered the first diffusion models for images—and who co-invented core generative AI techniques such as Direct Preference Optimization, Flash Attention, and Decision Transformers.
A dLLM is a drop-in replacement for a typical autoregressive LLM, supporting all its use cases, including RAG, tool use, and agentic workflows. When prompted with a query, instead of producing the answer one token at a time, the answer is generated in a coarse-to-fine way, as illustrated in the above animation. Improvements are suggested by a neural network – in our case a Transformer model – which is trained on large amounts of data to globally improve the quality of the answer by modifying multiple tokens in parallel.
Mercury Coder is a dLLM specifically optimized for code generation. When evaluated on standard coding benchmarks, Mercury Coder achieves excellent quality across numerous benchmarks, often surpassing the performance of speed-optimized autoregressive models like GPT-4o Mini and Claude 3.5 Haiku while being up to 10x faster.
What sets dLLMs apart is their speed. While even speed-optimized autoregressive models run at most at 200 tokens per second, we can serve Mercury Coder on commodity NVIDIA H100s at speeds of over 1000 tokens per second, a 5x increase. Compared with some frontier models, which can run at less than 50 tokens per second, we offer a more than 20X speedup.
The throughput achieved by dLLMs was previously achievable only using specialized hardware, such as Groq, Cerebras, and SambaNova. Our algorithmic improvements are orthogonal to hardware acceleration and speedups would compound on faster chips.
Speed Comparison; Output Tokens per Second; Coding Workload
We are also excited to report that developers prefer Mercury’s code completions compared to existing code models. When benchmarked on Copilot Arena, Mercury Coder Mini is tied for second place, surpassing the performance of speed-optimized models like GPT-4o Mini and Gemini-1.5-Flash and even of larger models like GPT-4o. At the same time, it is the fastest model, about 4 times faster than GPT-4o Mini.
We invite you to explore the capabilities of our models firsthand in our playground, hosted in partnership with Lambda Labs. Experience Mercury Coder's accuracy in generating high-quality code in a fraction of the time, as demonstrated in the video below.
What’s next?
Mercury Coder is the first in a series of upcoming dLLMs. A model designed for chat applications is in closed beta.
Diffusion language models will unlock a new set of capabilities for LLMs, including:
Improved agents — dLLMs' speed and efficiency make them ideal for agentic applications that require extensive planning and lengthy generation.
Advanced reasoning — dLLMs can leverage error correction to fix hallucinations and improve answers while still thinking in seconds, unlike current autoregressive reasoning models that take minutes.
Controllable generation — dLLMs can edit their output and generate tokens in any order, allowing users to infill text, align outputs with objectives like safety, or produce outputs that reliably conform to user-specified formats.
Edge applications – Given their efficiency, dLLMs excel in resource-constrained environments such as edge deployments on phones and laptops.
Mercury Coder is available
for testing in our playground.