Frozen MTP makes on-device AI faster

TL;DR: Frozen MTP attaches a trained drafting head to an existing model, so mobile AI can predict multiple tokens per pass without rebuilding the base model. Google reported 50%+ faster Gemini Nano token generation on Pixel 9, up to 130 MB saved per instance, and identical final outputs.

Frozen MTP matters because Google says it made Gemini Nano token generation 50% or more faster on Pixel 9 while saving up to 130 MB per instance. Same answer. Less waiting. According to Google Research, the final output stays bit-for-bit identical to the base model.

That last phrase is the big part. If you run AI inside a phone, browser, kiosk, warehouse scanner, medical tablet, or call-center laptop, every extra megabyte and every extra second shows up as user pain. I’ve seen teams ship useful AI features, then quietly disable them because latency felt worse than the manual workflow.

Cost pressure is real, too. According to McKinsey’s 2026 Global Survey, one in five organizations are limiting AI use because of operating costs, including token costs. Faster decoding won’t fix bad product design. But it can change the math for private, local, low-latency AI.

What is Frozen MTP and why does it matter?

Frozen MTP is a way to speed up an existing language model by adding a small multi-token prediction head while keeping the main model weights unchanged. The base model still verifies the drafted tokens, so wrong guesses are discarded before users see them. That’s why this is interesting for production teams: it improves inference speed without reopening the full training, safety, and release cycle.

According to Google Research, frozen MTP was rolled out to Pixel 9 and Pixel 10 for production features such as AI Notification Summaries and Proofread, with final output kept bit-for-bit identical to Gemini Nano.

Eden Cohen, Research Product Manager at Google Platforms and Devices, and Michelle Ramanovich, Research Manager at Google Platforms and Devices, state: “Retrofitting Multi-Token Prediction onto frozen production models.” That wording is plain, but the implication is huge. You don’t have to rebuild the house to make the hallway less crowded.

How does Frozen MTP work on Pixel devices?

Standard LLM decoding is slow because the model usually predicts one token, verifies that state, then repeats. Frozen MTP changes the pattern. The main model runs as usual, while an attached drafting head looks at the model’s internal representations and proposes more than one future token. The main model then checks those candidates in parallel. If they match, generation jumps forward.

According to Google Research, production workloads with frozen MTP predicted nearly two additional tokens per inference pass on average, and predictable structures such as smart replies saw up to 55% better token acceptance.

Here’s the catch. This isn’t magic compression, and it doesn’t make every prompt twice as fast. It works best when the future text has enough predictable shape: summaries, rewrites, replies, proofreading, form text, and guided agents. Our team of 10+ specialists has seen the same pattern in production ML systems. Bounded outputs usually win first.

How do the benchmarks compare?

The fastest way to understand Frozen MTP is to compare it with related speculative decoding methods. Google’s Pixel work is production-focused and memory-sensitive. Meta’s multi-token prediction research, EAGLE, and Medusa show the broader research trend: draft more tokens, verify safely, and reduce decoding latency without changing the visible answer distribution.

According to Google AI Edge LiteRT-LM docs, Gemma 4 MTP delivers more than 2x faster decode speeds on mobile GPUs with zero quality degradation, while Google’s Pixel implementation saved up to 130 MB per instance versus a standalone drafter.

Yaniv Leviathan, Matan Kalman, and Yossi Matias, Researchers at Google Research, state: “Speculative decoding has proven to be an effective technique for faster and cheaper inference.” That line explains why this family of methods keeps returning.

Where should companies apply Frozen MTP ideas first?

Companies should test Frozen MTP-style acceleration first in workflows where outputs are short, repetitive, private, and latency-sensitive. Think customer support drafts, field-service notes, financial summaries, inbox triage, compliance rewrites, and mobile data-entry assistants. These jobs are not glamorous. They’re valuable because users repeat them all day.

According to Counterpoint Research, GenAI-capable smartphones are forecast to reach 45% of global shipments in 2026, up from 36% in 2025 and projected to hit 52% in 2027.

When we implemented a RAG chatbot for a fintech client, support tickets dropped 40% in 3 months. That wasn’t only because the model answered well. It worked because retrieval, response time, and escalation rules matched the daily support queue. After 50+ projects, we’ve learned that speed is a product feature, not just an infrastructure metric.

Here’s a small Python sketch teams can use to reason about accepted draft tokens before building anything serious:

from statistics import mean

def estimate_speedup(baseline_passes, drafted_tokens_per_pass, acceptance_rate):
    accepted_extra = drafted_tokens_per_pass * acceptance_rate
    effective_tokens_per_pass = 1 + accepted_extra
    mtp_passes = baseline_passes / effective_tokens_per_pass
    return baseline_passes / mtp_passes

workloads = {
    "proofread": (120, 2, 0.70),
    "smart_reply": (40, 3, 0.80),
    "open_chat": (180, 2, 0.35),
}

for name, args in workloads.items():
    print(name, round(estimate_speedup(*args), 2))

This model is crude. I still recommend it because it forces the right question early: how many drafted tokens will your workload actually accept?

Top 5 business lessons from Frozen MTP

Frozen MTP is not only a Google phone story. It points to a broader shift in AI delivery: teams will increasingly improve model serving around the base model instead of rebuilding the model for every feature. That matters for regulated industries, where model changes trigger review, QA, documentation, and sometimes legal signoff.

According to Stanford HAI’s AI Index 2026, global corporate AI investment reached $581.7 billion in 2025, up 130% from the prior year, making inference efficiency a board-level cost and product issue.

1. Latency decides adoption

Users forgive a clever demo. They don’t forgive a slow daily tool. If an on-device proofreader takes too long, people turn it off, even when the answer is good.

2. Frozen backbones reduce release risk

Keeping the main model unchanged makes testing easier. It doesn’t remove QA, but it narrows the blast radius compared with retraining or replacing the model.

3. Memory is a product constraint

Counterpoint Research states: “Memory will remain a key factor” for GenAI smartphone expansion. That’s not an abstract hardware note. It decides which features run locally, which fall back to cloud, and which get cut.

4. Structured tasks are the early winners

Smart replies, summaries, proofreading, and form filling have patterns. Open-ended creative chat is harder because the next token path branches more often.

5. Architecture beats model size in many products

Bigger models can help, but architecture often decides whether a feature feels usable. We tested this with document workflows, support agents, and content systems. The best outcome usually came from model choice plus retrieval, caching, batching, guardrails, and user interface timing.

Can Frozen MTP replace model tuning?

Frozen MTP can’t replace model tuning when the model doesn’t know the domain, follows instructions poorly, or fails safety requirements. It speeds up decoding. It does not teach missing policy, missing vocabulary, or missing business logic. That distinction matters because some teams treat inference acceleration as a cure for every AI product issue.

According to Eurostat, 19.95% of EU enterprises used AI in 2025, up 6.47 percentage points from 2024, which means many firms are now hitting production issues beyond model demos.

When we implemented a document processing pipeline for a legal client, automation reached 80% of contract review and saved 120 hours per month. The speed mattered, yes. But the real work was schema design, exception handling, review queues, and confidence thresholds. Honest limitation: Frozen MTP won’t rescue a workflow with messy inputs, vague acceptance rules, or no owner for edge cases.

For companies building on Gemini, that means the smart path is practical: decide which parts can run on device, which need cloud models, which need RAG, and which need human review. Yaitec’s Gemini for companies work focuses on those production choices, from Google Workspace automation to private assistants and agent workflows. For a specific use case, you can also contact us.

Conclusion: faster edge AI without a rebuild

Frozen MTP shows where AI product engineering is heading: less drama around replacing models, more disciplined work around making deployed models faster, cheaper, and easier to run close to the user. The strongest signal is Google’s production rollout, not just a lab score, because Pixel features such as Notification Summaries and Proofread put the technique in normal hands.

According to IDC press coverage, more than 370 million GenAI smartphones were expected to ship globally in 2025, about 30% of the market, and Anthony Scarsella, Research Director at IDC, states that “On-device GenAI capabilities” are expected to become standard in mid-range phones.

After 50+ projects across fintech, healthtech, e-commerce, legal, and marketing, we’ve learned that AI wins when it fits the user’s real environment. Sometimes that means LangChain, LangGraph, CrewAI, or Agno. Sometimes it means a smaller model. And sometimes, as Frozen MTP shows, it means keeping the model frozen while making every generated token arrive sooner.

Sources

• Google Research — retrieved 2026-09-01
• McKinsey & Company — retrieved 2026-09-01
• Stanford — retrieved 2026-09-01