AI Pulse

OpenAI's Flagship Model Now Runs at 750 Tokens Per Second. Alibaba Dropped a 27B Local Rival the Next Day. The August 17 Pulse.

By Felix Maru · August 17, 2026 · 7 min read

The inference speed race now has two fronts. OpenAI and Cerebras announced this week that GPT-5.6 Sol, OpenAI's most capable model, can now run at up to 750 output tokens per second in a new Ultrafast API tier. The next day, Alibaba's Tongyi Lab released Qwen3.8-27B, a 27-billion parameter open-weight model under Apache 2.0 that posts competitive agentic coding scores against models several times its size. Both happened within 24 hours of each other. Neither is slow news.

What 750 Tokens Per Second Actually Means

On August 13, Cerebras and OpenAI announced the Ultrafast tier: GPT-5.6 Sol running at up to 750 output tokens per second, roughly 14 times faster than standard API processing. The speed comes from Cerebras's wafer-scale chip architecture. Standard GPU inference streams model weights from off-chip memory between each token. Cerebras keeps those weights in roughly 44 gigabytes of on-chip SRAM per wafer-scale chip, eliminating the memory bottleneck that limits generation speed.

To put that in context: the average reader moves through text at somewhere around 200 to 250 words per minute, which is roughly 4 to 5 tokens per second. At 750 tps, the model generates output more than 150 times faster than a person can read it. The bottleneck in a live AI conversation has shifted. It is no longer generation time. It is network latency, UI rendering, and the time the model spends actually reasoning through the problem.

The OpenAI blog confirmed a limited preview for a select group of API customers, with access expanding as Cerebras capacity grows. No public pricing has been announced. This is not something you can wire into production today. But understanding what it enables is worth doing now, because the integration patterns that take advantage of real-time generation are structurally different from ones designed for async processing.

Why Customer Support Is One of the Named Use Cases

OpenAI specifically cited customer support alongside coding, financial research, commerce, and incident response as the primary use cases for Ultrafast. That is not a generic list. These are workflows where latency changes the user experience, not just the speed metric on a benchmark page.

The specific scenario OpenAI described (paraphrasing from the announcement): a customer-service agent searching multiple internal systems while the customer is still speaking. At 750 tps, the model can fully process a knowledge base article, cross-reference account data, generate a relevant draft response, and return it within the natural pause in a voice conversation. That is a different product from what most teams are running today, where AI assistance arrives after the customer has already finished speaking and the agent has already started formulating their own answer.

For teams running AI-assisted chat or ticketing, today's bottleneck in a multi-step AI workflow is not generation speed at standard rates. It is the sequence of handoffs: customer sends message, AI classifies intent, AI queries a knowledge base or CRM, AI drafts a reply, human agent reviews. Each step adds latency. Ultrafast compresses the generation steps in that chain. What it does not compress is the judgment steps, and those remain where your human agents earn their value.

The practical action for most teams right now is not to redesign your integration around Ultrafast. It is to watch for when pricing and availability become public, and in the meantime, audit whether your current AI workflows are even structured to benefit from lower latency. If you are running async triage and batch reply drafting, you will not notice much difference. If you are building toward real-time AI in live chat or voice, the architecture question becomes important.

Qwen3.8-27B: The Open-Weight Side of the Same Race

On August 14, Alibaba's Tongyi Lab released Qwen3.8-27B under Apache 2.0. The model takes 27.78 billion parameters, accepts text, images, and video as input, carries a 262,144-token context window, and requires roughly 24 gigabytes of VRAM to run locally. That puts it in reach of a high-end professional GPU or a purpose-built local inference server, not a laptop, but not a data center either.

The benchmark numbers Alibaba published in the model card are notable for the 27B class. On SWE-Bench Pro for agentic coding, Qwen3.8-27B scores around 61.7%. On OSWorld for computer-use tasks, it reaches 84.3%. On Terminal-Bench for terminal automation, it hits 73.0%. These are self-reported figures from Alibaba and have not all been independently verified at the time of writing. With that caveat: the agentic coding and computer-use numbers place this model meaningfully ahead of prior open-weight models of comparable parameter count.

The practical significance is not that Qwen3.8-27B replaces frontier API models. It is that teams with data-residency or contractual constraints now have a more capable local option. When you cannot send ticket content to an external API because of jurisdiction rules or customer agreements, a model you run entirely inside your own infrastructure has become an increasingly real option over the past few months. Meta's Muse Glimmer (covered in the August 11 pulse) was one signal. Qwen3.8-27B raises the ceiling on what a self-hosted model can do on agentic tasks.

Google's Eval Tools Go GA: Measurement Catches Up to Deployment

Also on August 13, Google announced that agent and model evaluations in the Gemini Enterprise Agent Platform are generally available. The release includes 20-plus pre-built and custom metrics, adaptive rubrics, simulation tools for testing agent behavior before production deployment, and built-in online monitoring. The platform also supports multi-turn autoraters that evaluate the logic of a full conversation, not just a single response in isolation.

This is specific to Google's Enterprise Agent Platform. But the signal is broader: systematic measurement of AI agent quality at production scale is arriving as a product, not just a research problem.

Most teams running AI agents in support today do not have a principled way to evaluate whether those agents are making the right decisions on a per-conversation basis. They monitor CSAT scores, escalation rates, and resolution times. What they do not have is a continuous, per-conversation quality signal running against a consistent definition of what "good" looks like.

Multi-turn autoraters change that. If you evaluate a conversation as a unit, you can catch failure modes that single-response metrics miss. An agent that gave technically correct answers to each question but never actually resolved the customer's underlying problem. A handoff to a human that happened three messages too late because the agent kept attempting resolution when it should have escalated earlier. These are real patterns in production AI support, and aggregate metrics rarely surface them clearly enough to act on.

Whether you are running on Google's platform or another, the questions this tooling asks are the right ones to be asking about your own setup.

The Read Across All Three Stories

Three things happened in 48 hours: a frontier model now runs fast enough to respond within a spoken conversation, a locally-hostable open-weight model reached competitive scores on agentic tasks, and a major platform shipped production-grade evaluation tooling for AI agents. Each is useful individually. Together, they describe a layer of AI infrastructure that is maturing from "does this work at all" to "how do we run it well."

For support teams, the practical read is this: faster inference and more capable local models expand the surface area where AI can operate. Expand it faster than your measurement capability can keep up and you create risk. Know what your AI is getting right before you give it more real-time contact with customers.

Faster AI gives your human agents better information in less time. It does not replace the empathy and judgment that built trust with those customers in the first place.

Sources

If something here changes how you are thinking about your AI stack or your support automation setup, reach out. Happy to look at what you are running and help you think through the tradeoffs.