OpenAI’s engineering post How OpenAI’s Performance Factory keeps ChatGPT fast describes an autonomous system that finds, root-causes and fixes performance regressions in ChatGPT. In just over three months its Regression Workflow tracked more than 1,700 investigations and root-caused over 600 of them, about 17 automated investigations a day.
The post publishes no source code, so this is a reading of the architecture and the reasoning behind it, not a code walkthrough. Everything factual below comes from OpenAI’s article. Where I add interpretation, I say so.
The workflow in motion
From manual work to a factory
OpenAI describes six stages. Read them as a ladder where each rung removes one place where a human was acting as glue.
- Pre-Codex. Engineers read metrics, logs and traces by hand. Repeated problems, such as first-token latency regressions, produced more than 30 performance tools.
- Codex as a partner. Engineers ask “why did this get slower?” and steer. Codex searches code, finds metrics and follows calls across services, but a person still picks the tools.
-
Reusable skills. The team’s investigation process became a shared skill,
perf-forensics. It lists the tools, when to use them, and a playbook: when did it start, which requests are affected, could a deployment or config change explain it. Evals built from past regressions tested it. - The human proxy. Engineers noticed they were copying alerts into Codex and pasting findings back into Slack. That is the signal that a human is now a router.
- Automated investigations. An alert starts a cloud Codex session and the result lands in a Slack thread. It still ends there: someone must fix and verify, and finding which threads matter is its own chore.
- The factory. A system tracks what each task needs next and assigns an agent when a step is ready.
The pattern worth stealing is stage 4. When you catch yourself acting as a proxy between a trigger and an agent, that proxy is the next thing to build.
Architecture
The factory has four parts: a task service with configurable workflows, a dispatch loop, Codex in the cloud, and event subscriptions.
Task service. Engineers define a workflow with configuration files plus Markdown prompts. The configuration lists the states a task can pass through, the allowed transitions, and which steps need an engineer. A separate prompt tells the agent what to do in each state. Every piece of work is a task, visible in a UI, so people can follow it or hand it to a teammate.
Dispatch loop. It periodically checks which tasks are ready for their next step and starts a Codex turn with the instructions for that task’s current state. The agent writes its progress on the task and moves it forward if the step is done.
Codex in the cloud. An internal API starts a Codex turn inside OpenAI’s network, with the repositories and observability tools an engineer would use: metrics, logs, traces, CPU profiles and memory snapshots. The agent runs under a service identity, so permissions can be scoped and its actions are attributable. Moving off laptops allows parallel runs.
Subscriptions. Some work waits on a person or a deployment, and an agent should not burn a session while it waits. A task can subscribe to events such as a GitHub review comment or a failed CI check. When a matching event arrives, the task becomes ready and the dispatch loop assigns an agent again.
One regression through the state machine
The Regression Workflow moves a task through four named stages: needs investigation, root caused, fix proposed, resolved.
The recovery check has three outcomes. If performance is back at baseline, the task resolves. If it only partly recovered, the original regression resolves and a new task opens for the remainder, which may have a different cause. If nothing improved, the agent records what happened and sends it back for investigation.
The design ideas worth keeping
The facts in each idea come from the article. The framing and the conclusions I draw are my interpretation.
1. State lives in the task, not in the agent
The task records status and key findings outside any one agent’s context. That is why work can move between agents and engineers, pause for a code review, and resume after a deployment. My interpretation: that makes individual agent turns replaceable, and the task is the memory. It resembles how durable workflow engines work, applied to LLM turns.
2. Waiting is an event, not a running process
An agent that polls for a review comment wastes a session and a budget. Subscriptions turn “wait for a human” into a state plus a wake-up condition. My interpretation: this is what lets one factory serve long-lived work such as a fix that takes a day to review.
3. Split big jobs into focused turns, and add an adversary
Early agents stopped at the first plausible explanation, and some were wrong. OpenAI added a separate Codex session whose job is to disprove the diagnosis by rechecking assumptions and testing counterfactuals: did similar requests not exposed to the rollout stay healthy, and did the metric recover when the change was rolled back?
Their example: an agent blamed image-download errors for failing conversations. The reviewer saw that the file-lookup service had already recovered while image requests were still failing, so the diagnosis could not explain the ongoing failures, and the investigation reopened. In a manual review of 100 root-cause suggestions, 70% were graded accurate and 30% needed more work or did not establish causality. Splitting investigation, challenge and fix into separate turns is intended to raise that number, though the article gives no before-and-after result.
4. Fixing is harder than finding
Agent-proposed fixes can be more complicated than needed or introduce new issues, so fixes get an adversarial review too. Some fixes need an architectural call: a cache may speed things up and also serve stale data. Those stay with an engineer who knows the service. The human step is part of the design, not a gap to be closed by default.
5. Spend human attention on decisions, not on reconstruction
A raw investigation is a wall of text. Each task has a generated report: regression summary, impact, root cause, proposed fix, with metric time series, CPU flame graphs and traces so the evidence is one click away. The first UI simply listed findings and left prioritisation to people. The workflow now links related incidents, groups them by the metrics teams care about, and ranks them by estimated impact.
6. Evaluate the agent and the handoff separately
Changing a skill or upgrading a model can improve some cases and hurt others, so OpenAI tests two things independently.
- Investigation quality. Past production incidents become a backtest suite. Each case starts a fresh investigator with only the original signal, stripped of hints. The agent may read code and query telemetry within limits, but not saved memories, the incident write-up or prior runs, and the transcript is checked afterwards for leaked answers. Scoring covers when the problem began, how large it was, which requests were affected, and how the agent ranks causes and states uncertainty. Right timing with the wrong deployment still counts as a miss.
- Handoff quality. Saved task snapshots are graded by a separate grader on whether an engineer unfamiliar with the system could understand what happened and why it matters, whether useful charts survived, and whether the next step is supported by the evidence. Snapshots do not depend on telemetry that expires, so they backtest well.
7. Monitor the agents, then let the factory repair itself
A broken profiler or a missing permission can quietly cripple an investigation. A daily agent scans recent runs for tool failures and access errors. When the same error blocks several investigations it opens a task, an agent proposes a fix, an engineer reviews it, and evals confirm that agents get past the step that used to block them.
A sketch of the shape
OpenAI shows no config, so this is my illustrative reconstruction of what “states, transitions and human steps” could look like. It is not their format.
workflow: regression
states:
needs_investigation: { agent: investigate, next: root_caused }
root_caused: { agent: challenge, next: fix_proposed, on_reject: needs_investigation }
fix_proposed: { human: review_and_deploy, wait_for: [pr_comment, deploy], next: verify }
verify: { agent: verify_recovery, next: resolved, on_partial: new_task, on_none: needs_investigation }
My reading is that the engineer writes very little: states, who acts in each, which events wake it, and Markdown prompts.
Where I would push back
- The 70% figure is from a manual review of 100 suggestions, and the article does not say how accuracy changed once the adversarial review was added. The architecture claim is more convincing than the quality claim.
- Nothing is published about how many proposed fixes engineers merge unchanged. OpenAI says it is building evals by comparing proposals with merged changes.
- Backtests lean on telemetry that expires, which the authors flag themselves. A long-lived eval suite needs frozen evidence.
What to take away
- Build the loop, not the agent: state machine, dispatcher, execution environment, event wake-ups. The model is the replaceable part.
- Put memory in the task so agents and humans can swap in and out.
- Give every important claim a separate adversary session with counterfactual tests.
- Keep humans on the decisions that carry trade-offs, and spend the engineering effort on making their review fast.
- Evaluate the agent and the report it hands over as two different products.
- Monitor the monitors. Agents break quietly.
The wider claim in the article is that this is a reusable agent factory pattern: tasks move through explicit states, agents advance them, reviewers challenge conclusions, engineers decide when judgment is needed. Other workflows reuse the machinery with different states, prompts and events, and one workflow can open tasks in another.
Source: OpenAI, How OpenAI’s Performance Factory keeps ChatGPT fast.
Building agent workflows for your own team and want a second pair of eyes on the design? Get in touch.
Click to load Disqus comments