Before OpenAI evals go read-only: how to preserve your testing workflow

A platform shutdown is disruptive when your team has made it part of the release rhythm. Eval runs often contain more than a test result: they carry prompt variants, grading rules, baseline outputs, failure thresholds and the team’s shared memory of what “good” looks like. If those pieces are not documented, migration becomes guesswork rather than a controlled change.

OpenAI’s documentation points teams toward Datasets for new evaluation workflows and refers developers to a migration path via Promptfoo. That tells you something important: the underlying practice of evaluation still matters, but the hosted workflow is changing. The goal is not to preserve the old interface exactly. It is to preserve coverage, comparability and decision-making.

0 of 5 complete

This step sounds administrative, but it is where migration risk becomes visible. A team may believe it has “one eval”, when in practice it has a mixture of reusable test cases, informal manual checks and approval habits. Those are different assets. They do not all move in the same way.

When teams migrate tools, they often try to rebuild the whole old workflow. That can waste time. A better approach is to recreate the smallest set of checks that protect production decisions first.

1. 1

   Rebuild the core test set

   Keep the examples that cover the most common and the most risky user journeys. Do not wait for a perfect taxonomy.

2. 2

   Rebuild the scoring rules

   If you use custom graders or rubric logic, document them clearly and test whether they behave the same in the new environment.

3. 3

   Rebuild comparison views

   You need a way to compare new model or prompt outputs against a baseline so regressions are obvious, even if the interface changes.

4. 4

   Rebuild reporting

   Decide what evidence a release owner needs to see before a change is approved. That may be a short summary, not a full dashboard.

The biggest danger is silent drift. A migration can look complete if the new tool runs the same examples, but still produce different decisions because the scoring rules changed slightly, the baseline is missing, or reviewers no longer see the same failure patterns.

Another risk is over-automating the new setup too soon. OpenAI’s broader model-optimisation guidance treats evals as part of the process for measuring and tuning model performance during model changes. That is useful, but it does not mean every decision should be left to automated scoring alone. For user-facing or consequential actions, a human review step still matters.

0 of 6 complete

This staged approach helps because it separates continuity from improvement. You do not need the new workflow to be identical on day one. You need it to be trustworthy enough for the decisions you already make. If the new setup is better organised, that is a bonus, not the starting requirement.

You do not need outside help for every migration. But it can be useful when your evals are tied to a production assistant, a customer-facing workflow or a release process that other teams depend on. In those cases, the real task is not only moving tooling. It is preserving operational control.

Support is often valuable when you need to recreate evaluation coverage in a different environment, decide which checks should stay human-reviewed, or turn a set of informal prompts and comments into a proper testing workflow. For teams that also need the surrounding product, cloud or integration work rebuilt, this is usually part of a wider delivery change rather than a standalone script swap.

The right outcome is simple: a testing setup that still tells you whether the model is behaving acceptably, even after the platform underneath it has changed.

If you act now, the migration is mainly a question of discipline. If you leave it late, the same change becomes a rescue job. The difference is whether your evaluation workflow is something you own, or something the platform owns for you.