Modernizing an aging Ember test suite

Most of the work we do on a mature Ember app — an LTS hop, an Octane conversion, an Embroider swap, a strangler-route migration — depends on one thing the client usually has not thought about: whether anyone believes the test suite. If a red build is routinely explained away as "that one's flaky," you do not have a safety net. You have a long CI job.

That matters more in Ember than in most ecosystems, because Ember's upgrade story is deliberately mechanical. Deprecation warnings, codemods, and LTS-to-LTS hops are designed to be verified by a suite, not by a QA pass. Remove the suite's credibility and you remove the reason the upgrade path is cheap.

So when an assessment finds an aging suite, we usually sequence test work before the change the client actually asked for. Here is the order we use.

1. Measure before you touch anything

Collect four numbers first, because you will be asked later whether any of this helped.

• Wall-clock CI time for a full run, and how much of it is the test job.
• Failure rate of the same commit re-run: run main five times, count the reds. That is your flake rate.
• Count of tests by style: old-style moduleFor/moduleForComponent, andThen, global test helpers (visit without an import), versus modern module/test with setupRenderingTest and awaited helpers.
• Deprecation output during the test run, deduplicated.

That last one is free and worth more than it sounds. The test run is the cheapest place to inventory deprecations, and the inventory is the upgrade plan.

2. Get to modern test setup, file by file

The target is uncontroversial: module and test from qunit, setupApplicationTest / setupRenderingTest / setupTest from your app's tests/helpers, and @ember/test-helpers for everything async — visit, click, fillIn, settled, render. Every helper returns a promise. Every call gets awaited. The andThen chains and the implicit global test helpers go away.

The conversion is largely mechanical, and ember-qunit-codemod plus the ember-test-helpers-codemod family handle most of it. Do not run them across the whole repo in one commit. We convert one directory at a time — usually tests/unit first because it is the least entangled, then tests/integration, then acceptance tests last — and land each as a reviewable PR. A 900-test suite converted in a single diff is a diff nobody reads.

Two things the codemods will not fix, and that you should fix by hand as you pass through:

• Timing hacks. await this.pauseTest() left behind, arbitrary setTimeout/later waits, wait(500). Replace with settled() or, where you are waiting on a specific condition, waitFor/waitUntil from @ember/test-helpers. An arbitrary sleep is a flake with a delay fuse.
• Container archaeology. Old tests reach into this.container.lookup or a private registry. Move to this.owner.lookup and this.owner.register, which is what modern setup gives you and what will still exist several LTS lines from now.

3. Drain the flakes, do not tolerate them

Flakes are a budget item, not a personality trait of the suite. Work the list in this order, because the causes cluster:

1. Un-awaited async. By far the most common. The codemod era left click(...) without await in places; the test proceeds against a half-rendered DOM and passes or fails by machine speed.
2. Shared state between tests. A service that caches, a stubbed Date, a Mirage seed mutated in place, localStorage not cleared. Symptom: the test passes alone and fails in a full run, or fails only in a particular order. Reproduce with a fixed seed rather than guessing.
3. Real timers and real network. Polling services, setInterval in a component, a fetch that escapes Mirage or @ember/test-waiters registration. Register long-running async with a test waiter so settled() knows to wait for it.
4. Animations and transitions. CSS transitions that the test does not wait out. Disable them in the test environment rather than sprinkling waits.

Quarantining is acceptable as a temporary move — skip the test, open an issue with the failing seed, keep the build green and honest. What is not acceptable is a permanent quarantine folder nobody revisits. We set a date on each skip and treat an aging skip list as a defect.

4. Then make it fast

Only after the suite is trustworthy is speed worth buying. In that order, because a fast unreliable suite just produces wrong answers sooner.

• Partition. ember exam --split=N --parallel across CI containers is the single biggest lever and needs no test changes. Four-way splitting of a twenty-minute suite is a realistic five-minute job.
• Move tests down the pyramid. A large share of the acceptance tests in an eight-year-old app are testing component behaviour through the whole router. Rendering tests cover the same assertions in a fraction of the time. This is slow work; do it opportunistically, when you are in the file anyway.
• Build once, test many. Make CI produce one build artifact and run the split jobs against it rather than rebuilding per container.
• Randomize order permanently. ?order=random with a recorded seed. It surfaces the shared-state class of failure early, rather than during your upgrade.

5. Now do the upgrade

With a green, order-randomized, believable suite, the work the client hired you for gets much duller — which is the goal. Deprecation warnings from the test run become the backlog. Each LTS hop is a dependency bump, a codemod, a green run, a deploy. The Octane conversion happens component by component with the rendering tests as the contract. A strangler-route migration can prove the React route renders the same data because the Ember acceptance test that covered it still runs against the parallel deployment.

When not to do this

Be honest about two cases.

If the app is heading off Ember entirely on a short horizon, and the suite is genuinely thin, modernizing tests you plan to delete is not a good use of money. Write characterization tests around the routes you are actually migrating and leave the rest alone.

And if the suite is so small that it covers almost nothing, "modernize the test suite" is the wrong framing. The work is writing acceptance coverage for the handful of flows the business cannot lose, which is a smaller and more targeted project than a suite-wide conversion.

What this usually costs

For a suite in the several-hundred-test range with a meaningful flake rate, the codemod-and-cleanup pass tends to run a few weeks of one senior engineer's time, spread across PRs, while the app keeps shipping. Flake-draining is the variable part and depends entirely on how much shared state has accumulated. CI partitioning is days, not weeks, and usually pays for itself first.

If you are staring at an upgrade you cannot start because nobody trusts the build, that is a normal reason to call us. An app assessment measures the suite alongside version and addon health and comes back with the sequence and effort range — including the case where the answer is to test less, not more. Or just tell us the version you are on and what it is blocking.