Parity Suite 3.0.0: measuring the wrong thing

Since the board started it has published one number per emulator: a pass rate over the tests that emulator attempted. Skipped tests left the sum entirely.

Which means the number could move without the engine changing a line. In July one emulator gained 8.4 points having shipped nothing at all.

3.0.0 replaces that number. Two figures instead of one, a letter grade on every target, and the suite is up to 1,054 tests with the first coverage of DynamoDB's new vector search.

Two figures, never added together

Coverage is how much of DynamoDB the emulator supports. Divergence is how much of it the emulator gets wrong. Both are measured over the whole suite, so neither can hide inside the other.

They're reported apart because a skip and a fail are different problems. An operation your emulator declines is something you find in minutes and plan around. One it gets quietly wrong is something you find in production.

The July case was Dynalite, and it works as an example precisely because Dynalite did nothing. It passed 673 tests before and 673 after. Two things happened, both of them mine.

First I grew the suite by 44 PartiQL tests Dynalite doesn't implement. Every one landed as a skip, and the old number didn't move at all, because skips left the sum entirely. The job got 44 tests bigger and the score couldn't see it.

Then I improved the probes that spot Dynalite doesn't do transactions, which turned 88 failing tests into skips. That the old number could see. It climbed from 76.0% to 84.3%, and I published an 8.4 point gain for an engine nobody had touched.

Under the new figures both show up. The transaction reclassification drops divergence and coverage by 8.8 points each, together, measured over the suite as it stood in July. The suite growth drops coverage while divergence barely moves, which is what "the target didn't change, its job got bigger" should look like.

Nothing was re-run for this. No pass, fail or skip changed, so a figure that looks different from last month is the same measurement described better.

Letter grades

Two figures are harder to skim than one, so every target wears a letter as well.

Divergence sets it: A under 5%, B under 15%, C under 25%, D under 35%, F beyond. Coverage can only pull it down. A third of whatever a target leaves unimplemented is added to its divergence before the bands are read, so an engine covering the whole suite is graded on divergence alone.

A+ is zero divergence at full coverage.

The criteria are versioned and dated in the methodology. Moving a threshold regrades engines whose results never changed, and that should be a visible act rather than a quiet edit.

The board today

Dynoxide is mine, so a board I author has my own engine at the top of it. The disclosure sits on the card and on the methodology page, and the oracle is derived at build time rather than by me.

Those are the 12th August run. The board re-runs against live AWS on a schedule, so the live figures are the ones to trust if these have moved.

Dynoxide had zero divergence before this release. It's at 0.9% now, and all ten of its failures are in tests/tier1/putItem/indexConsumedCapacity.test.ts, a file this release added after spaceemotion filed an issue pointing out that the suite wasn't measuring index write costs at all. The tests caught ten Tier 1 failures in my own engine on their first run, and the board published them the same day. Tier 1 is the tier the suite describes as what most DynamoDB users rely on, so that's the worst place to find them.

Real DynamoDB reads baseline rather than a letter and sits in a panel above the standings now. Grading the yardstick against itself would put it in a band the engines have to earn.

Dynalite and LocalStack are both being held down by coverage. Each would be a B on divergence alone, and in both cases the coverage shortfall drops them to C. Their rows say so, because the two Cs on the board aren't the same shape of C.

How to actually use it: if you need specific operations, ignore the letter and go to the target's page, where What it doesn't attempt names the coverage gap directly. That's the answer to "will this emulator do what my app does". If you're picking blind for general CI use, the letter is the shortlist and divergence breaks the tie, because a high-coverage engine that gets things wrong will cost you more debugging than a narrow one that's honest about its limits.

What's new in the suite

Vector search. AWS shipped it on 5th August and the suite covers its deterministic surface as of this release: 42 tests, all characterised against real DynamoDB in eu-west-2 first. The interesting part was index readiness, where the documentation is wrong three separate ways: the pages contradict each other on what a premature search does, they describe a Backfilling state the index never enters, and the status they tell you to wait for arrives before the index can answer. That's written up in when a DynamoDB vector index is actually ready, with both sides of the contradiction captured in the repo. No emulator implements any of it yet, so every coverage figure on the board dropped when these landed, while divergence stayed untouched.

Index write costs. The suite's only per-index capacity assertion was on a Query, so the write side, the half you get billed extra for, went unmeasured. Fourteen new Tier 1 tests fix that. Moving an item to a new GSI key costs two units on that index rather than one, and an overwrite that changes nothing costs nothing at all. The full set of numbers is in what a write to an indexed DynamoDB table costs.

Housekeeping

A target's page should be readable by someone who's never seen the project. How to run it lists every way the target is distributed, Docker image, npx, cargo, a JAR, each linking to the project's own page. What it doesn't attempt names the coverage gap. Builds of an engine now nest underneath it rather than taking a row alongside.

The history chart split into two plots. One line couldn't be trusted on its own, because divergence falls whenever tests a target used to fail stop being run against it, which is how July rendered as an improvement.

Tier bars use length for coverage and colour for divergence. Movement reads 4.9pp less instead of -4.9pp. Every heading on the prose pages has an id you can link to.

The badges were still publishing the correctness percentage the board had retired, so a badge could disagree with the table it linked to. They show the grade now, on the same URL.

Corrections

The methodology page used to claim that measuring over the whole suite stopped a thin emulator posting a perfect score. It doesn't. Zero fails is 0.0% at any coverage, which is exactly why the grade caps on coverage instead.

The baseline row was overstating itself. Real AWS runs in three passes and only the main one reached the published artefact, so the row claimed 998 of 998 on a run that recorded 981. The passes are merged before scoring now, and the row stays pinned and says so until all three report.

The suite counts its own tests. Until this release "the whole suite" meant whichever target had run the most tests, which put one of the measured things in charge of everyone else's denominator. There's a generated manifest of all 1,054 now, and CI fails if it drifts from the tests.

What breaks

JSON consumers. The data endpoints jump two schema versions. Anything still on 2 gets both breaks at once, and there are four of them.

The dangerous ones are movement.state and movement.delta. State's values changed from up/down to improved/regressed, so the old names still parse and now mean the opposite direction. Delta inverts the same way without changing shape, so the sign means the opposite of what it did and nothing in the data says so.

The other two announce themselves. A tier carries divergence, coverage and correctness instead of a bare pct and value, and the whole-suite correctness percentage is correctness rather than total, which also named a raw count elsewhere in the same payload.

Correctness is still published under its own name, so anything built on the old figure keeps working. It just isn't the headline any more.

The full notes are in the 3.0.0 release, the suite is at paritysuite/dynamodb-conformance, and the board itself is at paritysuite.org.

The tests and the scoring code are both in the repo. If something looks wrong, an issue is welcome.