You can't pin DynamoDB to one region

A conformance suite is only as good as its ground truth. Mine works by running every test against real DynamoDB and recording what comes back, so the expected answer is never my reading of the docs - it's what the service actually did when I asked it. That only holds up if real DynamoDB does the same thing everywhere. It mostly does. Not always.

Take PutItem with an attribute value of { NULL: false }. As long as I'd been testing it, real DynamoDB rejected it: a NULL attribute has to carry true, and anything else came back as a ValidationException. Then, a few weeks ago, eu-west-2 started accepting it and quietly storing it as { NULL: true }. Same API, same input, a different answer depending on which region you called.

That's a problem, because my suite drew its ground truth from a single region: eu-west-2, the one I'm nearest to and had built the whole thing against. So the moment eu-west-2 started accepting { NULL: false }, the suite decided accepting it was correct, and marked every engine that rejected it - which was most of them - as wrong. They weren't wrong. They were doing what real DynamoDB still does in Virginia, in Sydney, and everywhere else the change hadn't reached. The suite was measuring conformance to eu-west-2 and calling it conformance to DynamoDB.

I'd already caught it in June

In June I watched real DynamoDB reword its validation errors while the suite was pointed at it, and the same dig turned up the { NULL: false } split. I fired the same inputs at four regions: eu-west-2 and eu-central-1 had the new behaviour, us-east-1 and an Asia-Pacific region still had the old. Two and two. I read it as AWS rolling a change out region by region, caught mid-flight.

I even wrote the real conclusion down at the time - that the suite's number "has always really been conformance to DynamoDB in eu-west-2, not conformance to DynamoDB". Then I filed it as a cosmetic thing about error strings and left it for a month. It wasn't cosmetic. It was deciding scores.

AWS ships behaviour region by region, which makes "which region" part of the answer, not a footnote. Pin one and you bake that region's place in the queue into the definition of correct, then mark every other real region wrong for being somewhere else in it.

Ground truth, per region

So the suite stops pinning. The full run now goes out against real DynamoDB in every commercial region on a schedule, and records what each one actually does. There are thirty-four of them - GovCloud and China are separate partitions and out of scope. Each engine is scored against every region the sweep trusts, and its headline is the region it matches best.

That label is worth having on its own. Most of the time an engine matches every region equally and it just reads "all regions"; where an engine lines up with a region eu-west-2 disagrees with, the label names that region instead. Either way the score now says what it was measured against, which a bare percentage never did.

An engine is conformant on a behaviour if it matches some real region coherently. It fails only when no region does what it did.

Match one region, not any

When I wrote about the plan in July I said an engine would conform if it matched any real region. The RFC I opened said the same. That's not what I shipped.

Match-any lets an engine take eu-west-2's answer on one behaviour and us-east-1's on the next and pass on both, which is a DynamoDB deployment that exists nowhere. So the headline is the best single region an engine matches, scored one region at a time. Stitch together answers from different regions and you still fail. It's stricter than I said it would be, and stricter than the RFC asked for, on purpose.

An absent answer isn't a different answer

The hard part wasn't measuring disagreement. It was not mistaking absence for it.

A region that times out or gets throttled into the ground hasn't told you anything. If a non-answer looked like a behavioural difference, the suite would invent a split out of a bad afternoon in Sydney and loosen its own baseline for good. So there's a third result alongside pass and fail: indeterminate. It scores nothing on either side, can't mint a split, and can't fail anyone. A region that can't finish a sweep is published as unresolved rather than quietly dropped, and two misses in a row take it out of scoring. Either way it logs an issue rather than going silent.

Because the headline is the best of the trusted regions, dropping one can only lower a score or leave it alone. It can never raise one, which is what makes losing a region safe.

The registry only a human writes

Detection is automatic. Admission isn't.

When the sweep spots regions giving different definite answers to the same test, it opens an issue with the evidence and stops. I read it and commit the row by hand, and the row records what every named region returned, and when. Anything that wrote its own rows would launder whatever a region did that day - a regional fault or a half-finished change - straight into the baseline every engine is then scored against. The whole value of the suite is that the baseline is real DynamoDB rather than my opinion, and an automatic writer would quietly swap one for the other.

What the first full sweep found

The design leaned on there being more than one split. If regions only ever disagreed about a single attribute value, this would be a lot of machinery around a footnote.

The honest count came back at four, across nine hundred and fifty-odd tests and thirty-four regions. Small, which is roughly what you'd want from a managed service, and exactly what makes the handful that do disagree worth looking at.

Then the sweep did the thing I built it for. Its verdict on { NULL: false }: eu-west-2 and eu-central-1 had reverted. The two regions that briefly accepted the value were rejecting it again, in line with everywhere else, and the split I'd built the whole redesign around had closed.

I'd taken it for a rollout in progress and guessed the difference would vanish once it reached every region. It vanished the other way. Maybe AWS shipped the change to two regions and thought better of it. Maybe it was never a deliberate change at all - a regression that slipped into eu-west-2 and eu-central-1 and got pulled once someone noticed. From outside I can't tell the two apart, and that's the point. The behaviour I'd been treating as ground truth turned out to be a few weeks of weather, and if I'd still been pinning eu-west-2, the suite would now be failing every engine on earth for not matching a copy of DynamoDB that no longer exists.

A region's answer can be temporary, whether it's a rollout or a glitch, and from inside that one region you can't tell which. That's the case against pinning one, in a single test.

The trap reaches the engine too

The same mistake runs one level down, into Dynoxide, my own engine. Dynoxide is built to pass this suite, so eu-west-2 was effectively its ground truth as well. When eu-west-2 spent a few weeks accepting { NULL: false }, Dynoxide was built to accept it. Now that every region rejects it again, Dynoxide is the one engine of the seven that still doesn't - carrying a behaviour AWS has withdrawn everywhere, for exactly the reason the suite nearly did: it learned "correct" from a single region.

So Dynoxide gets the same fix as the suite. Match what the regions actually agree on, which here means rejecting { NULL: false } like all of them now do. Build against one region and you inherit its quirks, including the ones with a shelf life.

The digits barely moved

Nothing dramatic happened to the percentages. What changed is what they mean, and how far you can trust them. A score used to say "conformant" and quietly mean "conformant to eu-west-2, this week". Now it names the region it was measured against, and re-checks that region every week - and the first check already caught one region reverting.

Real DynamoDB isn't a fixed target. It's thirty-odd regions that mostly agree, with AWS changing them one at a time and every so often changing its mind. A conformance suite that pins one of them isn't measuring DynamoDB, it's measuring a postcode on a particular Tuesday, and I'd been doing exactly that for longer than I'd like without clocking what it cost.

One number was never enough

Pinning one region was really one case of a wider habit this release sets out to break: letting a single figure stand in for a messy truth. That "conformant to eu-west-2" number was hiding a region. It was also hiding which operations made it up.

An emulator can be flawless on the everyday reads and writes and come apart on a whole capability, and one overall percentage blends the two into a shrug. So every target page now carries a per-operation breakdown: each operation the target implements, grouped by tier, with its own pass rate and a mark for full, partial, failing or unsupported. LocalStack, as I write this, passes every getItem test and only about three-quarters of its putItem ones - a split the headline number never showed.

Underneath, the suite grew: 954 tests now, up 81, all characterised against real DynamoDB. The new coverage went into the corners AWS doesn't document and everyone eventually trips over - UpdateTable reconciling its attribute definitions, projection-expression validation, a 4096-byte cap on every expression, empty members in string and binary sets.

paritysuite.org now shows all of it - per region and per operation, with a page on how the regional ground truth works. The suite itself is Apache-2.0 with everything in the repo: the registry rows with every region's recorded answer, the sweep, the scoring. Point it at your own region and see what comes back.