Parity Suite 3.6 and a month of emulator fixes

When I wrote up 3.0.0 in August, Floci was a C on the board with 209 failing tests. By 3rd September the suite had grown to 1,251 tests and it was a D with 337. This morning's run has it failing one.

That's the real news in this release, and none of it is mine. The suite barely grew. The emulators got better.

The board

The suite reached 1,251 tests on 3rd September, so that's the fairest place to measure from. It's at 1,266 now.

All three now pass every Tier 1 test, the core of DynamoDB that most applications rely on. Five weeks ago Floci failed 50 of those, MiniStack 28 and LocalStack 14.

Floci didn't get there by doing less, either. Its coverage, the share of the suite it attempts rather than skips, went up from 95.1% to 99.4% over the same stretch.

Its one remaining failure is an edge case. A transaction carries a value nested deeper than DynamoDB's 32-level limit, but only in a condition, so it never gets written. Most regions don't check its depth and go on to evaluate the condition. eu-west-2 and the two other regions that now validate transactions up front (more on that below) refuse the whole request. Floci does neither: it cancels the transaction.

Most of LocalStack's 31 are PartiQL (21 of them), with the rest in exact error wording and limits. MiniStack's 84 are nearly all Tier 3, mostly exact error messages and limits.

It's good to see the board being used for what I built it for. Floci's tests and pull requests cite it by name, and MiniStack's maintainers point contributors to it when the question of how close to real DynamoDB they are comes up. Thanks to both teams for that.

AWS moved in the middle of a run

3.6.0 was tagged on 8th October and never published. While it was being measured, eu-west-2 changed how it validates transactions, so five tests failed against real DynamoDB and no board came out.

It's part of the same rollout I noticed back in June, and it still hasn't finished. eu-west-2, eu-north-1 and ap-northeast-2 now check a transaction's request before running it. The other 30 regions don't yet, and a few of the changes are more than wording: the same request can succeed in one region and fail in another.

For example, take an item that's already there, and a single SET that brings it up to exactly DynamoDB's 400KB limit (409,600 bytes), sent as an Update inside TransactWriteItems:

// bigValue brings the item to 409,600 bytes
await ddb.send(new TransactWriteItemsCommand({
  TransactItems: [{
    Update: {
      TableName: 'orders',
      Key: { pk: { S: 'Y' } },
      UpdateExpression: 'SET p = :p',
      ExpressionAttributeValues: { ':p': bigValue },
    },
  }],
}))

In us-east-1 that succeeds and the item is stored at 409,600 bytes. In eu-west-2, since 8th October, it fails with ValidationException: Item size to update has exceeded the maximum allowed size. If your tests run against one region and production sits in another, the two can disagree on this request. It's another reason you can't pin DynamoDB to one region.

3.6.1 is the fix. Those five tests accept either answer until the regions settle, and no engine loses a test over it. Pinning to the old answer would fail real DynamoDB in eu-west-2, which is the region the suite measures against, and pinning to the new one would fail every engine that matches the 30 regions still giving the old answer. Both answers are what DynamoDB actually does today, so both count.

Smaller changes

The board now sorts by grade, best first. It used to sort on divergence alone, which could put a target with a worse grade above a better one. No grade or figure changed.

The suite grew by fifteen tests across 3.5.0 and 3.6, fourteen of them from exoego, covering size and nesting limits on batch and transaction writes.

Coming next

The suite's going to grow again soon. I've found some gaps in what it covers, and there are around 160 new tests waiting to go in, all checked against real DynamoDB first. I'll probably merge them early next week, and I'd expect most engines' figures to move when they do.

The full notes are in the 3.6.1 release, the suite is at paritysuite/dynamodb-conformance, and the board is at paritysuite.org. If something looks wrong, an issue is welcome.