testing/cross_backend/differential.ts

The cross-backend differential: each impl is the oracle for the other.

A conformance case pins its status and reason on one backend at a time, and an output schema is often loose (z.looseObject, or Zod's default strip), so an extra field from one spine — possibly a leak — passes, as does any body difference on a case pinned by status alone. describe_differential_conformance_tests sends every case to two backends, normalizes both responses (normalize_differential_response), and asserts they are equal: status, header names, and the whole body.

Each case runs on a fresh fixture per backend — each backend's own setup_test, so separate databases and separate keepers, which is why the normalization is positional. The principal resolves and the request is sent through the table runner's own send_conformance_case, so a case means the same thing here as in describe_conformance_table_tests. The per-case expectations (expect.status, error_reason, …) are not asserted here — each backend's conformance leg already holds them; the differential holds the two answers to each other.

Every HTTP request asks for an uncompressed response (DIFFERENTIAL_REQUEST_HEADERS): content negotiation is the transport's, not the app's — Deno's Deno.serve compresses a response whose request accepts gzip, as Node's fetch sends by default, while other servers send it as is — so letting a server choose would read as a content-encoding and vary difference on every case.

A difference that is tracked and accepted goes in the known-divergence ledger (KnownDivergence), matched by case names or by methods, naming the paths that differ (what_differs), each optionally with the value each side must show there. A case may match several entries and differ at any of their paths, with the declared values, and nowhere else: any other difference fails like any other. The ledger cleans itself, with the xfail_until semantics — red when a divergence closes, naming the entry to delete: a case an entry names by case_name must still show it, checked in that case's own test, and an entry matched by method must still show it on at least one of its cases, checked by a closing test once they have all run. The closing test also fails an entry that matches no case, or that names a case that doesn't exist, and leaves an entry whose cases didn't all run (a -t filter, a send that threw) unjudged rather than calling it closed.

Use the backends' cross-process fixtures — the differential needs two live backends, so it runs in a dual-spawn project. fuz_app's own wiring is the cross_backend_parity project's differential.cross.test.ts.

view source

Declarations
#

8 declarations

check_known_divergence_ledger
#

testing/cross_backend/differential.ts view source

(cases: readonly { name: string; request: { method: string; as: "token" | "keeper" | "daemon" | "invalid_daemon" | "daemon_browser" | "bearer_browser" | "malformed_bearer" | "scoped_token" | "anonymous" | "fresh_non_admin" | "role_holder" | "wrong_role" | "expired_session"; params?: unknown; params_json?: string | undefined; acting?: "principal" | ... 2 more ... | undefined; verb?: "GET" | ... 1 more ... | undefined; }; expect: { ...; }; note?: string | undefined; xfail?: { ...; } | undefined; }[], ledger: readonly KnownDivergence[]): string[] import {check_known_divergence_ledger} from '@fuzdev/fuz_app/testing/cross_backend/differential.js';

The ledger's static problems against a case list: an entry that matches no case, and every case_name it lists that no case carries, or every method no case sends — a dead name would otherwise ride along behind a live one in the same entry.

Needs no backend; the differential's closing test reports these, and a consumer can check its ledger with it directly.

cases

the cases the ledger is run against

type readonly ConformanceCase[]

ledger

the known divergences

type readonly KnownDivergence[]

returns

string[]

one message per problem, empty when the ledger names only real cases

describe_differential_conformance_tests
#

testing/cross_backend/differential.ts view source

(options: DifferentialConformanceOptions): void import {describe_differential_conformance_tests} from '@fuzdev/fuz_app/testing/cross_backend/differential.js';

Register a describe block sending every case to both backends and asserting their normalized responses are equal, or — for a case in the known-divergence ledger — that they still differ, and only where listed.

options

the cases, the two backends, and the shared surface

returns

void

DIFFERENTIAL_REQUEST_HEADERS
#

testing/cross_backend/differential.ts view source

Readonly<Record<string, string>>

type Readonly<Record<string, string>>

import {DIFFERENTIAL_REQUEST_HEADERS} from '@fuzdev/fuz_app/testing/cross_backend/differential.js';

Headers every HTTP request the differential sends carries, beneath the principal's: accept-encoding: identity, so each backend answers uncompressed and a runtime's own compression (Deno.serve compresses for a request that accepts gzip) never reads as a header difference.

DifferentialBackend
#

testing/cross_backend/differential.ts view source

DifferentialBackend import type {DifferentialBackend} from '@fuzdev/fuz_app/testing/cross_backend/differential.js';

One side of the differential.

name

Short label used in failure output (ts, rust).

type string

readonly

setup_test

Per-test fixture producer for this backend.

type (): Promise<TestFixture>

readonly
returns Promise<TestFixture>

ws?

The backend's WebSocket endpoint. Set on both backends to send the cases over a socket instead of HTTP — only the cases is_ws_conformance_case admits, and the ledger is checked against those. Set on one alone, the suite throws at registration.

type WsEndpointTarget

readonly

DifferentialConformanceOptions
#

testing/cross_backend/differential.ts view source

DifferentialConformanceOptions import type {DifferentialConformanceOptions} from '@fuzdev/fuz_app/testing/cross_backend/differential.js';

cases

The cases to send to both backends, in order.

type readonly ConformanceCase[]

readonly

backends

The two backends; a and b in diffs are these, in order.

type readonly [DifferentialBackend, DifferentialBackend]

readonly

surface_source

Surface spec — supplies the RouteSpecs for the REST branch.

type AppSurfaceSpec

readonly

rpc_endpoints

RPC endpoints — resolved to find each method's action spec.

type RpcEndpointsSuiteOption

readonly

live_rpc_endpoints?

The RPC endpoints both backends live-mount, for methods off the declared rpc_endpoints — resolved the way the table runner resolves them (resolve_conformance_rpc_endpoints).

type RpcEndpointsSuiteOption

readonly

session_options

Session options — needed to resolve the rpc_endpoints factory form.

type SessionOptions<string>

readonly

principals?

Maps the role_holder / wrong_role principals to seeded usernames.

type ConformancePrincipalConfig

readonly

known_divergences?

The tracked divergences these cases may show.

type readonly KnownDivergence[]

readonly

suite_name?

describe block label. Defaults to 'differential conformance'.

type string

readonly

KnownDivergence
#

testing/cross_backend/differential.ts view source

KnownDivergence

type ({ readonly case_name: string | readonly string[]; readonly method?: undefined; } & { readonly tracking_id: string; readonly reason: string; readonly what_differs: readonly (string | KnownDivergencePath)[]; }) | ({ readonly method: string | readonly string[]; readonly case_name?: undefined; } & { readonly tracking_id: string; readonly reason: string; readonly what_differs: readonly (string | KnownDivergencePath)[]; })

import type {KnownDivergence} from '@fuzdev/fuz_app/testing/cross_backend/differential.js';

A tracked, accepted difference between the two backends.

what_differs lists the normalized-response paths that differ (DifferentialDiff.path — status, headers.<name>, body.error.data.reason), each optionally with the value each side shows (KnownDivergencePath). A matched case may differ there and nowhere else. An entry naming cases must still differ on each of them; one naming methods, on at least one of its cases.

A pin is compared with the normalized leaf value at the differing path: a placeholdered value never matches the raw one, an object pin must list its keys sorted, and a pin on a parent path never matches, since diffs are reported at their leaves.

KnownDivergenceMatch
#

testing/cross_backend/differential.ts view source

KnownDivergenceMatch

type { readonly case_name: string | readonly string[]; readonly method?: undefined; } | { readonly method: string | readonly string[]; readonly case_name?: undefined; }

import type {KnownDivergenceMatch} from '@fuzdev/fuz_app/testing/cross_backend/differential.js';

What a ledger entry matches: cases by name, or every case on a method — one, or a list.

KnownDivergencePath
#

testing/cross_backend/differential.ts view source

KnownDivergencePath import type {KnownDivergencePath} from '@fuzdev/fuz_app/testing/cross_backend/differential.js';

One path a known divergence differs at, with what each side must show there.

path covers itself and everything under it (differential_path_covers). a / b, when given, are the value that side's diff must carry at the differing path — undefined meaning the side has nothing there — so a divergence that changes shape (the side that lacked a field starts sending a different one, or the other side's value moves) is no longer covered and fails. Omit a side to leave its value open. A bare string is a path with both sides open.

A pin is compared with the normalized leaf value at the differing path: a placeholdered value never matches the raw one, an object pin must list its keys sorted, and a pin on a parent path never matches, since diffs are reported at their leaves.

path

type string

readonly

a?

type unknown

readonly

b?

type unknown

readonly

Depends on
#