03 — Testing and Verification
Commandment III: Verify, don’t declare.
Two different things, often conflated:
- Tests — an automated safety net you run repeatedly.
- Verification — proof that this specific change does what it was meant to, observed, not assumed.
Both are mandatory. A passing test doesn’t mean the feature works in the browser; manual verification without tests won’t protect you from tomorrow’s regression.
The pyramid (from the reference project)
| Level | Tool | What it covers | When |
|---|---|---|---|
| Unit | pytest / Jest | pure logic (normalizer, matcher, helpers, validators) | every commit |
| Integration | Supertest (HTTP) | routes, auth guards, JSON API — An agreed way for two programs to talk to each other — one asks, the other answers in a set format. Through an API your app connects to outside services (payments, maps, AI). Treat an API key like a password. | every commit |
| E2E / browser | Playwright | real browser paths, on a separate port | before deploy / when touching UI |
| Smoke | script (smoke.ps1) | “does the server come up and do N routes respond correctly” | after restart / before and after deploy |
Rules that have proven themselves:
- Run the tests before committing the critical path.
- Jest with
--runInBandwhen a dev/preview server is running (port/DB conflicts). - Smoke test — A quick “does it even work” test right after a deploy — checks the most important paths (e.g. login). Catches disasters in 30 seconds before a user sees them. A cheap way to sleep well after a deploy. have markers — they check that the right page rendered (e.g.
price-gatefor an anonymous user, not just HTTP 200). - Test auth from both sides — anonymous user bounced to login and logged-in user sees the content.
”Verify, don’t declare” — in practice
After a change observable in the app, show proof; don’t ask the user to check it themselves:
- HTTP status /
redirect_url(curl-o /dev/null -w), - a confirming HTML fragment (e.g.
og:imagepoints at the right file), - numbers from the database (how many accounts, how many prices,
integrity_check: ok), - a screenshot for visual changes.
After a deploy — a smoke test against the live server (via localhost, behind the maintenance
flag); and when you find a bug inside the window, fix it in the window if it’s trivial and safe (in the
reference project this is how we caught and fixed a pre-existing ERR_HTTP_HEADERS_SENT on /mapa).
The report must be honest
- Tests fail → say so with the output; don’t hide it.
- Something was skipped → say it was skipped.
- When 2 tests fail from data drift (e.g. cohort fixture
85% vs 85%) rather than a regression — flag it explicitly, don’t let it block the deploy, but propose a follow-up (regenerate the fixtures). - Say “done and verified” only when it’s actually verified — no hedging, but no bluffing either.
Verify numbers at the source
Don’t trust a number from memory or from the docs — query the database/test. Docs go stale;
SELECT COUNT(*) doesn’t lie. (Commandment X: “verify the numbers”.)
Web — a proven set of checks
For web sites/projects this set of tests has already proven itself in practice (a typical set runs to a few dozen methods and a few hundred subtests) — carry it over by default:
- SEO meta (title/description per page), canonical + hreflang — A tag that tells the search engine what language a page is in and where its versions in other languages are. Without it a multilingual site confuses Google — the language versions “fight” over the same phrase., JSON-LD — Structured data: a description of the page invisible to humans that the search engine reads (e.g. FAQ, author, breadcrumbs). Opens “rich snippets” in Google — richer results, higher CTR..
- Accessibility: color-token contrast, keyboard navigation, sensible
alt/aria. - EN↔PL parity (and every language pair): a missing key/page in one language = a test that fails.
- Dead internal links — no link in the build leads into the void.
- Theme consistency (theme cookie / dark-light) and build correctness (every page rendered).
Anti-patterns
- 🚫 “Should work” as a conclusion.
- 🚫 Running a single test and declaring “suite green”.
- 🚫 Skipping the smoke test after deploy because “the tests passed anyway”.
- 🚫 Confusing “the code compiles” with “the feature works for the user”.
TDD — the default working mode
Test first, not after the fact. The cycle:
- Red — write a test that describes the expected behavior and fails.
- Green — the smallest code change until the test passes.
- Refactor — clean up against a green suite (→ 02).
You fix a bug the same way: first a test that reproduces the bug, then the fix — the test stays as a regression guard so the bug doesn’t come back.
Every change in a commit carries a test. Hard rule: a commit that changes behavior but doesn’t add/change a test is incomplete. New Endpoint — A single “address” in an API you send a request to for a specific thing (e.g. the list of orders). Apps talk through endpoints — one endpoint = one function you expose. → route test; new threshold/fallback → threshold test; fixed bug → regression test. “I’ll add tests later” = a sin (later never comes). → 00, 08
A guard is proven by the incident, not by the fixture
A check written after an outage must be replayed against that outage’s real data before you trust
it. Fixtures are written by the same head that wrote the bug, so they inherit its blind spot: in the
reference project a data-freshness check passed nine green unit tests and then failed to flag the very
outage it was written for — the comparison was > where it needed >=, and the frozen sources sat
exactly on the threshold. A guard that lets its own reference case through is worse than no guard: it
converts an open question into false calm. Point it at production (or a snapshot), confirm it fires,
and only then freeze the boundary in a fixture — with an explicit test on the boundary itself.
- Test the contract, not the implementation — otherwise a refactor crumbles tests with no real regression.
- Keep the suite fast and deterministic — a slow or flaky suite stops getting run;
isolate I/O, set seeds,
--runInBandon a shared port/DB. - CI gates — red tests block merge and deploy; you run critical-path tests before committing.