A chapter about building so you can change and grow later — without rewriting everything from scratch. And about the equally important other side: don’t over-engineer. A simple architecture you know how to extend beats a distributed one you don’t need.

Three habits give you flexibility:

Separate the layers. The database, the data-fetching and the page are three separate worlds joined by a contract (an agreed format), not by shared code. Then you can swap one part without touching the rest.

Change by “adding”, not “rebuilding”. A new column, a new Endpoint — A single “address” in an API you send a request to for a specific thing (e.g. the list of orders). Apps talk through endpoints — one endpoint = one function you expose., a new language — done so that what already works keeps working.

Hide risk behind a switch (Feature flag — An internal switch that turns a new feature on for some people, or off, without reworking the code. You roll a novelty out gradually and roll it back in a second — the cheapest insurance with any change.). A new feature ships to the app turned off; you enable it first for yourself, then for a fraction, then for everyone. Without it every change is “all or nothing”.

And above all: scale when a metric forces you, not “because it’ll grow someday”. SQLite — The simplest database — the whole thing lives in one file, with no separate server. A great default to start: zero configuration, easy backup (you copy the file). As you grow, you migrate up. and a static site carry a surprising amount of traffic; you move to heavier solutions when the numbers (cost, latency, database size) demand it — deliberately, with a written rationale.

Example: microservices and Kubernetes for an app with a hundred users is premature complexity — you add the network and a new class of failures and gain nothing. Start with a Monolith — An app built as one whole (one deploy, one database), not a patchwork of separate services. A simple, cheap default to start — easy to grasp. You split it only when scale really demands. (one application).

12 — Flexibility and scalability

The golden rule of altitude applied to architecture: slow where a mistake is expensive; fast where it’s cheap. Scale when a metric demands it — not sooner, not “just in case.”

Flexibility is the ability to change without a rewrite; scalability is the ability to grow without a rebuild. Both come from the same habits: separate the layers, make changes additive, hide risk behind a flag, don’t couple state where you don’t have to. And — just as important — don’t over-engineer: a simple architecture you can grow into beats a distributed one you don’t need.

Separate the layers

DB ↔ ingest ↔ web are three separate worlds. In the reference project: SQLite — The simplest database — the whole thing lives in one file, with no separate server. A great default to start: zero configuration, easy backup (you copy the file). As you grow, you migrate up. (data) ↔ Python — A popular, readable programming language — The Craft’s default for scripts, data and the back end. A proven default: lots of ready tools and libraries, easy to drive with AI. scrapers (ingest) ↔ Node/Express (web) — bound by a contract (the schema, → 11), not by shared code. You can swap the Scraping — Automatically gathering data from websites with a program instead of copying by hand. Powerful for acquiring data, but it needs manners: respect others’ rules (robots.txt), don’t overload the server. without touching the web layer.
The API — An agreed way for two programs to talk to each other — one asks, the other answers in a set format. Through an API your app connects to outside services (payments, maps, AI). Treat an API key like a password. contract is the boundary — e.g. a Next.js frontend ↔ a Python backend via an explicit contract (OpenAPI). A boundary you hold to lets you replace either side independently. → 08
A shared backend for web + a future mobile app. An ADR ahead of time: SQLite vs PostgreSQL — A solid, “grown-up” database for bigger apps — runs as a separate server. The default step after SQLite, when you need many concurrent users and advanced features., session vs JWT (e.g. sessions for web; mobile later → consider JWT on the shared backend). Write the decision down, don’t keep it in your head. → 01

Make changes reversible and rampable

Additive / backward-compatible — a new column, a new Endpoint — A single “address” in an API you send a request to for a specific thing (e.g. the list of orders). Apps talk through endpoints — one endpoint = one function you expose., a new language; don’t break what works (→ 11).
Feature flag — An internal switch that turns a new feature on for some people, or off, without reworking the code. You roll a novelty out gradually and roll it back in a second — the cheapest insurance with any change. — feature_flags in the database + toggles like BETA_ALL_PREMIUM. Ship dark, then ramp: the code ships off, you turn it on for a slice, then for everyone. Without flags every change is all-or-nothing.
Stateless where you can — the less state in the process, the easier it is to scale horizontally.

Platform: scale-to-zero vs always-on

A real decision from two projects, a deliberate cost/latency tradeoff:

Cloud Run (scale-to-zero) — you pay for usage, zero traffic = zero cost, but cold start adds latency to the first request. Good for uneven, global traffic.
VPS — A rented piece of a cloud server “just for you”, where you put an app “live”. Predictable cost and full control. One VPS easily carries a few small projects. always-on (Hetzner — A cheap, solid server (hosting) provider you put an app “live” on. The codex’s default server choice — predictable cost and performance without overpaying. + pm2 — A manager that keeps the (Node) app running all the time — restarts it after a crash. Without it, after a crash or server restart the app just sits idle. pm2 keeps it “alive”.) — e.g. a fixed cost, zero cold start, full control. Good for predictable traffic and SQLite on disk. The choice comes down to traffic profile and budget, not fashion. Record it as an ADR.

Cache and pipelines

Cache — A temporarily remembered result, so the same thing isn’t computed again on every request. Speeds the app up, but can be a trap: a stale cache shows old data. layers with explicit invalidation: e.g. cache invalidation in the full pipeline, rankings cached for 10 min, statics with max-age (an hour in dev / 7 days immutable in prod).
Composable, Idempotency — A property of an operation you can run many times with the same result — no duplication. Key for scripts and events: a re-run doesn’t break data. Like an “ON” switch. pipelines — normalize → metadata → enrich → validate → stats; each stage runnable on its own, a re-run is safe (→ 04).
i18n / multi-market from the start, if global (e.g. 16 languages from day one, not bolted on later — → 10).

Don’t over-engineer

Start simple. SQLite + a static build (e.g. a static build in Python) handle a surprising amount of traffic. The reference project on SQLite/WAL — Write-Ahead Log: a database mode where changes go to a journal first, then to the data. Gives safety (recovery after a crash) and better read/write concurrency. serves a few thousand items without Postgres.
Scale when a metric demands it — not “because it’ll grow someday.” Right-size to real traffic.
A SQLite→Postgres Migration (database) — A controlled change to the database layout — adding a column, a table or moving data — step by step. Like a renovation to plan: rebuilding data in a set order so nothing “collapses”., Monolith — An app built as one whole (one deploy, one database), not a patchwork of separate services. A simple, cheap default to start — easy to grasp. You split it only when scale really demands.→services: when the numbers demand it, with an ADR, not preemptively.

Anti-patterns

🚫 Premature distributed complexity (microservices/Kafka/k8s for 100 users).
🚫 Stateful coupling that blocks scaling (session state in process memory with no store).
🚫 Big-bang rewrite instead of additive changes (→ 04).
🚫 No feature flags → every change forced into all-or-nothing, no ramp/dark-ship.
🚫 Ignoring the cost of always-on (paying for idle when scale-to-zero would fit) — and vice versa.
🚫 Rewriting to Postgres “just in case” while SQLite hasn’t even broken a sweat.

For new projects

On Day 0 (→ 07) set three boundaries (DB / ingest / web) and write three ADRs: database (SQLite vs Postgres), session vs JWT, platform (VPS vs Cloud Run). Introduce feature_flags from the start — it’s the cheapest insurance policy on flexibility. Start with the simplest stack that delivers (statics/SQLite), and defer scaling to the moment a concrete metric (latency, cost, database size) demands it — and then decide by the numbers, not by hunch (→ 13).