A chapter about three powers that drive many products today: pulling data from the web (Scraping — Automatically gathering data from websites with a program instead of copying by hand. Powerful for acquiring data, but it needs manners: respect others’ rules (robots.txt), don’t overload the server.), delegating tasks to AI, and chatbots. You do them “with a contract”, not by magic.

Scraping — the official source first. If there’s an API — An agreed way for two programs to talk to each other — one asks, the other answers in a set format. Through an API your app connects to outside services (payments, maps, AI). Treat an API key like a password., a feed or an export — you take it; manually “tearing off” HTML is a last resort, because it breaks with every redesign of the site. And respect the source: honour robots.txt, don’t flood the server with requests — the goal is data, not a DoS.

AI for what ordinary code can’t handle. A language model is great at extracting meaning from unstructured text, classifying, summarising. But don’t use it for what a regex or SQL will do — that’s pricier, slower and less certain. And always force the response format (e.g. JSON) instead of parsing prose “on hope”.

A chatbot is a product feature, not a toy. Keep its personality and rules as configuration (a file/database), not hard-wired in code — then you change the tone without a deploy. And it must answer from your data (grounding), not from the model’s memory — otherwise it will “confidently” make up a price or a fact.

Example of a threat (Prompt injection — An attack on AI: someone weaves a hidden instruction into text to make the assistant act against the rules. If AI reads outside content (emails, pages), treat it as untrusted — otherwise it can be manipulated.): treat content from a user or a web page as data, not commands. If a chatbot blindly executes an instruction hidden in someone else’s text, that someone can steer it against your rules.

15 — Scraping, AI APIs, and configurable chatbots

Pulling data from the web and delegating tasks to AI is the core of many products today. Do it effectively, cheaply, and with a contract — not by magic. The operational layer (timeouts, backoff, rotation, quotas, resumability) lives in → 14; this chapter is about using those powers as product features: where to get data, how to delegate tasks to AI, and how to build an assistant you can steer without a deploy.

Effective scraping — data from the web

Official source first. An API — An agreed way for two programs to talk to each other — one asks, the other answers in a set format. Through an API your app connects to outside services (payments, maps, AI). Treat an API key like a password., a feed, an export, a sitemap — HTML Scraping — Automatically gathering data from websites with a program instead of copying by hand. Powerful for acquiring data, but it needs manners: respect others’ rules (robots.txt), don’t overload the server. is the last resort, not the first move. An official contract doesn’t break on every site redesign.
Respect the source. Honor robots.txt/ToS, throttle reasonably, identify yourself. The goal is data, not a DoS — an aggressive scraper is a legal and ethical problem (→ 09).
Parse defensively. Selectors break on redesigns — validate the shape of the result, not just the status (the classic “200 + error page,” → 14). Selector drift → an alert, not silent zeros.
Normalize at the entry point. Web data is dirty: fuzzy-match and dedup against lookup tables, Slug — A readable, short part of a page address that describes its content in words instead of a mysterious number. Better for humans and SEO; a stable slug doesn’t break links when things change., one source of truth (→ 11). Raw material goes in, a clean record stays.
Incrementally. Fetch the delta (what changed since last time), not the whole catalog every night — cheaper, faster, lighter on the source. Checkpoint and resumability → 14.
Idempotency — A property of an operation you can run many times with the same result — no duplication. Key for scripts and events: a re-run doesn’t break data. Like an “ON” switch. loaders. Re-importing the same data creates no duplicates (→ 04).

Anti-patterns

🚫 Scraping instead of an API that exists — fragility for no reason.
🚫 A selector with no shape validation — a silent failure floods the database with junk/zeros.
🚫 A full re-scrape every day instead of the delta — waste and a ban risk.
🚫 No throttle/identification — a DoS for the source, a legal problem for you.

AI APIs for specific tasks — not for everything

An LLM where deterministic code can’t cope: extraction from unstructured text, classification, summarization, normalizing descriptions, generating content. Not for what a regex, SQL, or a plain function will do — that’s pricier, slower, and less certain.
Match the model to the task. Cheap/fast for the simple things (classification, extraction), powerful for complex reasoning. Claude as the default (→ 08). Measure cost and latency, don’t guess.
Force an output contract. Structure (JSON schema / tool use), validation, and a retry on mismatch — don’t parse prose and hope. Treat LLM output as untrusted input (→ guardrails below).
Cache — A temporarily remembered result, so the same thing isn’t computed again on every request. Speeds the app up, but can be a trap: a stale cache shows old data. + idempotency. The same inputs → a stored result; don’t call the LLM in a loop over what hasn’t changed. Expensive calls are cost and latency (→ 13).
Quotas and budget. A limit per user/Endpoint — A single “address” in an API you send a request to for a specific thing (e.g. the list of orders). Apps talk through endpoints — one endpoint = one function you expose., cost monitoring, a hard quota on a paid API (→ 14).
Batch when you don’t need realtime. Enriching data offline in a pipeline with checkpoints (→ 14) beats calling the LLM live inside a user request.

Anti-patterns

🚫 An LLM for what a regex/SQL will do — cost and uncertainty where code suffices.
🚫 No output contract — you parse free prose and pray for the format.
🚫 No cache on expensive calls — the bill grows linearly with traffic.
🚫 Blind trust in the output as code/SQL — The query language for a database — the way you “ask” the database for data or change it. The universal standard for talking to a database. A badly written query can bog down the whole app./HTML — an injection vector.

Configurable chatbots — the assistant as a feature

Persona and rules = configuration, not code. Keep the system prompt, scope, tone, and boundaries as data (a file/database), versioned in git — you iterate the assistant without a deploy. A prompt “baked into” the code is dead configuration.
Grounding is mandatory. The assistant answers from your data (context from the database / RAG), not the model’s memory. Cite the source (→ 11). A hallucinated price/fact is a bug, not an “AI feature.”
Define scope and refusal. What the assistant does and doesn’t do; a safe “I don’t know” / “that’s outside my scope” instead of making things up. The model’s confidence ≠ correctness.
Streaming — Sending a large response in chunks, as it goes, instead of loading everything into memory at once. The app doesn’t choke on huge data — the user sees a result sooner. UX. A token-by-token response (SSE), conversation history, clear limits (→ 13, 14).
Security (Prompt injection — An attack on AI: someone weaves a hidden instruction into text to make the assistant act against the rules. If AI reads outside content (emails, pages), treat it as untrusted — otherwise it can be manipulated.). User content and web data are data, not instructions. Don’t perform side-effect actions from the chat without confirmation, and don’t leak secrets or the system prompt (→ 09).
Eval like tests. A golden set of control questions; a prompt change → run it through and measure regressions (→ 03). A prompt with no eval drifts silently.
Law. A disclaimer (“this is not professional advice”), GDPR — The EU’s data-protection law — how you may collect, keep and delete users’ data. It concerns every app with people’s data. Better to write in consents and retention from the start than pay fines later. for storing conversations and data (→ 09).

Anti-patterns

🚫 Persona baked into the code — every tone change is a deploy instead of a config edit.
🚫 No grounding — the assistant “confidently” invents facts/prices.
🚫 No limit/quota — chat is an open bill and an abuse vector (→ 14).
🚫 Actions from the chat without confirmation — the model does something irreversible on a user’s word.
🚫 Treating user content as instructions — prompt injection leaks the prompt/secrets.

For new projects

If the product lives on web data: start from the official source, add shape validation and an idempotent loader before you collect the first thousand records. If you use AI: an output contract and a cache from the first call, a quota from the first user. Design the assistant to be configurable and grounded from the start — the system prompt as data, a golden set as a test (→ 03), a limit as a rule (→ 14). The rest grows with the project.