The architecture behind a per-recipient interest graph — four decoupled stages, an immutably versioned taxonomy, and the failure handling that makes it survivable.
By Pavan Malpani and Sandeep Aradada
SUMMARY:
- Monolithic crawl-and-tag scripts fail at enterprise volume because one slow page stalls the whole batch.
- Four decoupled stages over a shared state layer give each step its own failure domain and its own retry.
- The taxonomy is data, not code: edited in an admin UI, versioned immutably, and pinned to every result.
- Strict output contracts and tag-ID validation stop a model from inventing categories that corrupt the taxonomy.
- Boilerplate stripping, batching and link-level caching bring the cost to about $0.004 per link.
Table of Contents
- Why a Decoupled AI Content Classification Pipeline Matters
- The Four Stages of the Auto-Tagging Pipeline
- A Taxonomy That Evolves Without Breaking History
- Predictable Output at Throughput
- Accuracy: Thresholds, Review Queues, and Bootstrapping
- Cost Engineering
- Runs on AWS, Microsoft Fabric, or Databricks
- Frequently Asked Questions
- The Bottom Line
Context. This is the engineering half of a two-part piece. The business case, the university engagement, and what the output is used for are covered in Beyond Open Rates: Turning Campaign Clicks Into Audience Intelligence. Short version: every outbound campaign link gets classified against a topic taxonomy before the send, so clicks aggregate into a per-recipient interest profile instead of a spreadsheet of URLs.
Why a Decoupled AI Content Classification Pipeline Matters
Monolithic scripts that crawl, extract, and tag content in a single execution loop are fragile by design. One slow page load or one API rate limit stalls the entire batch — and at enterprise volume, that is a when, not an if.
Splitting the work into four independent stages, coordinated through serverless orchestration and a shared lakehouse state layer, gives each stage its own failure domain. Each does one job and hands off asynchronously, so a slowdown in one never takes down the rest of the pipeline. That is the whole design thesis; everything below is a consequence of it.
The Four Stages of the Auto-Tagging Pipeline
Throughout, we will follow one real link — a feature on the engineering school’s robotics lab — from raw campaign log to tagged record.
| Stage | Compute | Input | Core job | Output |
|---|---|---|---|---|
| 1. Aggregation | Distributed ETL | Raw campaign links | Normalize and dedupe incoming links, and classify which are crawlable and/or taggable | Central state inventory |
| 2. Crawling | Headless browser (serverless) | Pending link queue | Render DOM; capture snapshots and page code | HTML & PDF artifacts |
| 3. Extraction | Text processing workers | Stored HTML artifacts | Strip boilerplate, extract core text | Clean plaintext context |
| 4. Tagging | Async inference handlers | Plaintext + taxonomy | Evaluate text against taxonomy via LLM | Tag mapping records |
1. Ingestion and state tracking
Incoming links are pulled from raw campaign links and normalized into a silver-layer state inventory table in a data lake or lakehouse. Each record tracks its own crawl, extraction, and classification status — so if a third-party service hiccups mid-run, the pipeline resumes cleanly instead of reprocessing everything from scratch. Our robotics-lab URL enters here, deduplicated against the thirteen other links in the same send and against every prior campaign that used it.
The state table is the only thing the four stages share. No stage calls another directly, which is what makes each independently deployable and independently restartable.
2. Serverless headless rendering
Standard HTTP requests fail on modern, JavaScript-heavy pages. Serverless headless browser instances (Puppeteer or Playwright) render the full DOM and capture raw HTML alongside page screenshots and PDF audit artifacts, all persisted to cloud object storage. The robotics page is client-rendered; a plain GET returns an empty shell and would have been classified as nothing at all.
The screenshot and PDF artifacts are not decoration. When someone asks six months later why a page was tagged the way it was, the page as it looked at classification time is the only honest answer — sites change underneath you.
3. Noise reduction and text parsing
Raw DOM output is full of navigation bars, ad scripts, tracking pixels, and footers — content that inflates token usage and drags down tagging accuracy. Dedicated extraction workers apply main-body parsing (via Trafilatura) to isolate clean, relevant text before it ever reaches the model. On a typical university page this removes roughly 78% of tokens before inference, which is most of the cost story further down.
4. Stateless LLM inference
Clean text is batched and sent to stateless inference workers, which pair it with a dynamic taxonomy payload. The model evaluates the content, selects the best-matching tags with confidence scores, and writes validated results back to the analytics store. The robotics page returns tagged Research › Engineering › Robotics at 0.91 and Student Life › Undergraduate Research at 0.74, pinned to taxonomy version 14.
Workers are stateless on purpose: the taxonomy arrives with the request, so scaling out is adding containers, and swapping model providers is a config change rather than a migration.
A Taxonomy That Evolves Without Breaking History
Hardcoding categories into application code or prompt strings turns into a maintenance headache the moment business requirements shift. The taxonomy is managed separately, through database-backed admin controls and immutable versioning.
- Hierarchical structure. Categories nest naturally using self-referencing keys in a relational store. Feeding the structure to the model provides lightweight RAG-style context without standing up a separate vector database — for a taxonomy of a few hundred nodes, a vector store is overhead with no payoff.
- In-memory caching. Querying the database on every inference call adds latency at scale. Classification workers cache taxonomy snapshots in memory and refresh automatically when admins publish changes.
- Immutable versioning. Every edit increments a version counter, and historical classifications retain a reference to the version used — so results stay auditable and reproducible even as categories evolve. A tag assigned in March is still explainable in November.
- Multi-model support. Link metadata and AI-generated tags are stored separately, so you can run side-by-side experiments across providers while keeping full model attribution history.
- Owned by the business. Taxonomy edits happen in an admin UI, not a deployment. At the university, advancement operations publishes revisions themselves — about four times a year, around new campaign themes and fund designations — with no engineering involvement and no reclassification of historical data.
Predictable Output at Throughput
At high throughput, prompt design has to enforce strict output contracts — otherwise downstream systems end up doing fragile, error-prone parsing, and a single chatty response poisons a batch.
- Strict tag syntax. Models return tag assignments in standardized key-value pairs with explicit delimiters.
- No conversational filler. Prompts explicitly block greetings, commentary, and explanatory prose.
- Automated validation. Every returned tag ID is checked against the active taxonomy, and anything outside approved bounds is rejected rather than stored. A model that invents a plausible-sounding category is the failure mode that quietly corrupts a taxonomy, and this is the check that stops it.
Fault tolerance
When a crawl or inference call fails from a rate limit or network drop, the framework logs the error to a dedicated audit store and retries automatically up to a configurable threshold. Records that exhaust retries move to a dead-letter queue for manual review — without stalling the rest of the batch. The state layer makes this cheap: a failed record is a status value, not a lost run.
Scale tuning
Concurrency caps control parallel worker threads to stay inside provider rate limits while maximizing throughput, and centralized JSON exclusion lists let teams bypass authentication portals, restricted domains, or retired tags with no code deployment.
Accuracy: Thresholds, Review Queues, and Bootstrapping
This is the central risk of the whole system. A pipeline that confidently mis-tags at scale is worse than no pipeline, because it produces segments that look defensible and are not.
Every classification returns a confidence score alongside its tags. Assignments above 0.80 write straight through; anything below routes to a review queue where a marketing or advancement staffer confirms or corrects the tag — and those corrections become the validation set for the next taxonomy revision. On the university engagement, 87% of links cleared the auto-accept threshold, and a human-reviewed sample of 500 pages measured 94% precision.
Bootstrapping the taxonomy is a workshop, not a guess. We start from the client’s existing content architecture — program pages, fund designations, site navigation — and reconcile it against what their campaigns actually link to, which is usually a narrower and messier set. A typical starting point is 180 nodes across three levels, revised twice in the first quarter as real click data exposes categories nobody engages with.
Accuracy tracks taxonomy quality far more closely than model choice. Overlapping or vaguely named categories produce vague tags regardless of which model you point at them, and no amount of prompt engineering fixes a taxonomy where two sibling nodes mean nearly the same thing.
Cost Engineering
Serverless rendering plus LLM inference at volume is not free, and three levers account for most of the difference between a workable unit cost and an eye-watering one:
- Boilerplate stripping. Removes roughly 78% of tokens before they are ever billed — the single largest lever, and it improves accuracy at the same time.
- Batching. Bundling multiple pages per inference call amortizes fixed prompt overhead — the taxonomy payload is sent once per batch, not once per page.
- Caching against the link. A URL reused across six campaigns is classified once, with a 30-day re-crawl cadence for content that changes.
Net result at the university’s volume: approximately $0.004 per link at steady state, or under $20 a month for an average of 15 links per send (with some duplicates within and across sends), around 300 sends a month, and a 30-day recrawl — dominated by inference rather than by rendering.
Runs on AWS, Microsoft Fabric, or Databricks
The university implementation runs on an AWS data lake. None of that is load-bearing. The pattern is four decoupled stages over a medallion-structured lakehouse, and it ports directly to whichever platform you have already standardized on.
If you are already standardized on Microsoft Fabric, the serving row is arguably the better ending: a Direct Lake semantic model over the interest matrix lets advancement and marketing staff explore their own segments in Power BI with no copy step and no extract refresh. The interest graph stops being a pipeline output and becomes something the business queries directly — which is usually where the adoption problem gets solved.
Frequently Asked Questions
Why not use embeddings and a vector database instead of an LLM with a taxonomy?
Embeddings give you similarity, not a governed label. The requirement here is that every tag maps to a node the business defined, can be audited, and can be pinned to a version — which is a classification problem, not a retrieval one. For a few hundred taxonomy nodes, passing the hierarchy in-context outperforms a vector store on both accuracy and operational overhead.
How do you stop the model inventing categories?
Validation on the way in, not trust on the way out. Every returned tag ID is checked against the active taxonomy and rejected if it falls outside approved bounds. Combined with strict output syntax and no-filler prompting, invented categories never reach storage.
What happens when the taxonomy changes — do you reclassify everything?
No. Historical classifications keep a reference to the version they were judged under, so old results stay reproducible and explainable. Reclassification is a deliberate backfill you choose to run, not a side effect of an edit.
How do you handle pages behind authentication or paywalls?
Centralized JSON exclusion lists let teams bypass authentication portals and restricted domains without a code deployment. Links that cannot be rendered exhaust their retries and land in the dead-letter queue for review rather than silently classifying as empty.
What does it cost to run at scale?
Approximately $0.004 per link at steady state, dominated by inference rather than rendering. Boilerplate stripping, batching, and link-level caching account for most of the difference between that figure and a naive implementation, which typically runs several times higher.
The Bottom Line
The hard part of link auto-tagging is not the model call. It is the state management, the versioning, and the validation that let tens of thousands of classifications stay correct, explainable, and affordable as content and taxonomies move underneath you.
Want to see it against your own content? Send us 100 links from your last campaign. We will classify them against a draft taxonomy and return the tagged output with confidence scores, so you can judge precision on pages you actually know. Get in touch.
Start here instead — Part 1: Beyond Open Rates: Turning Campaign Clicks Into Audience Intelligence. What the pipeline is for: the university engagement, the recipient interest matrix, how segments get written back into the sending platform, and the governance model behind behavioral profiling.
Related service offerings: Enterprise AI Solutions, Microsoft Fabric Consulting, and Databricks Consulting.