• What is the unit of retrieval? (sentences, paragraphs, chunks, files, documents)
• Does the application need semantic similarity, exact-match search, or both?
• What metadata filters matter? (tenant, date, repo, workflow state, permissions)
• Is the product fundamentally a search engine, a database with search, or a retrieval substrate?
• Is the workload closer to consumer AI search, enterprise document retrieval, codebase chunk retrieval, or multimodal retrieval?

A typical stack looks like:

content
  ↓
parsing / chunking / field selection
  ↓
embeddings
  ↓
retrieval
  + lexical retrieval
  + vector retrieval
  + metadata filters
  + reranking
  + application logic
  ↓
final results

Each layer has its own decisions. In practice, the right architecture depends less on the phrase “vector database” and more on the shape of the workload.

Vector databases: role in the stack

An embedding model maps a raw object — a sentence, paragraph, image, or code chunk — into a point in high-dimensional space. Nearby points correspond to similar meaning.

A vector database stores and indexes those vectors so that approximate nearest-neighbor (ANN) search is practical at production scale. It typically provides:

• storage for vectors and IDs
• ANN indexes
• metadata filters
• updates and deletes
• multitenancy or namespace isolation
• replication, scaling, and operations

That makes a vector database more than an in-process nearest-neighbor library but less than a complete search product. It handles one stage of the pipeline well. The rest — chunking, lexical search, reranking, and application logic — lives outside it.

An ANN index (approximate nearest-neighbor index) is the data structure that makes vector lookup fast at scale. Given a query embedding, the goal is to retrieve the top-k vectors closest under a distance metric (often cosine distance or L2). Doing that exactly would require comparing the query to every stored vector, which is too slow and too memory-heavy when there are millions or billions of points in hundreds or thousands of dimensions. ANN indexes avoid scanning the full corpus by organizing vectors (for example via clustering, graphs, hashing, or compressed representations) so the engine visits only a small candidate set. The tradeoff is explicit: recall (how often the true nearest neighbors appear in the top-k) versus latency, memory, and ingest cost — tuned with index parameters and revisited as data and traffic grow.

For many applications, the decisions around chunking, hybrid retrieval, filters, and reranking matter more than the specific ANN backend.

Embeddings: inputs, outputs, geometry

Think of an embedding model as a function that maps a raw object to a point in d-dimensional space.

Typical pipeline:

Raw input → tokenizer / preprocessing → neural encoder → embedding vector
                                                      ↓
                    search · clustering · classification · recommendation · RAG

You rarely inspect the coordinates directly. What matters is relative position:

• similar meaning → nearby vectors
• unrelated meaning → distant vectors

Training usually tries to make this geometry useful for the task by pulling related examples together and pushing unrelated examples apart.

Common training patterns

• Masked / causal language modeling — predict missing or next tokens; useful representations emerge in the hidden states
• Contrastive learning — positive pairs should be close, negatives far apart
• Supervised classification with an embedding bottleneck — encoder → embedding → classifier
• Triplet loss — anchor, positive, negative; enforce $d(\text{anchor}, \text{positive}) \ll d(\text{anchor}, \text{negative})$

For retrieval-focused models, hard negatives usually matter a lot more than easy random negatives. For example, “Python list comprehension” vs “Python for loops tutorial” teaches a retrieval model much more than “Python list comprehension” vs “banana smoothie”.

Choosing an embedding model

Choosing an embedding model is not just a benchmark exercise. It depends on the shape of the retrieval problem.

The main decision axes are:

• Modality — is the corpus text-only, image-heavy, or truly multimodal?
• Task — is the goal retrieval, clustering, classification, recommendation, or reranking support?
• Query/document asymmetry — should queries and stored documents use different embedding modes?
• Domain — is the corpus general text, code, legal, finance, biomedical, or something else with specialized language?
• Language coverage — is the corpus multilingual, or is cross-lingual retrieval important?
• Dimension and storage cost — higher-dimensional vectors may improve quality, but they also increase storage, bandwidth, and retrieval cost
• Latency, privacy, and deployment constraints — can the embeddings be generated through a hosted API, or do they need to run in a private environment?

A practical sequence is:

modality → task → domain → language coverage → query/document asymmetry → cost/latency → provider choice

For many teams, the right first move is to start with a strong general-purpose retrieval embedding model, measure it on a realistic evaluation set, and only then decide whether a domain-specific or multimodal model is justified.

One subtle but important point: query embeddings are not always the same as document embeddings. Cohere’s Embed API explicitly distinguishes search_query and search_document. Vertex AI supports task-type-aware embeddings for document retrieval, question answering, fact verification, clustering, and more.

Voyage AI’s Voyage 4 announcement describes asymmetric retrieval: the Voyage 4 models share one compatible embedding space, so vectors produced by different models (for example, queries with voyage-4-lite against documents indexed with voyage-4-large) still match under the same similarity search. That is useful when embedding the corpus is a one-time or infrequent cost but embedding queries is continuous at serving time—you can favor a larger model for stored documents and a smaller, faster model for live queries, trading a little operational complexity for better accuracy per dollar and lower query latency. The Voyage embeddings API still exposes query vs. document input_type for retrieval-oriented behavior.

Domain-specific and task-specific embeddings

A strong general-purpose embedding model is often the right place to start. But it is not always the right place to stop.

Some retrieval problems benefit from domain-specific embeddings because the meaning of similarity is different in different fields. Code, legal documents, financial text, and biomedical corpora often contain specialized language, structure, and relevance criteria that a generic model may not represent as well.

Task-specific behavior matters too. A model optimized for document retrieval may not be ideal for clustering, and a model optimized for queries may not be ideal for stored corpus documents.

In practice, the progression often looks like this:

general retrieval model
→ realistic evaluation set
→ identify failure cases
→ domain-specific or task-specific model if needed

That sequence is usually better than prematurely fine-tuning or choosing a niche model before understanding the retrieval workload. Voyage explicitly recommends domain-specific models for areas like law, finance, and code, while Cohere and Vertex AI expose task-aware embedding modes for retrieval and related use cases.

Embedding provider cheat sheet

The choice of vector database is only half the story. The embedding provider matters just as much.

A useful way to think about providers is:

• OpenAI — strong general-purpose baseline
• Cohere — retrieval-first and RAG-friendly
• Voyage — retrieval specialist, especially for domain-specific workloads
• Vertex AI — task-aware and multimodal, especially attractive inside Google Cloud

Provider choice is not only about benchmark quality. It also depends on deployment model, privacy requirements, batch throughput, dimensionality control, multimodal support, and compliance constraints.

Embedding models are additionally available as open weights on Hugging Face for self-hosted inference, fine-tuning, or experimentation alongside the hosted providers above; which checkpoint to use still depends on modality, languages, license, and your own retrieval evaluation rather than any universal pick.

After training: how vectors are used

Vectors support several kinds of applications:

• semantic search — nearest-neighbor retrieval
• classification — linear probes or downstream heads
• clustering — grouping similar items
• recommendation — users and items embedded in a shared space
• RAG — retrieve chunks to condition an LLM
• code retrieval — retrieve relevant files, symbols, or chunks
• multimodal search — align text and images or other modalities

In many systems, vectors are the first-stage retriever, not the final answer generator.

Multimodal retrieval: when OCR is not enough

Many retrieval systems are described as “multimodal,” but there are really three different cases.

1. Text-only retrieval

This is the simplest case. The corpus is already text, or can be reduced to text without losing much meaning.

Examples:

• plain documents
• knowledge bases
• contracts
• source code
• emails

2. OCR-first retrieval

This is common for scanned PDFs and forms. The retrieval pipeline extracts text with OCR, then treats the result as ordinary text retrieval.

This works well when most of the important information is still captured in words.

3. True multimodal retrieval

This is needed when the visual structure itself carries meaning, not just the text.

Examples:

• screenshots
• slide decks
• diagrams
• tables where layout matters
• charts and figures
• image-heavy PDFs
• document page images
• search by screenshot or image

In these settings, OCR alone can lose important information. A multimodal embedding model can place text, images, and mixed inputs into a shared retrieval space.

This matters in practice because many enterprise corpora are only partially textual. A text-heavy invoice workflow may be well served by OCR plus text embeddings. A slide deck, dashboard screenshot, or visually complex form may require true multimodal retrieval.

A useful rule of thumb is:

if the meaning survives text extraction → OCR-first may be enough
if the meaning depends on layout, figures, or images → consider multimodal embeddings

Vertex AI multimodal embeddings generate vectors from image, text, and video in a shared semantic space. Voyage multimodal embeddings support text and content-rich images such as figures, screenshots, slide decks, and document images. Cohere embeddings also support text, image, and mixed inputs for newer embedding models.

Chunking and indexing strategy

Retrieval quality is often dominated by what gets indexed and how it gets chunked.

Questions to answer early:

• What is the retrieval unit? A sentence, paragraph, page, section, table, file, or whole document?
• Should chunks overlap, or should they be strictly disjoint?
• Should metadata such as title, section name, page number, repo path, or document type be copied into every chunk?
• Should some structures — tables, forms, headers, footnotes, captions, code blocks — be indexed separately?
• Is there a parent-child relationship between chunks and larger source documents?

The right strategy depends on the workload:

• enterprise document retrieval often benefits from section-aware, field-aware, or table-aware chunking
• code retrieval often benefits from symbol-aware or file-aware chunking
• slide decks and screenshots may require page-level or multimodal chunking
• RAG often benefits from chunks that are small enough to retrieve precisely but large enough to preserve local context

A useful heuristic is:

chunk for the unit you want to retrieve, not the unit you happen to store

This is one reason why the retrieval stack is broader than the vector database. The database stores the vectors, but chunking determines what those vectors mean.

Precision, recall, and the practical knobs

Precision — among returned hits, how many are relevant?

Recall — among all relevant items in the corpus, how many appear in the result set?

Usually there is tension between them:

• stricter thresholds means: ↑ precision, ↓ recall
• broader retrieval means: ↑ recall, ↓ precision

In practice, the biggest quality levers are often:

1. strong evaluation sets
2. good chunking
3. metadata modeling
4. hard negatives
5. hybrid lexical + vector retrieval
6. reranking
7. threshold and top-K tuning
8. domain fine-tuning

A good production recipe is usually:

eval set → chunking → stronger embeddings → filters → reranker → threshold / K → domain tuning

before exotic modeling.

BM25 and lexical search

BM25 ranks documents for a keyword query. It rewards:

• rare terms more than common terms
• multiple mentions, but with diminishing returns
• shorter, more focused documents over long noisy ones

For each query $q$ and document $d$:

where $t \in q$ are query terms, $n_t$ is the number of documents containing $t$, $f(t,d)$ is term frequency, $\lvert d \rvert$ is document length, $N$ is corpus size, $\text{avgdl}$ is average document length across the corpus, and $k_1 \approx 1.2$–$2.0$ (term-frequency saturation) and $b \approx 0.75$ (length normalization) are the tunable parameters.

BM25 vs vectors

• BM25 is strong on exact terms, IDs, names, and phrases
• vectors are strong on semantic similarity and paraphrase
• hybrid often works best in production

This is especially true in enterprise systems where both exact identifiers and fuzzy semantic matches matter.

The real retrieval stack: lexical, vector, hybrid, reranking

Modern retrieval systems usually fit one of four patterns.

1. Pure lexical search

Best when exact token matching dominates:

• product codes
• case IDs
• SQL keywords
• API names
• legal citations

2. Pure vector search

Best when semantic similarity dominates and exact tokens matter less:

• recommendations
• some semantic FAQ lookup
• some multimodal applications

3. Hybrid search

Best when both matter:

• enterprise documents
• code search
• support knowledge bases
• RAG over heterogeneous corpora

4. Two-stage retrieval

Often best in serious systems:

retriever → top-K candidates → reranker → final results

The retriever may be lexical, vector, or hybrid; the reranker adds precision.

Not all modern retrieval is just “dense vectors vs BM25.” Some systems also use sparse learned retrieval, late-interaction models such as ColBERT-style designs, or dedicated rerankers when ranking quality matters more than keeping the first-stage index simple.

ANN index choices and tradeoffs

Approximate nearest-neighbor search speeds up retrieval by giving up some exactness for much lower latency and cost.

The central tradeoffs are:

• exact vs approximate search — exact search gives maximum recall but can be too slow or expensive at scale
• recall vs latency — more aggressive ANN settings are faster but may miss some true nearest neighbors
• memory vs compression — some index types are memory-heavy, others compress vectors more aggressively
• update cost vs query cost — some ANN structures are friendlier to frequent updates than others

A few common patterns:

• HNSW — strong quality and very common in production vector search systems
• IVF / PQ-style compression approaches — attractive when memory efficiency matters more than maximum recall
• exact search — still useful for smaller corpora, evaluation, and some latency-insensitive workflows

In practice, the right ANN choice depends on corpus size, update rate, latency budget, and the acceptable recall loss.

The vector database landscape

The term vector database is used loosely, but the landscape actually has three broad categories.

1. Dedicated vector databases / vector engines

These are built primarily around vector storage and similarity search:

• Pinecone
• Milvus
• Qdrant
• Weaviate
• Turbopuffer
• Chroma
• LanceDB

These vary in maturity, operational model, and how strongly they also support lexical or hybrid search.

2. Search engines with strong vector support

These are broader search/ranking systems that also handle vectors well:

• Vespa
• OpenSearch
• Elasticsearch

These often make more sense when search, ranking, and serving are core to the product.

3. General databases with vector support

These keep vectors close to operational data:

• MongoDB Vector Search
• Postgres + pgvector

These are attractive when data locality and operational simplicity matter more than having a specialized search stack.

A conceptual map of the major systems

Pinecone, Milvus, Qdrant

These are easiest to think of as dedicated vector database products.

They are often chosen when the main need is:

• vector similarity search
• metadata filtering
• scalable ANN
• production operational support

Weaviate

Weaviate is best thought of as a vector-native / AI database with strong built-in support for:

• vector search
• keyword search
• hybrid search

That makes it a strong middle ground between “pure vector DB” and “full search engine.”

See also: Weaviate concepts

Turbopuffer

Turbopuffer is best thought of as a retrieval substrate optimized for large-scale search over vectors and metadata, with support for full-text and hybrid behavior as well.

It is especially interesting for workloads with:

• many isolated namespaces
• high update rates
• lots of small chunks
• low-latency nearest-neighbor retrieval

Vespa

Vespa is not best understood as a vector database. It is an open-source search + ranking + serving engine.

Its strength is not just storing vectors, but combining:

• lexical retrieval
• vector retrieval
• filters
• business logic
• multi-stage ranking
• serving logic

Vespa is a strong choice when relevance engineering is central.

OpenSearch and Elasticsearch

OpenSearch and Elasticsearch are not “pure vector DBs” either. They are Lucene-based distributed search engines that support:

• BM25 full-text search
• filters and aggregations
• vector search
• hybrid search patterns

They are especially strong when traditional search features matter alongside vectors.

MongoDB Atlas Search and Vector Search

MongoDB provides both:

• Atlas Search for Lucene-backed lexical search
• Atlas Vector Search for semantic nearest-neighbor retrieval

These are strongest when MongoDB is already the system of record and the goal is to keep retrieval close to application data and aggregation pipelines.

Postgres + pgvector

This is often the simplest option when:

• the app already uses Postgres
• scale is moderate
• operational simplicity matters
• vector retrieval is important, but not the entire product

It is frequently a very good default for early-stage products.

How different products use different retrieval systems

The best way to understand the landscape is by application shape.

Perplexity: search and ranking are the product

Perplexity publicly describes using Vespa to power AI search at scale.

That makes sense because Perplexity’s problem is not just semantic retrieval. It is closer to:

• search engine retrieval
• ranking
• freshness
• structured filtering
• serving at scale

This is a natural fit for a search-and-ranking engine rather than a pure vector DB.

Cursor: code retrieval is a chunked nearest-neighbor problem

Cursor publicly documents using Turbopuffer for codebase indexing: chunk files, embed them, store vectors plus obfuscated metadata, then perform nearest-neighbor search at inference time.

This also makes sense. Cursor’s problem looks like:

• lots of small code chunks
• high churn
• many user/repo namespaces
• metadata filters such as path and line range
• extremely fast retrieval

That shape favors a fast retrieval substrate over a heavy search-engine stack.

MongoDB: integrated database + search

MongoDB’s model is different. It says: keep your operational data in MongoDB, and add lexical and vector retrieval in the same platform.

This is strongest when the system already needs:

• document storage
• app data
• workflow state
• search
• vector retrieval
• filters and aggregation

with minimal extra infrastructure.

DocRouter-style document retrieval

A DocRouter-style workload is usually not just a vector search problem. It is a document retrieval and workflow problem.

Typical needs include:

• exact IDs and exact phrases
• semantic similarity
• metadata filters
• grouped or field-aware retrieval
• hybrid search
• reranking
• explainability
• workflow and permission logic

That usually means the right architecture is:

lexical retrieval
+ vector retrieval
+ metadata filters
+ reranking
+ application logic

not just “pick a vector DB.”

Workload tilt: DocRouter-style vs code-editor retrieval

Feature matrix (high level)

Choosing the right system by application need

Choose MongoDB Search / Vector Search when

• MongoDB is already the system of record
• you want minimal infrastructure sprawl
• metadata-heavy filtering and app integration matter
• retrieval is important, but not a standalone serving product

Choose Weaviate when

• retrieval is central to the application
• you want strong keyword + vector + hybrid search
• you want a dedicated retrieval database without going all the way to a search-engine platform

Choose Vespa when

• search and ranking are core differentiators
• you want multi-stage ranking and richer relevance engineering
• the product looks more like search/recommendation/serving than a CRUD app with vectors

Choose Turbopuffer when

• the workload is mostly fast chunk retrieval
• there are many namespaces or tenants
• metadata filters matter
• the application looks like a code assistant, retrieval substrate, or large-scale vector index

Choose Pinecone, Qdrant, or Milvus when

• you want a dedicated vector database
• the main need is vector retrieval plus filtering
• you do not need the full complexity of a search-engine platform

Choose pgvector when

• you already use Postgres
• scale is moderate
• simplicity matters
• vector retrieval is important, but not the center of the platform

Choose OpenSearch or Elasticsearch when

• you already need classic search-engine features
• lexical search remains central
• vectors are an addition to a search stack, not the entire product

Metrics that matter

Beyond raw ANN benchmarks, what actually matters depends on the application.

Retrieval metrics

• Precision@K — of the top K results returned, what fraction are actually relevant?
• Recall@K — of all relevant items in the corpus, what fraction appear in the top K?
• MRR (Mean Reciprocal Rank) — averages the reciprocal of the rank of the first relevant result across queries
• MAP (Mean Average Precision) — summarizes ranking quality and coverage across recall levels
• NDCG (Normalized Discounted Cumulative Gain) — rewards highly relevant results appearing early and supports graded relevance

System metrics

• latency
• throughput
• freshness
• update cost
• filter correctness
• multitenancy behavior
• operational burden

Application-specific metrics

• document systems: exact identifier retrieval, section relevance, workflow correctness
• code assistants: chunk relevance, namespace isolation, freshness after edits
• consumer search: ranking quality, freshness, personalization, serving latency

Failure modes

Common retrieval failures include:

• easy negatives instead of hard negatives
• bad chunking
• poor metadata modeling
• domain mismatch
• ignoring lexical exact-match needs
• too much faith in vector similarity alone
• evaluating only on easy benchmarks
• no reranking
• no permission or tenant filtering

Many disappointing “vector database” results are actually failures of the surrounding retrieval design.

Practical takeaway

The most important conceptual point is:

A vector database is usually not the right abstraction to optimize first.

For most real systems, the better question is:

What retrieval architecture does this application need?

• For consumer AI search, search and ranking engines like Vespa may be the right center of gravity.
• For code retrieval, fast namespace-heavy ANN systems like Turbopuffer may be a better fit.
• For application-integrated retrieval, MongoDB Search / Vector Search or pgvector may be simplest.
• For semantic + hybrid retrieval as a product, Weaviate, Qdrant, Pinecone, or Milvus may be the right class.
• For enterprise document retrieval, the answer is often hybrid lexical + vector + filters + reranking, not just “choose a vector DB.”

At DocRouter.AI we treat document pipelines as retrieval-shaped problems: exact fields, semantics, metadata, and workflow context—not “vector search alone.”

Tags, prompts, and structured extraction are how teams operationalize that stack on real documents.

If you’re new to Kubernetes, the Kubernetes for Docker Users primer covers Pods, Deployments, Services, PVCs, and Helm basics. For packaging and GitOps, see Kubernetes Packaging and Deployment.

Why not Bitnami?

The obvious choice for an in-cluster MongoDB is the Bitnami chart, which is widely used and simple to install. The problem is vector search. Applications that need semantic search or Atlas-style indexes require the mongot process — a sidecar that runs alongside mongod and handles full-text and vector indexes. Bitnami deploys a plain community MongoDB without mongot, so Atlas Search is simply not available.

The only supported path to mongot in a self-hosted environment is the MongoDB Kubernetes Operator, which introduces the MongoDBCommunity and MongoDBSearch custom resources. The operator manages the StatefulSet, replica set initialization, user creation, and TLS — and, when MongoDBSearch is enabled, injects the mongot sidecar with the right configuration.

Our chart wraps the operator’s CRDs with sensible defaults and a single helm upgrade --install interface, so operators don’t need to understand the operator’s internals to get a working cluster. You can run MongoDB with or without search; if you don’t need vector or full-text search, you can disable the mongot sidecar and save resources.

Two-phase install

mongot requires a running, authenticated replica set to connect to — it cannot start on a fresh cluster. The install therefore happens in two phases:

# Phase 1: bring up the replica set without search
helm upgrade --install mongodb oci://ghcr.io/analytiq-hub/mongodb-atlas-local \
  --version 2.0.1 --namespace mongodb \
  --set mongodb.adminPassword="..." \
  --set mongodb.appUser.password="..." \
  --set search.enabled=false

# Wait for replica set Ready
kubectl wait --for=condition=ready pod -l app=mongodb-mongodb-atlas-local \
  -n mongodb --timeout=300s

# Phase 2: enable search
helm upgrade mongodb oci://ghcr.io/analytiq-hub/mongodb-atlas-local \
  --version 2.0.1 --namespace mongodb --reuse-values \
  --set search.enabled=true

Attempting a single-phase install with search.enabled=true results in mongot crash-looping because the replica set isn’t ready to accept its connection.

Node sizing for stateful workloads

Adding MongoDB changes the cluster sizing arithmetic considerably. Each replica pod runs two containers: mongod (500m CPU, 400Mi) and mongodb-agent (500m CPU, 400Mi), plus a mongot sidecar (250m CPU, 250Mi) when search is enabled. A 3-replica set therefore requests ~2.25 vCPU and ~3.15 Gi of memory, on top of whatever other workloads you run.

The scheduler must fit the entire pod on one node. On a cluster with two t3.medium nodes (2 vCPU / 4 Gi each), if existing workloads already consume ~1.7 vCPU in requests, there may be ~2.2 vCPU free across both nodes — but never more than ~740m on a single node. A MongoDB pod that needs ~750m CPU cannot be scheduled. Adding a third node (or sizing nodes with enough headroom) resolves it.

The practical lesson: account for stateful pods when sizing the initial node group, or ensure the autoscaler can provision new nodes quickly enough not to block workloads.

EBS CSI Driver and the gp2 trap (EKS)

When we added MongoDB to an EKS cluster, PVCs sat in Pending indefinitely with the error:

no persistent volumes available for this claim and no storage class is set

EKS creates a gp2 StorageClass by default, but it has two problems. First, it is not marked as the default class — PVCs with an empty storageClassName get no provisioner assigned. Second, and more importantly, gp2 uses the legacy in-tree kubernetes.io/aws-ebs provisioner, which was removed in Kubernetes 1.27. On EKS 1.35, it is simply gone.

The fix is to create a gp3 StorageClass backed by the EBS CSI driver (ebs.csi.aws.com) and mark it as the cluster default:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gp3
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
parameters:
  type: gp3
  encrypted: "true"

WaitForFirstConsumer is important — it delays EBS volume creation until the pod is actually scheduled to a node, which ensures the volume is created in the correct availability zone. allowVolumeExpansion: true enables online resizing without pod restarts.

Provision this StorageClass (and the EBS CSI driver) via Terraform or your preferred IaC so new clusters get it automatically.

Summary

We use this chart for Doc Router and other applications that need MongoDB with vector search. For the full Doc Router deployment story (Helm chart, workers, CI/CD, multi-cloud), see Deploying Doc Router on Kubernetes.

Andrei Radulescu-Banu is the founder of DocRouter.AI (document processing with LLMs) and SigAgent.AI (Claude Agent monitoring). His company AnalytiqHub.com provides consulting services for cloud and AI engineering.

If you’re new to Kubernetes, start with Kubernetes for Docker Users: A Practical Primer, which covers the core concepts — Pods, Deployments, Services, Namespaces, Secrets, PVCs, Helm, and Kind — before diving into this post. For packaging and GitOps (Kustomize, Helm, Flux), see Kubernetes Packaging and Deployment.

Why Kubernetes?

Doc Router was originally deployed using Docker Compose, which worked well for single-node setups. As we started onboarding enterprise customers with availability and scalability requirements, we needed:

• Horizontal scaling — multiple replicas behind a load balancer
• Automated failover — pods restarted on failure without manual intervention
• Rolling deployments — zero-downtime upgrades
• Resource isolation — CPU and memory limits per component

Architecture

The production deployment consists of two main workloads:

• Frontend — Next.js server (SSR + API routes via NextAuth)
• Backend — FastAPI application with embedded background workers

Both run as Kubernetes Deployments behind a shared nginx ingress with TLS terminated by cert-manager (Let’s Encrypt).

MongoDB can run outside the cluster (MongoDB Atlas) or in-cluster via our mongodb-atlas-local Helm chart — see Self-Hosted MongoDB on Kubernetes with Atlas Search for the install guide. AWS S3 remains an external dependency.

Helm Chart

We packaged the deployment as a Helm chart (deploy/charts/doc-router) published to GitHub Container Registry (ghcr.io) as an OCI artifact. The chart is versioned independently of the Docker images, so we can update deployment configuration without rebuilding the application.

Key design decisions:

• Single values.yaml with sensible defaults — operators override only what differs per cluster
• ConfigMap for non-secret config — NEXTAUTH_URL, FASTAPI_ROOT_PATH, worker count, S3 bucket
• Kubernetes Secret for credentials — MongoDB URI, API keys, NextAuth secret — created by the deploy script, never stored in the chart
• Ingress host derived from APP_HOST — a single variable drives the entire URL configuration

Choosing a Container Registry

We evaluated two natural options: Amazon ECR (since we’re already on AWS/EKS) and GitHub Container Registry (ghcr.io) (since our source is on GitHub).

ECR has one significant operational advantage for EKS: nodes authenticate via IAM role, so there is no image pull secret to manage. Costs are low — $0.10/GB stored, with no data transfer charge for pulls within the same AWS region. However, ECR is tightly coupled to AWS. A second deployment on Digital Ocean or a customer’s on-premises cluster would need separate registry credentials and mirroring, making it a poor fit for a multi-cloud or self-hosted product.

ghcr.io is cloud-neutral — any cluster anywhere can pull images with a single token. It integrates naturally with GitHub Actions (the GITHUB_TOKEN secret already has packages: write permission), so publishing images is zero-configuration. The chart package also appears directly on the repository’s GitHub page alongside the source code and releases, which is the right home for an open-source project.

The catch: ghcr.io packages are private by default for organizations, and GitHub’s free tier includes only 500 MB storage and 1 GB transfer per month. For clusters that pull large images repeatedly, those limits are reached quickly. Making packages public eliminates the cost entirely, but requires an organization admin to enable public package creation in the org settings — it is disabled by default.

We chose ghcr.io and made our packages public. The images contain no secrets — only application code — so public visibility is appropriate and keeps infrastructure simple. Clusters pull anonymously with no credentials required.

For customers who need private images (for example, an enterprise build with proprietary integrations), the REGISTRY_PROVIDER variable in the overlay .env file can be switched to aws or do to use ECR or Digital Ocean Container Registry instead, with registry login handled automatically by the deploy scripts.

Merging Workers into FastAPI

The original architecture ran the background workers (OCR, LLM, KB indexing, webhooks) as a separate process alongside uvicorn. In Kubernetes, this meant each backend pod ran two Python processes, consuming ~375 MB of memory.

We merged the workers into the FastAPI lifespan using asyncio.create_task:

@asynccontextmanager
async def lifespan(app):
    # startup
    worker_tasks = start_workers(n_workers)
    yield
    # shutdown
    for task in worker_tasks:
        task.cancel()
    await asyncio.gather(*worker_tasks, return_exceptions=True)

This halved per-pod memory usage (~190 MB) and eliminated the process management overhead. The workers share the same event loop as the API, which is safe because all worker I/O is already async.

Worker Polling Optimization

With multiple replicas, each pod runs a full set of worker coroutines polling MongoDB queues. At idle with 4 workers per pod, that was ~80 MongoDB queries per second cluster-wide.

We implemented exponential backoff with shared state across parallel workers:

_queue_idle_sleep: dict[str, float] = {}  # shared across all workers on a queue

# on idle: back off
sleep = _queue_idle_sleep.get("ocr", POLL_MIN_SLEEP)
await asyncio.sleep(sleep)
_queue_idle_sleep["ocr"] = min(sleep * 2, POLL_MAX_SLEEP)

# on message found: reset for all workers on this queue
_queue_idle_sleep["ocr"] = POLL_MIN_SLEEP

This reduces idle polling to near-zero while keeping response latency low when work arrives.

Graceful Shutdown

When Kubernetes scales down a pod (HPA scale-in or rolling update), it sends SIGTERM. We needed in-flight jobs to be marked as failed rather than silently abandoned.

Since workers are asyncio tasks, cancellation arrives as asyncio.CancelledError — a BaseException, not caught by except Exception. We added explicit handling in each worker:

try:
    await ad.msg_handlers.process_ocr_msg(analytiq_client, msg)
except asyncio.CancelledError:
    logger.warning(f"Worker cancelled mid-flight on msg {msg.get('_id')}, marking failed")
    await ad.queue.delete_msg(analytiq_client, "ocr", str(msg["_id"]), status="failed")
    raise  # allow the task to actually cancel

The failed job can then be retried on another pod.

Database Migrations as a Helm Pre-Upgrade Hook

Running database migrations safely in a multi-replica environment requires that migrations complete before any new application code starts serving traffic. In Docker Compose this is handled by startup ordering, but in Kubernetes rolling updates, new pods can start before old ones are gone — with no guarantee about migration timing.

We solved this with a Helm hook Job that runs migrate.py using the same backend image, annotated to execute before the upgrade rolls out:

annotations:
  "helm.sh/hook": pre-upgrade,pre-rollback
  "helm.sh/hook-weight": "-5"
  "helm.sh/hook-delete-policy": hook-succeeded,before-hook-creation

The pre-upgrade hook ensures migrations run and complete successfully before Helm touches any Deployment. If the migration Job fails, Helm aborts the upgrade entirely — the old version keeps running. hook-delete-policy: hook-succeeded cleans up the completed Job automatically, keeping the namespace tidy. The before-hook-creation policy ensures the old Job is removed if a previous run left one behind.

One subtlety: at pre-upgrade time, the ConfigMap has not yet been updated by Helm (hooks run before regular resources). The migration Job therefore mounts only the Secret — which contains MONGODB_URI — and not the ConfigMap:

envFrom:
- secretRef:
    name: doc-router-secrets
# ConfigMap intentionally omitted — not yet updated at hook time

This means migrate.py must be written to need only the database connection string, with no dependency on application config values.

The result is a safe, atomic upgrade sequence: migrate → roll out new pods → terminate old pods — with automatic rollback if the migration fails.

HPA Tuning

We configured Horizontal Pod Autoscaler on the backend with both CPU and memory targets:

metrics:
- type: Resource
  resource:
    name: cpu
    target:
      type: Utilization
      averageUtilization: 80
- type: Resource
  resource:
    name: memory
    target:
      type: Utilization
      averageUtilization: 80

A subtle issue: HPA scale-down uses ceil(currentReplicas × currentUtil / targetUtil). With 5 pods at 72% memory utilization against an 80% target, ceil(5 × 72/80) = ceil(4.5) = 5 — the ceiling arithmetic created a deadlock where the cluster could never scale below 5 pods.

The fix was increasing the memory request from 512 Mi to 768 Mi. After the worker merge reduced actual usage to ~190 MB, utilization dropped to ~25% — well below the threshold — and the cluster scaled back down to the minimum of 2 replicas.

Environment Configuration

Next.js NEXT_PUBLIC_* variables are baked into the browser bundle at build time, not injected at runtime. This caused a subtle bug: our local .env.local file set NEXT_PUBLIC_FASTAPI_FRONTEND_URL=http://127.0.0.1:8000. Because .env.local wasn’t listed in .dockerignore, it was copied into the Docker build context and read by Next.js during npm run build — silently overriding the intended production value and baking the localhost URL into every image.

We fixed this in two steps:

1. Exclude all .env.* files from the Docker build context by adding **/.env.* to .dockerignore, so local development env files can never leak into images.

2. Remove NEXT_PUBLIC_FASTAPI_FRONTEND_URL entirely. Rather than baking an absolute URL into the bundle, the frontend now always calls /fastapi — a relative path that works from any hostname. Next.js rewrites proxy /fastapi/:path* to the backend service URL at the server layer:

// next.config.mjs
async rewrites() {
  return [{
    source: '/fastapi/:path*',
    destination: `${process.env.FASTAPI_BACKEND_URL}/fastapi/:path*`,
  }];
}

FASTAPI_BACKEND_URL is a server-side runtime variable (not NEXT_PUBLIC_) pointing to the in-cluster backend service (http://backend.<namespace>.svc.cluster.local:8000). It is never exposed to the browser. The result is a truly environment-agnostic frontend image that requires no rebuild when moving between clusters.

CI/CD Pipeline

Structure

We use three GitHub Actions workflows:

• backend-tests.yml — runs Python tests against a local MongoDB Atlas instance (with vector search via mongodb-atlas-local) plus TypeScript tests. Triggered by workflow_call or workflow_dispatch.
• frontend-build.yml — runs npm run build for the Next.js frontend. Also triggered by workflow_call or workflow_dispatch.
• ci.yml — runs both test workflows on every pull request to main.
• release.yml — triggered on semver tags (v[0-9]*.[0-9]*.[0-9]*). Runs both test workflows first, then builds and pushes Docker images if they pass.

Why semver tags, not branch pushes

An early version of the pipeline ran tests on every push to main and triggered builds from there. This caused two problems:

1. Tests ran twice per release — once on the branch push, once triggered by the tag.
2. The tag trigger didn’t wait for tests — if a tag was pushed immediately after a commit, the build could race ahead of a still-running test run.

The current design avoids both: release.yml is only triggered by a semver tag, and the build-push job declares needs: [test-backend, test-frontend], so Docker images are never built unless all tests pass on that exact commit. Tests run exactly once per release.

The ci.yml workflow handles the PR gate separately — developers get test feedback on their branch without triggering a build.

Reusable test workflows

Making the test workflows workflow_call-able (rather than duplicating the job definitions in both ci.yml and release.yml) keeps the test logic in one place. Both workflows call the same definitions; any change to the test steps is automatically reflected in both gates.

workflow_dispatch is kept on each test workflow so that individual test suites can be re-run manually from the GitHub Actions UI without needing to push a commit or tag.

Image tagging

The build step computes image tags from the git tag:

TAG="$"          # e.g. v27.0.1-rc2 or v27.0.1
FRONTEND_TAGS="${FRONTEND}:${TAG}"
# :latest only for stable releases (no pre-release suffix)
if [[ "$TAG" =~ ^v[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
  FRONTEND_TAGS="${FRONTEND_TAGS},${FRONTEND}:latest"
fi

Release candidates (v27.0.1-rc2) get a versioned tag only. Stable releases (v27.0.1) also update :latest. This means a cluster running :latest auto-updates on the next helm upgrade, while a cluster pinned to a specific tag is unaffected.

Helm chart publishing is manual

The Helm chart is published separately with ./deploy/scripts/publish-chart.sh <overlay>. We kept this manual for two reasons: the chart version is independent of the app version (you might push 10 image releases without any chart changes), and publishing the chart is a deliberate operator action — it should not happen automatically on every tag.

Egress IPs and External Service Whitelisting

A practical difference between EKS and DOKS emerged when connecting to MongoDB Atlas, which requires IP whitelisting for all incoming connections.

On EKS, the cluster’s private node group sits behind a single NAT gateway. All outbound traffic from every pod — regardless of which node it runs on — exits through one stable public IP. Adding that single IP to MongoDB Atlas’s allowlist is all that’s needed, and the IP never changes when nodes are replaced or the cluster scales.

On DOKS, there is no NAT gateway by default. Each node is assigned its own public IP, and pods reach the internet directly through the node they’re scheduled on. This means:

• There is no single egress IP — the source address MongoDB sees depends on which node the backend pod happens to be running on.
• With two nodes, you need two IPs in the allowlist. With autoscaling, new nodes get new IPs, and the allowlist breaks until you add them.

For a fixed-size dev cluster, the workaround is to whitelist all current node IPs. For a production DOKS cluster with autoscaling, the correct solution is to provision a Digital Ocean Load Balancer as a NAT gateway, routing all cluster egress through a single stable IP. This adds ~$12/month but is the only reliable option when the external service requires a static source address.

For our dev cluster (doc-router-dev), we whitelist the two node IPs directly. For production DOKS deployments, a managed NAT gateway is required.

Overlay-based Deploy Scripts

Rather than a one-size-fits-all deploy script, we use an overlay pattern:

.env              # shared defaults (local dev values)
.env.eks-test     # overrides for the test EKS cluster
.env.eks-prod     # overrides for production

The deploy scripts (k8s-deploy.sh, build-push.sh) accept an overlay name and source both files, with the overlay taking precedence. A single variable — APP_HOST — drives all URL configuration, making it straightforward to add a new environment. k8s-deploy.sh is idempotent — it uses helm upgrade --install and handles both fresh installs and rolling updates without any distinction.

What’s Next

• On-premises distribution — Helm chart and images are public on ghcr.io; self-hosted MongoDB is available via the mongodb-atlas-local chart (see Self-Hosted MongoDB on Kubernetes with Atlas Search); documentation for a one-command on-prem install is the next step
• Offline license keys — JWT-based licenses signed with a private key, verified against a public key baked into the image, for air-gapped installations
• Multi-cloud support — Digital Ocean Kubernetes is now supported alongside EKS; Azure Kubernetes Service support is planned

Andrei Radulescu-Banu is the founder of DocRouter.AI (document processing with LLMs) and SigAgent.AI (Claude Agent monitoring). His company AnalytiqHub.com provides consulting services for cloud and AI engineering.

The manifest problem

A real Kubernetes application needs dozens of YAML files: Deployments, Services, ConfigMaps, Secrets, Ingress rules, HorizontalPodAutoscalers, PodDisruptionBudgets. Writing them by hand is feasible once, but the moment you need the same app running in three environments — local, staging, production — you face a choice:

• Copy the files for each environment and keep them in sync manually (fragile)
• Use a tool that handles the variation for you

Two tools dominate: Kustomize and Helm. They solve the same problem differently, and many projects use both — Helm for third-party software, Kustomize for their own app.

Kustomize — layered YAML patches

Kustomize ships with kubectl (no install needed) and works with plain YAML. The idea is a base + overlays structure:

manifests/
  base/
    deployment.yaml      # canonical deployment
    service.yaml
    kustomization.yaml   # lists the resources
  overlays/
    dev/
      kustomization.yaml # patches for dev
      patch-replicas.yaml
    prod/
      kustomization.yaml # patches for prod
      patch-replicas.yaml
      patch-resources.yaml

The base defines the resource once. Each overlay patches only what differs. A typical patch looks like:

# overlays/prod/patch-replicas.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: backend
spec:
  replicas: 4       # override base value of 2

To deploy the prod overlay:

kubectl apply -k overlays/prod/

Kustomize merges the base YAML with all patches before sending anything to the API server. You always see plain, readable YAML — there is no templating language to learn, and the output is predictable.

Variable substitution

For values that vary by environment (hostnames, image tags, resource sizes), Kustomize offers substituteFrom: it reads variables from a ConfigMap or Secret and injects them into the manifests at apply time:

# kustomization.yaml
configurations:
  - var-references.yaml
vars:
  - name: APP_DOMAIN
    objref:
      kind: ConfigMap
      name: project-values
      apiVersion: v1
    fieldref:
      fieldpath: data.domain

This is less flexible than Helm’s full templating but keeps the YAML closer to what Kubernetes actually receives.

What Kustomize does not do

Kustomize has no concept of a release, no revision history, and no built-in rollback. If you apply a broken overlay, you must fix it and reapply, or manually apply a previous version. For the same reason, there is no --atomic safety net — if a deployment fails mid-rollout, you notice from kubectl output, not from the packaging tool.

Helm — templated packages

Helm wraps Kubernetes YAML in a full templating engine (Go templates) and adds lifecycle management on top. A chart is a directory:

doc-router/
  Chart.yaml          # name, version, appVersion
  values.yaml         # default values
  templates/
    deployment.yaml   # Go template
    service.yaml
    ingress.yaml
    _helpers.tpl      # reusable template fragments

A template looks like:

# templates/deployment.yaml
spec:
  replicas: 
  template:
    spec:
      containers:
        - name: backend
          image: ":"
          resources:
            requests:
              cpu: 

To install with custom values:

helm upgrade --install doc-router ./doc-router \
  --set replicaCount=4 \
  --set image.tag=v1.2.3

Or via an override file:

helm upgrade --install doc-router ./doc-router -f values-prod.yaml

Release history and rollback

Helm records every install and upgrade as a numbered revision in the cluster. You can inspect history and roll back:

helm history doc-router -n doc-router
helm rollback doc-router 2 -n doc-router   # back to revision 2

With --atomic, a failed upgrade automatically triggers a rollback — the old version keeps running uninterrupted.

Publishing charts as OCI artifacts

A packaged chart can be pushed to any OCI-compatible registry (ghcr.io, ECR, Docker Hub) and pulled from anywhere:

helm push doc-router-0.3.7.tgz oci://ghcr.io/analytiq-hub
helm upgrade --install doc-router oci://ghcr.io/analytiq-hub/doc-router --version 0.3.7

This means a customer cluster can install your app with a single command, pulling both the chart and images from the same registry, with no Git access required.

Kustomize vs Helm — when to use each

In practice many projects use both: Helm for installing third-party dependencies (ingress-nginx, cert-manager, MongoDB operator), and Kustomize for their own application manifests. The two are compatible — a Kustomize overlay can reference a Helm chart as a generator.

GitOps — the cluster manages itself

Both Kustomize and Helm, as described so far, are imperative: a human (or a CI job) runs a command that pushes changes into the cluster. GitOps flips this model.

In GitOps, the desired cluster state is declared in a Git repository (or an OCI artifact registry). A controller running inside the cluster continuously watches that source and reconciles actual state to match it. No one runs helm upgrade — the cluster pulls its own updates.

Developer pushes to Git / CI pushes OCI artifact
         ↓
  Source of truth updated
         ↓
  In-cluster controller detects drift
         ↓
  Controller applies the diff
         ↓
  Cluster matches desired state

The key property: the cluster self-heals. If someone manually deletes a Deployment or edits a ConfigMap, the controller notices the drift and reverts it within seconds. The Git repo (or OCI artifact) is always the authoritative source.

Flux — a GitOps controller

Flux is one of the two dominant GitOps controllers (the other is Argo CD). It runs as a set of controllers in the cluster and watches sources:

Sources

Flux can watch:

• Git repositories — on every push, Flux reconciles the cluster
• OCI artifact registries — on every flux push artifact, Flux pulls and applies
• Helm repositories — for managing Helm releases declaratively

Core resources

GitRepository / OCIRepository — defines where Flux watches:

apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: OCIRepository
metadata:
  name: my-app
  namespace: flux-system
spec:
  interval: 1m
  url: oci://123456789.dkr.ecr.us-east-1.amazonaws.com/my-app-manifests
  ref:
    tag: latest

Kustomization — tells Flux what to apply from the source:

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: my-app
  namespace: flux-system
spec:
  interval: 5m
  sourceRef:
    kind: OCIRepository
    name: my-app
  path: ./manifests/kubernetes/overlays/prod
  prune: true      # delete resources removed from source
  healthChecks:
    - apiVersion: apps/v1
      kind: Deployment
      name: backend
      namespace: my-app

HelmRelease — manages a Helm release declaratively:

apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: ingress-nginx
  namespace: flux-system
spec:
  interval: 1h
  chart:
    spec:
      chart: ingress-nginx
      version: "4.11.3"
      sourceRef:
        kind: HelmRepository
        name: ingress-nginx
  values:
    controller:
      replicaCount: 2

CI/CD with Flux

A typical Flux-based pipeline looks like:

1. Developer opens a PR
2. CI runs tests
3. PR merged to main
4. CI builds Docker image → pushes to ECR
5. CI packages Kustomize manifests as OCI artifact → flux push artifact → ECR
6. Flux detects new artifact version
7. Flux applies manifests to cluster
8. Cluster rolls out new Deployment

Steps 6–8 happen automatically, inside the cluster, with no deploy script and no human intervention.

Flux vs running deploy scripts

GitOps is the right choice for teams with multiple people deploying to shared clusters, or for production environments where drift must be detected and prevented. For a small team or a self-hosted product where simplicity matters, shell scripts with helm upgrade --install are easier to understand, debug, and hand off to a customer.

Summary

A mature production setup typically uses all three: Kustomize or Helm for defining manifests, Flux or Argo CD for reconciling them, and a CI pipeline that produces the artifacts both consume.

Next: Deploying Doc Router on Kubernetes walks through a real application deployment (Helm chart, workers, CI/CD, EKS and Digital Ocean). If you need in-cluster MongoDB with vector search, see Self-Hosted MongoDB on Kubernetes with Atlas Search.

Andrei Radulescu-Banu is the founder of DocRouter.AI (document processing with LLMs) and SigAgent.AI (Claude Agent monitoring). His company AnalytiqHub.com provides consulting services for cloud and AI engineering.

Here’s how the key concepts map across.

From containers to Pods

In Docker Compose, the unit of work is a container. In Kubernetes, it is a Pod — a group of one or more containers that always run together on the same machine and share a network namespace. Most Pods contain a single container, but some use sidecars: a main process plus a helper (a log shipper, a proxy, or in our case the mongot search process alongside mongod).

Pods are ephemeral. When a Pod dies, Kubernetes replaces it with a new one — possibly on a different machine, with a new IP address. You never SSH into a Pod or rely on its IP being stable.

Deployments — the equivalent of a Compose service

A Deployment tells Kubernetes: “keep N replicas of this Pod running at all times.” If a Pod crashes, the Deployment controller starts a replacement. If you push a new image, it performs a rolling update — starting new Pods before terminating old ones so traffic is never interrupted.

In Docker Compose terms, a Deployment is your service: block plus restart policies and rolling update logic built in.

Services — stable internal addresses

Because Pod IPs change on every restart, Kubernetes introduces Services: stable DNS names and virtual IPs that front a group of Pods. A Service named backend in the doc-router namespace is reachable at backend.doc-router.svc.cluster.local from anywhere in the cluster, regardless of how many backend Pods exist or where they are running.

This replaces the automatic DNS that Docker Compose sets up between containers on the same network.

Namespaces — isolation within a cluster

A Namespace is a logical partition of the cluster. Resources in different namespaces don’t collide even if they share a name. A typical setup uses separate namespaces for each concern: doc-router for the application, mongodb for the database, ingress-nginx for the load balancer, cert-manager for TLS certificates.

In Docker Compose terms, a namespace is roughly equivalent to a separate Compose project — distinct networks and name scopes.

ConfigMaps and Secrets — environment variables at scale

Docker Compose lets you set environment: variables inline or via an .env file. Kubernetes separates non-sensitive config from sensitive config:

• ConfigMap — key-value pairs mounted as environment variables or files. Used for things like FASTAPI_ROOT_PATH, worker count, S3 bucket name.
• Secret — base64-encoded values stored (optionally encrypted at rest) separately from your app manifests. Used for database URIs, API keys, and auth secrets. Pods reference Secrets by name; the values are injected at runtime, never baked into the image.

PersistentVolumeClaims — durable storage

Docker Compose uses named volumes (backed by the local filesystem) to persist data across container restarts. Kubernetes uses PersistentVolumeClaims (PVCs): a request for a piece of storage of a given size and access mode. The cluster fulfils the claim by provisioning a real volume — an EBS disk on AWS, a DO Block Storage volume on Digital Ocean — and mounting it into the Pod.

PVCs survive Pod restarts and rescheduling. If a database Pod moves to a different node, the volume is detached and reattached automatically. Storage is provisioned dynamically by a StorageClass, which specifies the provisioner (e.g. ebs.csi.aws.com on EKS) and volume type.

Ingress and the load balancer

In Docker Compose you typically expose one port from one container. In Kubernetes, multiple Services need to be reachable from the outside under different paths or hostnames, all through a single external IP.

ingress-nginx is a Kubernetes controller that runs an nginx reverse proxy inside the cluster. When deployed on EKS, it automatically provisions an AWS Network Load Balancer with a stable public IP. You define Ingress rules — “route /fastapi to the backend Service, everything else to the frontend Service” — and ingress-nginx handles the routing. On a new cluster, the load balancer is the only resource with a public IP; everything else is internal.

cert-manager — automatic TLS

cert-manager is a Kubernetes controller that watches Ingress resources and automatically requests TLS certificates from Let’s Encrypt. When you annotate an Ingress with cert-manager.io/cluster-issuer: letsencrypt-prod, cert-manager handles the ACME challenge, obtains the certificate, stores it in a Secret, and renews it before it expires. You never touch a certificate manually.

Helm — packaging it all together

Kubernetes resources are defined as YAML files. A real application needs dozens of them: Deployments, Services, ConfigMaps, Secrets, Ingress rules, PodDisruptionBudgets. Helm is the package manager for Kubernetes — it bundles all those YAML files into a chart, parameterises them with a values.yaml file, and installs or upgrades the whole bundle with a single command:

helm upgrade --install doc-router oci://ghcr.io/analytiq-hub/doc-router \
  --namespace doc-router --set appHost=example.com ...

A chart can be published as an OCI artifact to any container registry alongside the Docker images.

If Docker Compose is a docker run wrapper, Helm is closer to an apt package: versioned, reproducible, and upgradeable.

How Helm applies changes

Every time you run helm upgrade, Helm compares the new rendered YAML against what it last applied and sends only the diff to the Kubernetes API — resources that haven’t changed are left untouched. Helm records each upgrade as a numbered revision, stored as a Secret in the cluster:

$ helm history doc-router -n doc-router
REVISION  STATUS     CHART           APP VERSION  DESCRIPTION
1         superseded doc-router-0.3.5  v27.0.0    Install complete
2         superseded doc-router-0.3.6  v27.0.1    Upgrade complete
3         deployed   doc-router-0.3.7  v27.0.2    Upgrade complete

Rolling back to a known-good state

If an upgrade goes wrong, rolling back to the previous revision is a single command:

helm rollback doc-router -n doc-router        # rolls back to revision 2
helm rollback doc-router 1 -n doc-router      # rolls back to a specific revision

Helm re-applies the exact YAML from that revision — the same image tags, the same config values — so the cluster returns to the state that last worked. Using --atomic during an upgrade makes this automatic: if the new Pods don’t become healthy within the timeout, Helm rolls back on its own without any manual intervention.

Zero-downtime rolling updates

When Helm upgrades a Deployment with a new image, Kubernetes does not restart all Pods at once. It uses a rolling update strategy controlled by two parameters:

strategy:
  type: RollingUpdate
  rollingUpdate:
    maxUnavailable: 0   # never take a pod down before a new one is ready
    maxSurge: 1         # allow one extra pod above the desired count during the rollout

With maxUnavailable: 0, Kubernetes starts a new Pod with the new image first. Only after that Pod passes its readiness probe — meaning it is actually serving traffic — does Kubernetes terminate one of the old Pods. This continues one Pod at a time until all replicas are on the new version. At no point does the number of healthy Pods drop below the desired count.

The result: an upgrade from v27.0.1 to v27.0.2 with two replicas proceeds as:

1. Start new Pod (v27.0.2) — 2 old + 1 new running
2. New Pod passes readiness check
3. Terminate one old Pod — 1 old + 1 new running
4. Start second new Pod — 1 old + 2 new running
5. Second new Pod passes readiness — terminate last old Pod
6. Rollout complete — 2 new Pods running, zero downtime

If the new Pod fails its readiness check at step 2, the rollout pauses. No old Pods have been terminated, so the old version continues serving 100% of traffic. With --atomic, Helm then rolls the release back automatically.

Running Kubernetes locally with Kind

Before deploying to a real cluster, it’s useful to test locally using Kind (Kubernetes in Docker). Kind runs an entire Kubernetes cluster — control plane and worker nodes — as Docker containers on your laptop. There is no cloud provider, no load balancer, and no cloud volumes; Kind uses your local filesystem for storage and NodePort services for external access.

./deploy/scripts/setup-kind.sh   # creates the Kind cluster
./deploy/scripts/deploy-kind.sh  # installs the Helm chart locally

The same chart that runs on EKS runs on Kind, with a different values-kind.yaml override file. This lets you iterate on chart changes without incurring cloud costs or waiting for node provisioning.

Summary

Next: Kubernetes Packaging and Deployment: Kustomize, Helm, and GitOps goes deeper into packaging manifests and GitOps with Flux.

Andrei Radulescu-Banu is the founder of DocRouter.AI (document processing with LLMs) and SigAgent.AI (Claude Agent monitoring). His company AnalytiqHub.com provides consulting services for cloud and AI engineering.

Why we built it

We built the Document Agent to cut configuration time for parsing a yet-unseen document by about 90%.

What the agent does

The agent is a tool-calling LLM scoped to one document. It sees the document’s metadata, an OCR text excerpt, optional @-mentions (schemas, prompts, tags you’ve referenced), and the current extraction. It has 25 tools: schema CRUD and validation, prompt CRUD, tag CRUD, document list/update/delete, get OCR text, run extraction, patch extraction fields, and two help tools (help_schemas, help_prompts).

• Read-only tools run automatically; read-write tools (create schema, run extraction, update document, etc.) can pause and ask the user to approve or reject each call.
• Conversations are stored in threads per document so you can resume or start a new one.

Architecture overview

Three layers matter:

• The agent loop — how we call the LLM and handle tool calls.
• The context we give the LLM (system message).
• The state we keep between requests (memory vs MongoDB).

Agent loop

The core is a loop:

1. Call LLM with system message + conversation + tool definitions → get text and/or tool calls.
2. If any tool call is read-write and not auto-approved: stop, persist turn state, return turn_id and pending calls to the client.
3. Client shows approve/reject UI and calls POST /chat/approve with approvals.
4. Backend executes approved tools, appends results, calls LLM once.
5. If LLM returns more tool calls, return them to the client (repeat from step 2).

We cap tool rounds at 10 so a turn can’t run forever. The pause happens in the client, not in a long-running server loop.

Here’s the algorithm in plain form:

Figure 1: Agent loop.

Important: The LLM is not called once per request. Inside the loop, it can be called up to 10 times in a row when the model keeps returning auto-approved tool calls (e.g. read schema → read prompt → run extraction). Only when a write tool needs approval do we pause and return a turn_id; after the client approves, we call the LLM again (and may loop or pause again).

Context (system message)

Every turn gets a system message built from:

• Document ID and file name
• OCR excerpt of the document (truncated to ~8k characters so we don’t blow the context window)
• Resolved @-mentions — if the user referenced a schema, prompt, or tag, we resolve it server-side and inject the full content (e.g. full JSON schema) so the LLM doesn’t have to call get_schema just to see what they meant
• Working state — the last schema_revid, prompt_revid, and extraction result from this conversation, so the agent can say “run extraction with that prompt” without the user re-specifying IDs
• Instructions — use help_schemas / help_prompts when creating or modifying those artifacts, and always call validate_schema before create_schema or update_schema so we never persist invalid schemas

Figure 2: How the system message is built.

What we store in memory

Browser memory: Tool approval lives in the client. When the backend returns a turn_id and pending tool calls, the UI shows approve/reject cards and holds the user’s choices in browser memory until they submit POST /chat/approve. Nothing about which tools the user approved is persisted on the server until that request is sent.

Session memory (server): When we pause for approval, we keep turn state in server memory—message list, pending tool calls, working state, model—keyed by turn_id, with a TTL of 5 minutes. The approve endpoint loads by turn_id, runs tools, then discards that state. We don’t persist it: if the user closes the tab or the server restarts, the turn expires and they can resend. So “in-flight approval” is deliberately ephemeral.

What we store in MongoDB

We persist two things that matter for the agent:

1. Threads (agent_threads). Each document is a conversation thread: organization_id, document_id, created_by (user), plus title, messages, extraction, optional model, and timestamps. When a turn finishes (no more pending tool calls), we append the user and assistant messages and the latest extraction to the thread. Threads are what you list, load, and resume in the UI.

2. LLM provider config (llm_providers). We store which models are enabled and, for the document agent, which models appear in the chat dropdown. Per provider (Anthropic, OpenAI, Gemini, etc.):

   • litellm_models_available — discovered from the provider
   • litellm_models_enabled — which of those the org has turned on for general use
   • litellm_models_chat_agent — the subset allowed in the document agent UI
     Admins can enable many models for extraction but expose only a few in the agent. API keys (tokens) and enabled/disabled per provider live here too.

Tool definitions (which tools exist and whether they are read-only vs read-write) are not stored in MongoDB. They’re defined in code (the tool registry) and exposed via GET /chat/tools so the UI can show “these actions need approval.” That keeps the security model simple and consistent across environments.

Why MongoDB here? The database is document-oriented and schema-flexible by default, but we don’t treat it as a free-for-all.

• We run versioned migrations (same idea as SQL): a migrations collection tracks schema version; each migration can add indexes, rename or reshape collections, and backfill data.
• Result: a strict, explicit schema we evolve in a controlled way—portability and the same regularity you’d expect from Postgres—while keeping MongoDB’s strengths: horizontal scaling (sharding, replica sets), flexible documents where we need them, one deployment story for structured and semi-structured data. In practice, agent threads and LLM provider config are as regular as relational tables; we just don’t pay the cost of rigid columns until we need to scale out.

Implementation stages

We built the Document Agent in three stages, each one shippable on its own and visible in the product.

Figure 3: Left-to-right implementation stages.

From left to right:

• Stage 1 — UI + FastAPI: we started with the core product surface—document list, schemas, tags, prompts—and added FastAPI endpoints for every UI action. Anything you can point-and-click in the app (create/edit schemas, prompts, tags; run extraction; manage documents) can also be exercised through REST APIs.
• Stage 2 — MCP server: we wrapped all document, schema, tag, and prompt APIs into a TypeScript MCP server. At this point, we could use external agents (e.g. Claude Code) to operate DocRouter via MCP, turning our REST surface into a tool catalog without changing the backend.
• Stage 3 — Document Agent UI + loop + caching: we then built the Document Agent UI, added dedicated Copilot FastAPI endpoints and the agent loop described above, and finally layered in LLM caching—provider-level prompt caching for system messages and MongoDB-based embedding caching—to keep the experience both fast and cost-efficient.

Key decisions

Read-only vs read-write tools
We split tools into two sets:

• Read-only (e.g. get_ocr_text, list_schemas, validate_schema, help_schemas) never require approval—they’re safe to run as soon as the LLM asks.
• Read-write (e.g. create_schema, run_extraction, update_document) require approval by default. The client can send auto_approve: true (run everything) or auto_approved_tools: ["run_extraction"] (only those run without pausing).

That way power users can say “just run extraction when I ask” while still being prompted for “create a new schema.” The backend exposes GET /chat/tools returning the two lists so the UI can explain which actions will pause.

One LLM round per approve
When the user approves tool calls, we execute them and call the LLM once with the new tool results. If the LLM returns more tool calls, we return those to the client again—we don’t keep looping on the server. The reason is control: the user sees each batch of proposed actions and can approve or reject. If we looped server-side until “no more tool calls,” a single request could do many creates/updates before the user saw anything. So the “loop” is really a handshake: chat → (optional approve) → chat → (optional approve) → …

Sanitizing messages when loading from a thread
When the user resumes a thread, we send the saved messages back to the LLM. But the API requires that every assistant message with tool_calls is immediately followed by tool messages (one per call). If the user had left mid-approval or we stored a partial state, we might have an assistant message with tool_calls and no following tool results. So before building the LLM request we sanitize: we walk the message list and, for any assistant message with tool_calls, check that the next messages are tool results for those call IDs. If not, we strip the tool_calls from that assistant message and send it as content-only. That keeps the API happy and avoids confusing the model with an invalid history.

Working state in the loop
working_state (schema_revid, prompt_revid, extraction) is updated by the tool implementations as they run (e.g. run_extraction sets working_state["extraction"] and working_state["prompt_revid"]). The system message is built once per turn with the current working state. So when the agent says “I’ll use the prompt we just created,” the next LLM call already has that prompt_revid in the system message and the agent can call run_extraction without a prompt_revid argument (we use working state as default). Same for “update the total to 1,250”—the agent sees the current extraction JSON and can call update_extraction_field with the right path.

Streaming and approval
We support streaming (SSE) for the main chat endpoint: the client gets events for thinking chunks, text chunks, tool calls, tool results, and a final done payload. The approve endpoint is non-streaming: one request, one response. That keeps the approve flow simple (no need to stream a single round) and keeps the “pause for approval” contract clear: you get a full set of tool calls, you approve, you get one full response. Streaming is for the interactive chat experience; approve is for the control boundary.

Thinking blocks and API compatibility
Some models (e.g. Claude with extended thinking) return thinking_blocks in the response. When we continue the conversation (e.g. after tool execution), we must send those blocks back to the API in the right format—Anthropic requires a non-empty signature on each block. Our streaming path sometimes produces blocks without signatures, so we have a pass that only includes blocks that have a signature when we rebuild the message for the next call. We also avoid sending the thinking parameter when the last assistant message had tool_calls but no thinking_blocks, so we don’t trigger API warnings or rejections.

LLM caching
We use prompt caching at the provider level to make repeated calls cheaper and faster. For chat models that support prompt caching (via LiteLLM’s supports_prompt_caching), we convert the system message into content blocks with cache_control: {"type": "ephemeral"} so providers like Anthropic and OpenAI can reuse the long, stable system prompt across turns and tool rounds. We intentionally skip prompt caching for Gemini/Vertex—their cached-content APIs reject prompts under certain token thresholds, and our system prompts are often smaller than those limits—so for those providers we fall back to regular calls with no cache directive.

SPU and cost
Each LLM call in the agent (and each call inside run_extraction) checks SPU (our credit system) independently. We don’t reserve or estimate “total cost for this turn” upfront—we charge as we go. If the org runs out of credits mid-turn, that LLM call fails and we surface the error; the turn ends. That matches how the rest of the app works and avoids over-engineering reservation logic.

Frontend and API

The frontend includes:

• Chat panel — message list, input, model/tools settings
• Tool-call cards — approve/reject per call, with expandable arguments
• Thinking block — collapsible, with optional live timer
• Thread dropdown — list threads, create, load, delete

When the backend returns a turn_id and pending tool_calls, the UI shows the cards and disables send until the user approves or rejects. The same agent is exposed over REST: custom UIs or automation can call POST /chat (and POST /chat/approve when needed), with optional streaming and thread_id for persistence.

Summary

The Document Agent is a tool-calling LLM with:

• A bounded loop — LLM → optional user approval → tools → LLM again
• Rich context — document, OCR, @-mentions, working state
• What we store in memory (browser: tool approval; server: turn state) vs what we store in MongoDB (threads, LLM config)

Splitting read-only and read-write tools keeps approval predictable; one round per approve keeps the user in control; message sanitization keeps thread reloads valid; and working state keeps “what we just created” visible to the agent without extra round-trips. If you’re building something similar—an in-context agent that can read and write—this architecture is a solid starting point.

To use the Document Agent, open any document in DocRouter and open the Chat / Agent tab. For API details, see Document Agent in the docs.

I use MongoDB as the primary database for AI-powered products like DocRouter.AI and SigAgent.AI. This post explains how it’s implemented—migrations, vector search, and knowledge bases—and why I prefer it over alternatives like Postgres for document-centric, JSON-heavy AI workloads. I want to store a very large number of documents (DocRouter) or logs (SigAgent) without spending much time tuning the database for horizontal scaling; MongoDB fits that need well.

Brief trade-offs vs Postgres

Postgres with jsonb can model the same document-style records and even integrate vector search via extensions, but it shines most when you need strong relational guarantees and complex joins around a relatively stable schema. MongoDB is a better fit when almost everything is a JSON document, the schema evolves quickly, and you care more about horizontal scaling and quick turnaround than about classic SQL features.

In my case, the workloads are heavily document- and log-centric, so the ergonomics and scaling model of MongoDB outweigh the benefits of staying inside the relational/Postgres ecosystem.

It’s also totally reasonable to split responsibilities: e.g. Postgres for relational metadata, a dedicated vector store like Pinecone/Weaviate/Qdrant for embeddings, or Redis as an embedding cache in front of another database. For DocRouter.AI and SigAgent.AI, I’m happy to trade some theoretical optimality for a single operational datastore (MongoDB handling documents, logs, and vectors) until scale or workload complexity justifies introducing extra systems.

DocRouter and SigAgent: One Backend, Two Products

DocRouter.AI is a smart document router: you upload documents, define schemas and prompts, and it extracts structured data (e.g. from invoices, medical records, forms) using LLMs. SigAgent is a Claude agent monitor with a different UX and product focus, but it’s built on the same stack.

Roughly 90% of the backend is shared. Both use the same Python package, analytiq_data: MongoDB client, migrations, queue layer, auth, and app startup. The same MongoDB database layout, indices, and migration history apply to both. Product-specific code lives in routes and frontends; the data layer is common.

Strict Schema via Discipline: Migrations as “Schema”

MongoDB doesn’t enforce a schema. I still want predictable structure and safe upgrades, so we enforce it in code and process:

• Consistent document shape per collection. We use a fixed set of field names and types in application code (and in TypeScript/Pydantic where applicable). In practice, each collection behaves like a table with a known “schema.”
• Every change goes through migrations. Renaming fields, adding/removing fields, splitting or renaming collections—all of it is done in versioned migration classes with up() and down(). The current schema version is stored in a migrations collection; on startup we run run_migrations(analytiq_client) and bring the DB to the latest version.
• Element types as schema. We treat document fields as if they were typed: e.g. schema_id, prompt_version, organization_id are always present where we expect them. New code assumes the post-migration shape. So we get comparable safety to Postgres (known structure, no surprise shapes) while staying in a document model—as long as we’re disciplined and never bypass migrations.

So: schema is “enforced” by convention and migrations, not by the database. That keeps development fast without giving up control.

How Migrations and Indices Are Implemented

At a high level:

• Migrations are versioned Python classes with up()/down() methods that evolve collections in lockstep with the code.
• Indices are either created inside migrations (long-term, repeatable) or via a small helper at runtime for per-module needs.
• Profiling tools (Atlas Query Insights, profiler, Compass/Performance Advisor) surface slow queries that suggest new compound indexes.

Migrations

Migrations live in migration.py. Each migration is a class with:

• description: short human-readable summary
• up(db): apply the change (e.g. rename field, backfill, add index)
• down(db): revert the change when possible

The runner loads the list MIGRATIONS, assigns a version by index, and stores the current version in db.migrations under _id: "schema_version". On startup we run pending migrations in order. Examples from the codebase:

• Field renames: e.g. RenameUserFields (camelCase → snake_case)
• New fields: e.g. LlmResultFieldsMigration (adds is_edited, is_verified, timestamps)
• Structural changes: e.g. RenameCollections (schemas → schema_revisions, schema_versions → schemas), UseMongoObjectIDs

Collection layout and index strategy evolve in one place, with a clear history and rollback path.

Indices

We manage indices in two ways:

1. Inside migrations for long-term indices. These are created with create_index(..., background=True) and dropped in down() so rollback is consistent.
2. At runtime via ensure_index() in analytiq_data/mongodb/index.py. Given a collection, index spec, and name, it creates the index if missing (and optionally drops other non-_id indexes). We use this for things like payments_usage_records (org_id, timestamp) when the payments module initializes.

Example index patterns:

• Queue collections: (status, created_at) for find_one_and_update({ status: "pending" }, sort: { created_at: 1 }).
• docs: (organization_id, upload_date desc) for paginated listing by org.
• llm_runs: (document_id, prompt_id, prompt_version desc) for “latest run by document and prompt,” and (document_id, prompt_revid) for exact revision lookup.
• document_index: unique (kb_id, document_id) and a non-unique document_id index for cascade deletes.

To find candidates for new indices, we use Atlas Query Insights (filter by operations returning >1,000 documents, then add a compound index matching the filter and sort) or, on Community Edition, enable the database profiler and filter system.profile by nReturned:

db.setProfilingLevel(1, { slowms: 100 })
db.system.profile.find({ nReturned: { $gt: 1000 } }).sort({ ts: -1 }).limit(20)

Level 2 profiling captures everything but has I/O cost, so we use it in short bursts or staging. Level 1 (slow only) is cheaper and still surfaces most heavy queries.

If you prefer GUIs over shell tools, both MongoDB Compass and the Atlas Performance Advisor surface similar slow-query/index recommendations in a more visual way, which is often easier when you’re just getting started tuning a workload.

With schema and indices in place, here’s how we run it.

Dev/Prod Setup and Vector Search

Quickly, the environments look like this:

• Local: MongoDB running on localhost or via the Atlas Local Docker image (bundling mongod + mongot) for easy vector search testing.
• Production: MongoDB Atlas or AWS DocumentDB, configured for majority writes and read scaling.
• Self-hosted: MongoDB 8.2+ with mongot in a replica set, using the same createSearchIndexes / $vectorSearch APIs.

For local development, MONGODB_URI defaults to mongodb://localhost:27017. For vector search, we use the MongoDB Atlas Local Docker image (mongodb/mongodb-atlas-local:latest), which bundles mongod and mongot (the search/vector process) in one container.

For production we use MongoDB Atlas; for on-prem consulting we use AWS DocumentDB. The client is configured with w='majority', readPreference='secondaryPreferred', and retryWrites=False (DocumentDB doesn’t support retryWrites).

For self-hosted Community Edition, vector search requires MongoDB 8.2+ with mongot running alongside mongod in a replica set. The createSearchIndexes command and $vectorSearch aggregation work the same way across Atlas, Community 8.2+, and the local Docker image—no separate code path.

Knowledge Bases on Top of Vector Search

We implement knowledge bases in the shared backend using MongoDB’s vector search, with three key pieces: an indexing pipeline, a search pipeline, and a reconciliation service.

Indexing: Blue-Green Atomic Swap

Each knowledge base gets its own vector collection (kb_vectors_<kb_id>). When a document is indexed, we chunk the text (via Chonkie, a small library for token/sentence/recursive text chunking), generate embeddings via LiteLLM, a lightweight multi-provider LLM/embedding client, and then atomically swap old vectors for new ones inside a MongoDB transaction:

async with await client.start_session() as session:
    async with session.start_transaction():
        # Delete old vectors for this document
        await vectors_collection.delete_many(
            {"document_id": document_id}, session=session
        )
        # Insert new vectors
        await vectors_collection.insert_many(new_vectors, session=session)
        # Update document_index entry
        await db.document_index.update_one(
            {"kb_id": kb_id, "document_id": document_id},
            {"$set": { ... }},
            upsert=True, session=session
        )
        # Update KB stats (document_count, chunk_count)
        await db.knowledge_bases.update_one(
            {"_id": ObjectId(kb_id)},
            {"$set": {"document_count": total_docs, "chunk_count": total_chunks}},
            session=session
        )

This blue-green pattern means a document is never in a half-indexed state: either all its new vectors are visible, or the old ones remain.

Embedding Cache

Embeddings are cached in a global embedding_cache collection keyed by (SHA-256 chunk hash, embedding model). When the same text chunk appears in multiple KBs (or is re-indexed after an edit), we skip the API call and reuse the cached vector. This saves both cost and latency—especially when re-indexing a large KB after a configuration change.

Search

The query string is embedded with the same model, then we run an aggregation whose first stage is $vectorSearch (with numCandidates = max(top_k * 10, 100) for better recall). We apply filters (organization, tags, date range) in the vector index definition, add vectorSearchScore via $meta, and optionally coalesce neighboring chunks for richer context.

Reconciliation: Keeping KBs in Sync

Documents and KBs can drift: a document’s tags change, a document is deleted, or vectors are orphaned after a failed indexing run. The reconciliation service (reconciliation.py) detects and fixes this:

• Missing documents: documents with matching tags but no document_index entry → queued for indexing.
• Stale documents: indexed documents whose tags no longer match the KB → removed.
• Orphaned vectors: vectors without a corresponding document_index entry → deleted.

Reconciliation uses a distributed lock (atomic find_one_and_update with a 10-minute TTL) so only one worker reconciles a given KB at a time. It processes documents in batches of 100 to keep memory bounded, and supports dry-run mode for auditing without side effects.

In short: we get strict schema via migrations, predictable indexing, vector search for RAG with atomic updates, and self-healing knowledge bases—with one backend shared between DocRouter and SigAgent, and the same patterns whether we run on local Mongo, Atlas, or DocumentDB.

This article originally appeared on KDnuggets. You can read the original version here.

Introduction

The AI industry is experiencing a wave of transformation comparable to the dot-com era, and entrepreneurs are rushing to stake their claims in this emerging landscape. Yet unlike previous technology waves, this one presents a unique characteristic: the infrastructure is maturing faster than the market can absorb it. This gap between technological capability and practical implementation defines the current opportunity landscape.

Andrei Radulescu-Banu, founder of DocRouter AI and SigAgent AI, brings a unique perspective to this conversation. With a PhD in mathematics from the Massachusetts Institute of Technology (MIT) and decades of engineering experience, Radulescu-Banu has built document processing platforms powered by large language models (LLMs) and developed monitoring systems for AI agents, all while serving as a fractional chief technology officer (CTO) helping startups implement AI solutions.

His journey from academic mathematician to hands-on engineer to AI entrepreneur was not straightforward. “I’ve done many things in my career, but one thing I’ve not done is actually entrepreneurship,” he explains. “I just wish I had started this when I was, I don’t know, out of college, actually.” Now, he is making up for lost time with an ambitious goal of launching six startups in 12 months.

This accelerated timeline reflects a broader urgency in the AI entrepreneurship space. When technological shifts create new markets, early movers often capture disproportionate advantages. The challenge lies in moving quickly without falling into the trap of building technology in search of a problem.

The Layering Of The AI Stack

Radulescu-Banu draws parallels between today’s AI boom and the internet revolution. “Just like in the past for computer networks, [you] had developers of infrastructure, let’s say, computer switches and routers. And then you had application layer software sitting on top, and then you had web applications. So what’s interesting is that these layers are forming now for the AI stack.”

To continue reading the full piece, visit KDnuggets here.

Single-prompt tools like DocRouter.AI shine for ~20-25 page docs… but what about massive collated files with labs, facesheets, insurance cards, and multiple patients mixed together? → One prompt = impossible.

Enter the solution: Temporal + DocRouter.AI in a smart, scalable workflow.

This post describes a real-world implementation that uses Temporal to orchestrate document processing workflows with DocRouter.AI, solving the challenge of processing massive medical records through intelligent multi-step orchestration.

The implementation is available at doc-router-temporal and processes medical documents containing hundreds of pages, extracting patient names, dates of birth, and medical insurance information.

The Problem: Massive Medical Records Need Smart Orchestration

Medical records often come as massive collated files containing 200+ pages with:

• Lab results and test reports
• Patient facesheets with demographics
• Insurance cards and coverage details
• Clinical notes and progress reports
• Multiple patients’ information mixed together

The challenge: These documents are too large to process in a single LLM prompt due to token limits (typically 128K-200K tokens). One prompt = impossible for comprehensive extraction.

The solution: A multi-step workflow that intelligently orchestrates the process:

1. Split: Break the massive PDF into individual pages
2. Classify: Identify each page’s type and which patient it belongs to
3. Group: Intelligently group pages by patient
4. Extract: Process each patient’s page bundle for precise, targeted extraction

Why This Pattern Rocks

This Temporal + DocRouter.AI combination delivers powerful advantages:

✅ Constant memory usage — scales effortlessly to 1,000+ pages without running out of resources

✅ Super general pattern → classify → group → process per group → works for any document type

✅ Fully durable & retry-safe thanks to Temporal’s built-in resilience

✅ Built lightning-fast in just a couple of days using AI tools

✅ Parallel processing — handles multiple patients simultaneously while maintaining order

✅ Production-ready with automatic error handling, timeouts, and state management

The Smart Workflow in Action

Here’s how the Temporal + DocRouter.AI workflow processes massive medical records:

🔹 Temporal splits the 200+ page PDF into individual pages

🔹 Uploads them one by one to DocRouter.AI for processing

🔹 DocRouter classifies each page → identifies patient name + document type (lab results, insurance card, facesheet, etc.)

🔹 Temporal intelligently groups pages by patient using fuzzy name matching and DOB correlation

🔹 Sends each patient’s page bundle back to DocRouter.AI for precise, targeted extraction

🔹 Temporal aggregates everything → clean, complete per-patient results

Technical Implementation

Why Temporal?

Temporal provides durable workflow orchestration that’s perfect for this use case. Unlike traditional approaches (queues, background jobs, or simple scripts), Temporal handles:

• Durable execution: Resumes from crashes during 200-page processing
• Parallel processing: Processes multiple pages simultaneously while maintaining order
• Error handling: Automatic retries for API rate limits and network issues
• State management: Tracks processed pages and identified patients
• Long-running workflows: Handles processes that take minutes to hours

Temporal’s architecture is built around two key concepts: Workflows (orchestration logic) and Activities (actual work). The diagram below illustrates how these components work together:

The Workflow Implementation

The implementation uses a hierarchical workflow structure with two main workflows:

1. Classify and Group PDF Pages (ClassifyAndGroupPDFPagesWorkflow): Chunks the PDF, classifies each page, and groups pages by patient
2. Extract Insurance Information (ClassifyGroupAndExtractInsuranceWorkflow): Creates patient-specific PDFs and extracts insurance card data

The main workflow (ClassifyGroupAndExtractInsuranceWorkflow) orchestrates the entire process:

Creating Schemas and Prompts with Claude Agent

Before building the Temporal workflow, we created the extraction schemas and prompts using the Claude Agent for DocRouter.AI (an MCP server at doc-router/packages/typescript/mcp).

The Claude Agent allows Claude Code to create extraction schemas and prompts. For example, you can prompt: “Create a schema for extracting patient information from medical record pages” and it will validate, create, and test the schema automatically.

The diagram below illustrates how DocRouter.AI operations work and how they integrate with Temporal workflows:

Key distinction: DocRouter.AI implements discrete operations (single prompt-and-schema processing per document), while Temporal implements the workflow orchestration (chunking, grouping, uploading pages for classification, and uploading chunks for extraction).

For this implementation, we created:

• medical_page_classifier: Classifies pages as labs, facesheets, insurance cards, clinical notes, or other document types
• insurance_card: Extracts insurance card information from patient pages

Workflow Implementation

The main workflow (ClassifyGroupAndExtractInsuranceWorkflow) orchestrates the entire process. Complete implementation: workflows/classify_group_and_extract_insurance.py.

Step 1: Classify and Group Pages

The workflow calls ClassifyAndGroupPDFPagesWorkflow to:

1. Chunk the PDF into individual pages
2. Classify each page using DocRouter.AI
3. Group pages by patient using name and DOB matching

The grouping logic (activities/group_classification_results.py) includes name normalization, DOB parsing, and fuzzy matching with Levenshtein distance to handle typos and variations.

Step 2: Extract Insurance Information

For each patient group, the workflow:

1. Creates patient-specific PDFs with only that patient’s pages (activities/create_and_upload_patient_pdf.py)
2. Uploads them to DocRouter.AI for insurance card extraction
3. Polls for completion and retrieves results

To avoid passing large binary data through Temporal, PDFs are read from disk and uploaded directly to DocRouter.AI.

Creating Temporal Workflows with Cursor

The Temporal workflow was developed in Cursor using natural language prompts. Cursor’s AI understood the codebase context and Temporal patterns, enabling rapid development without deep workflow expertise.

Key benefits:

• Context awareness across multiple files and existing activities
• Automatic Temporal pattern suggestions (activities, workflows, child workflows)
• Natural language refactoring and error handling implementation

Example development prompts:

“Create a workflow that processes each patient’s pages into separate PDFs, uploads them with insurance_card tag, waits for completion, then retrieves insurance extraction results.”

“Add fuzzy name matching to group pages with names differing by up to 2 letters using Levenshtein distance.”

“Handle edge cases where medical records contain individual patient names vs. multiple patient summaries.”

Cursor handled the complex Temporal implementation, error handling, and performance optimizations, resulting in production-ready code in just a few hours.

Key Implementation Details

Design Decisions

• Avoid large data transfer: PDFs are read from disk and uploaded directly to DocRouter.AI, not passed through Temporal
• Parallel processing: Multiple patients processed concurrently with status polling
• Error handling: Retry logic, graceful degradation, and timeout handling
• State management: Only document IDs and metadata flow through Temporal to keep history efficient

Results

The implementation successfully processes massive medical record documents with hundreds of pages, extracting patient names, dates of birth, and medical insurance information. It handles large documents (200+ pages), parallel patient processing, error recovery, and long-running operations.

Running the Workflow

# Start the Temporal worker
python worker.py

# In another terminal, run the client
python client_classify_group_and_extract_insurance.py <path_to_pdf>

See the README and client script for details.

The workflow returns JSON with file name, page classifications, schedule pages, and patient data with insurance information.

We’re in a New Era

What used to take months of engineering can now be shipped in days.

This Temporal + DocRouter.AI pipeline was built end-to-end using Claude Code-based Agent (for prompts + schemas) + Cursor (for Temporal workflows). I barely knew Temporal before starting — didn’t matter. AI tools let me iterate fast, prototype, and perfect the logic in record time.

The result: reliable, scalable document processing with durable workflows, parallel processing, and rapid schema iteration. The implementation took just 2 days to build and handles 200+ page medical records like a champ.

If you’re building AI-powered document workflows (especially in healthcare), this combo is 🔥.

Code available at doc-router-temporal.

• Temporal Documentation
• DocRouter.AI Documentation
• DocRouter.AI MCP Server

The Evolution: From Method to Theme

Analytiq Pages started as a methodology for building company websites using Jekyll, GitHub Pages, and Tailwind CSS. Today, we’re proud to release it as Analytiq Pages Theme - a fully packaged Jekyll theme that makes this powerful combination accessible to everyone.

✨ What’s New in Analytiq Pages Theme

🚀 Advanced Features

• Tailwind CSS Integration: Modern, responsive design with utility-first styling
• Enhanced Syntax Highlighting: Beautiful code blocks with copy functionality using highlight.js
• Interactive Diagrams: Full Excalidraw integration for creating and embedding technical diagrams
• Professional Blog Layouts: Complete blog system with sidebar, pagination, and category support
• Responsive Navigation: Mobile-first navigation with dropdown menus and hamburger menu
• Dark Theme Support: Built-in dark mode (Minima skin) for modern aesthetics

🛠 Developer Experience

• Three Customization Hooks: Override custom-head.html, custom-header.html, and custom-footer.html for site-specific modifications
• Reusable Components: Pre-built Tailwind components (alerts, buttons, cards)
• SEO Optimized: Integrated jekyll-seo-tag for better search engine visibility
• PDF Embedding: Native support for embedding PDF documents
• RSS Feed Generation: Automatic blog feed generation with jekyll-feed

🎨 Content Features

• MathJax Support: Render mathematical equations in your content
• Multiple Layouts: Specialized layouts for homepages, blog posts, documentation, and more
• Excalidraw Editor: Built-in diagram editor accessible at /excalidraw-edit
• Smart Embeds: Flexible diagram embedding with static, interactive, and link modes

Installation: Get Started in Minutes

Option 1: Quick Start with Existing Site

Add to your Jekyll site’s Gemfile:

gem "analytiq-pages-theme", git: "https://github.com/analytiq-hub/analytiq-pages-theme"

Or for a specific stable version:

gem "analytiq-pages-theme", git: "https://github.com/analytiq-hub/analytiq-pages-theme", tag: "v0.1.6"

Update your _config.yml:

theme: analytiq-pages-theme

Install and serve:

bundle install
bundle exec jekyll serve

Option 2: New Site from Scratch

The simplest way to get started:

# Create new Jekyll site
jekyll new my-company-site
cd my-company-site

# Replace minima theme with analytiq-pages-theme in Gemfile
sed -i 's/gem "minima".*/gem "analytiq-pages-theme", git: "https:\/\/github.com\/analytiq-hub\/analytiq-pages-theme", tag: "v0.1.6"/' Gemfile

# Configure theme in _config.yml
sed -i 's/^theme: .*/theme: analytiq-pages-theme/' _config.yml

# Install and serve
bundle install
bundle exec jekyll serve

Or if you prefer manual editing:

# Create new Jekyll site
jekyll new my-company-site
cd my-company-site

# Edit Gemfile: replace the minima line with:
# gem "analytiq-pages-theme", git: "https://github.com/analytiq-hub/analytiq-pages-theme", tag: "v0.1.6"

# Edit _config.yml: replace the theme line with:
# theme: analytiq-pages-theme

# Install and serve
bundle install
bundle exec jekyll serve

Visit http://localhost:4000 to see your new site!

Option 3: Local Installation (Alternative)

If you encounter repository access issues, you can install the theme locally:

# Download the theme release
curl -L https://github.com/analytiq-hub/analytiq-pages-theme/archive/refs/tags/v0.1.6.zip -o theme.zip
unzip theme.zip
mv analytiq-pages-theme-0.1.6/ _themes/analytiq-pages-theme/

# Or clone locally if you have access
git clone https://github.com/analytiq-hub/analytiq-pages-theme.git _themes/analytiq-pages-theme
cd _themes/analytiq-pages-theme && git checkout v0.1.6

# Add to _config.yml
echo "theme: _themes/analytiq-pages-theme" >> _config.yml

Key Improvements Over Manual Setup

Before (Analytiq Pages Method)

• Manual Tailwind CSS configuration
• Custom Jekyll setup for each project
• Repeated configuration of syntax highlighting
• Manual Excalidraw integration setup
• No standardized component library

After (Analytiq Pages Theme)

• One-line installation: theme: analytiq-pages-theme
• Pre-configured features: Everything works out of the box
• Professional components: Reusable Tailwind components included
• Advanced integrations: Excalidraw, MathJax, PDF embeds ready to use
• Consistent experience: Standardized layouts and styling across sites

Showcase: Real-World Examples

The theme powers several professional websites:

• Analytiq Hub - Business intelligence and analytics platform
• DocRouter.AI - AI-powered document routing solution
• SigAgent.AI - Signature analysis and automation platform
• Bitdribble - Technology consulting and development

Migration Guide: Upgrading from Analytiq Pages

If you’re currently using the Analytiq Pages methodology, migration is straightforward:

1. Add the theme to your Gemfile and _config.yml
2. Remove manual Tailwind configuration (now handled by the theme)
3. Update custom includes to use the new hook system
4. Migrate Excalidraw files to assets/excalidraw/ directory

Your existing content and configuration will continue to work seamlessly.

Technical Architecture

The theme is built with modern web standards:

analytiq-pages-theme/
├── _layouts/           # 5 specialized layouts
├── _includes/          # 16+ reusable components
├── assets/
│   ├── css/           # Tailwind + custom styles
│   └── js/            # Pagination, Excalidraw renderer
├── _config.yml        # Default configuration
└── analytiq-pages-theme.gemspec

Dependencies:

• Jekyll >= 3.9, < 5.0 (supports both GitHub Pages and Jekyll 4.x)
• jekyll-feed ~> 0.12
• jekyll-seo-tag ~> 2.6
• jekyll-pdf-embed ~> 1.1

Why Choose Analytiq Pages Theme?

For Startups & Small Businesses

• Zero hosting costs with GitHub Pages
• Professional appearance without design costs
• Content-first approach with Markdown simplicity
• Scalable foundation that grows with your business

For Agencies & Consultants

• Rapid deployment for client websites
• Consistent branding across projects
• Advanced features for technical content
• Easy customization for client-specific needs

For Enterprise Teams

• Git-based workflows for version control and collaboration
• Security compliance with GitHub’s enterprise infrastructure
• SEO optimization built-in
• Extensible architecture for custom requirements

Troubleshooting

Jekyll Version Compatibility

The theme supports both Jekyll 3.9+ (GitHub Pages) and Jekyll 4.x (modern installations). If you encounter version conflicts:

# For GitHub Pages compatibility (Jekyll 3.x)
gem "github-pages", group: :jekyll_plugins

# For modern Jekyll 4.x installations
gem "jekyll", "~> 4.3"

The theme will work with either version automatically.

Bundle Install Issues

# Clear bundle cache
bundle cache clean --force

# Clear bundler git cache
rm -rf ~/.local/share/gem/ruby/cache/bundler/git/

# Try installing again
bundle install

Theme Not Loading

• Verify _config.yml has theme: analytiq-pages-theme
• Clear Jekyll cache: rm -rf _site .jekyll-cache
• Rebuild: bundle exec jekyll build

Getting Help & Contributing

• Documentation: Comprehensive README at analytiq-pages-theme
• Issues & Support: GitHub Issues for bug reports and feature requests
• Contributing: Pull requests welcome for theme improvements
• Migration Support: Contact us for help upgrading from manual Analytiq Pages setups

How This Fits Into Your Stack

Analytiq Pages Theme transforms our proven methodology into a professional, reusable solution that makes building beautiful company websites accessible to everyone. Whether you’re launching a startup, building client sites, or managing enterprise web presence, this theme delivers the perfect balance of simplicity and power.

Ready to upgrade your web presence? Try Analytiq Pages Theme today!

This theme powers the very website you’re reading now. Experience the Analytiq Pages Theme in action and see the source code for implementation examples.

📢 Join the discussion on LinkedIn about modern Jekyll themes and web development workflows.