Building a multi-tenant RAG chatbot platform with FastAPI, ChromaDB and a four-level LLM fallback

The problem

Most hosted chatbot builders bill per message, keep their retrieval settings out of sight, and hold your content on their servers. The goal of this project was the opposite: a platform a company can host itself, where every retrieval parameter is visible, and where an outage at one LLM provider does not take the assistant down.

There are three kinds of users:

• a visitor, anonymous, talking to the widget on a client's website;
• the administrator, a single super-user who creates accounts and chatbots (there is deliberately no self-registration);
• managed users, created by the administrator and granted viewer, editor or admin rights on specific chatbots.

Architecture

Seven Docker Compose services. The two Celery workers are split on purpose: crawling needs a browser and a lot of memory, the analytics jobs need neither.

A small in-process asyncio scheduler replaces Celery Beat. It runs five jobs: a re-scrape check every 15 minutes, a retry of failed crawls every 5 minutes, a classifier backfill every 10 minutes, hourly content-gap detection and a weekly page-health check. One less container to run and monitor.

Ingestion: from a URL to vectors

The crawler is async Playwright (Chromium), because a large share of real sites render their content with JavaScript. It honours robots.txt, discovers sitemaps (including nested sitemap indexes, three levels deep), runs five tabs in parallel, retries three times, and de-duplicates pages by SHA-256 of their content. Uploads accept PDF, DOCX and TXT.

Then:

1. Chunking — RecursiveCharacterTextSplitter, 1,200-character chunks with 200 of overlap. It is section-aware: the section heading is re-prefixed onto every chunk of that section, so a chunk from the middle of "Refund policy" still says so.
2. Cross-page de-duplication — the first 200 characters of every chunk are fingerprinted. Navigation bars, cookie banners and footers appear on every page and would otherwise drown the index.
3. Embeddings — intfloat/multilingual-e5-small through sentence-transformers, with the passage: / query: prefixes the model expects. Multilingual matters here: the target clients write in French, Arabic and English.
4. Storage — ChromaDB, one collection per chatbot (chatbot_{id}). Crawl chunks are mirrored into a MongoDB collection with a text index, which becomes the fallback when the vector store is unavailable.

Answering a question

Details that turned out to matter:

• Language-aware re-ranking. The cross-encoder is trained on English. Used on French or Arabic queries it scrambled a ranking that embedding distance had got right. It now runs only when langdetect says the query is English, and only if its best score clears a threshold.
• Two retrieval fallbacks. If ChromaDB is down, or its circuit breaker is open, retrieval falls back to a MongoDB $text search. Worse answers beat no answers.
• A four-level LLM chain. By default: Gemini Flash-Lite → an OpenRouter model → a local Ollama model → a "frozen cache" of past good answers kept in Redis for 30 days. Every provider has its own circuit breaker (cloud providers trip after 5 failures and recover after 60 s) and a 15-second first-token timeout. There is no failover after the first token has been streamed: switching models mid-sentence is worse than an honest error.
• Write buffering. Conversation turns go to a Redis Stream first and are drained to MongoDB, so a slow database write never sits on the streaming path.

Six bugs worth remembering

1. A re-crawl wiped every uploaded document. A full re-crawl called delete_collection(), which also removed the chunks of PDFs the client had uploaded by hand. Fix: a clear_non_document_chunks() that keeps everything whose metadata says type == "document", and a "smart re-scrape" that only replaces chunks source by source.

2. The stale-page cleanup deleted fresh pages. After a crawl, pages that no longer exist on the site are removed. The set of visited URLs was normalized (no trailing slash); the database stored the raw URL. Every page whose URL ended with / was therefore "not visited" and deleted seconds after being saved. Fix: normalize both sides, and only run the cleanup after a full auto-discovery crawl that did not hit the page cap.

3. The vector index lived on an ephemeral layer. The ChromaDB volume was mounted at the path an older image used, while the current image persists to /data. The index survived restarts of the container but not its recreation, and the nightly backup was faithfully archiving an empty volume. One line in docker-compose.yml, found during a full audit of the deployment — and a reminder that a backup nobody has restored is not a backup.

4. Out-of-memory kills from loading the model twice. Concurrent asyncio.to_thread calls each triggered a load of the embedding model. Fixed with double-checked locking around the load, and by dropping to a single Uvicorn worker: two workers meant two copies of the model in RAM.

5. A worker that crash-looped in silence. Each Celery prefork child imports torch. With 1 GB of RAM and a concurrency of 5, celery_light died on start, restarted, and died again while more than 1,500 tasks queued up behind it. Fix: concurrency 2, 4 GB and a shared Hugging Face cache path.

6. A double-click started N parallel crawls. A classic check-then-act race. The "is a crawl already running?" check and the "mark as running" write are now one atomic find_one_and_update; the losers get a 409, and a full queue answers 429.

Security

Multi-tenant means the interesting attacks are between tenants:

• Tenant isolation — permission checks with ordered levels (viewer < editor < admin), one vector collection and one cache namespace per chatbot, and conversation lookups bound to the chatbot id (which closed a cross-tenant IDOR found during my own audit).
• SSRF guard on the crawler — the platform fetches URLs that users type. Every seed URL and every discovered link is resolved first, and private, loopback, link-local and reserved ranges are refused. It fails closed when a host cannot be resolved.
• Prompt-injection defence — retrieved passages and the visitor's current page are wrapped in <context> blocks with a guard prompt, and known injection patterns are defanged. 18 tests cover it.
• The public config endpoint returns a whitelist of fields. An early version leaked the system prompt and the owner id through it; that was the most serious finding of the audit.
• Auth — JWT in an HttpOnly, Secure, SameSite cookie; PBKDF2-SHA256 password hashes; TOTP two-factor with hashed single-use backup codes; double-submit CSRF tokens; Redis-backed rate limits (login 5/min, chat 30/min).
• The widget — per-chatbot domain allow-list, DOMPurify on rendered Markdown, and a sanitizer that strips @import, remote url() and javascript: from the custom CSS a client can paste in.

What the admin gets

Crawl launch with live progress, page management (pin, exclude, quality grade, version history), document upload, a playground, conversations, captured leads, and analytics: conversion funnel, a 7 × 24 activity heatmap, answer quality scored by an LLM-as-judge, cost per provider, anomalies, and content gaps: the questions visitors ask that the site does not answer.

What I would do next

• Pin the Python dependencies. Only one is pinned today, which is a reproducibility risk I would not accept on a client project.
• Add a multilingual re-ranker so French and Arabic queries get the same second-stage ranking English does.
• Measure retrieval quality continuously. Today I check a fixed set of questions by hand after changing chunking or embeddings; that belongs in CI.
• Replace per-IP rate limiting on login with per-account lockout as well.