I got tired of rebuilding the same chatbot backend

The same week of work, again

Each time I wanted to try a new chatbot idea I ended up writing the same things first: a FastAPI app, a LangChain model client, streaming, somewhere to keep the conversation. By the third time it was clear the interesting part, what the bot actually does, was maybe a tenth of the code.

So I split it. The boring half became shared components behind interfaces: the model, memory, embeddings, the vector database. The other half became what I call an expert, a small class that decides how one kind of bot behaves. I wrote three: plain Q&A, a RAG bot that answers from PDFs, and a deep-research bot that searches the web.

An expert is mostly one method

All three inherit from a BaseExpert that does the chores: save the user message, load the history, call the brain, save the reply, return a response. The only thing a subclass has to supply is the context that goes to the brain. The Q&A expert returns a fixed system prompt. The RAG expert searches the vector database and pastes the chunks into its prompt.

That's why adding a bot is cheap, and it's the part of the design I'm happiest with. A bot that needs embeddings asks for them in its constructor; one that doesn't, doesn't.

class BaseExpert:
    @inject
    def __init__(self, config: Config, memory: MemoryInterface, brain: BrainInterface):
        self.config = config
        self.memory = memory
        self.brain = brain

    def process(self, query: str, conversation_id: str, user_id: str) -> ChatResponse:
        history = self._prepare_history(query, conversation_id, user_id)
        context = self._prepare_context(query, conversation_id, user_id)
        response = self.brain.think(history, context)
        self.memory.add_message(
            role="assistant", content=response["content"], conversation_id=conversation_id
        )
        return ChatResponse(
            response=response["content"],
            conversation_id=conversation_id or "default",
            additional_kwargs=response["additional_kwargs"]
        )

    @abstractmethod
    def _prepare_context(self, sentence: str, conversation_id: str, user_id: str, **kwargs: Any) -> str:
        """Prepare the context for the brain."""

Why there's a brain in the middle

Experts never talk to a model client directly. They talk to a brain, which has think and stream methods, sync and async. One brain just sends history and context to the model; the other can call tools. Which model sits behind it (OpenAI, Azure, Gemini, Vertex, llama.cpp) is a setting.

It's one more layer, and I'm not sure every project needs it. I wanted provider details out of the experts. Gemini, for example, lives in its own client file plus a branch in the LLM factory.

Wiring it up with an injector

Everything is assembled in config_injector.py. It reads the configuration, asks a factory for each component, and binds the result as a singleton. Choosing in-memory history over MongoDB, or Chroma over the in-memory vector store, is a config value rather than a code change.

The injector also means a test can swap in a fake memory or brain. There is a tests folder, but I never measured how much of this it covers.

The PDF path, and a user id I never filled in

The RAG expert loads a PDF, tags every page with a user id and document id, splits it into semantic chunks and indexes them. I made each step fail loudly: a missing file, an empty extraction or zero chunks raises an error instead of quietly indexing nothing. Retrieval filters on the user id.

That filter only means something if the id is real, and right now the chat route passes an empty one. I built this as a single-user thing and never went back, so I wouldn't point it at several users' documents as it is.

What's still rough

The deep-research bot is the one that has needed the most fixing. Tavily search kept coming back empty, and the last commit in the repo is a fix for a recursion limit. The ChatEngine's expert switch only toggles between Q&A and RAG, so deep research can be chosen at startup but not switched to. The SQL memory backend is still commented out in the factory.

The README has drifted too. It calls the project production-ready and lists endpoints like /chat/message and /rag/upload, while the routes are /chat and /process-document. Take /docs on the running API over the README, and treat the "production-ready" as wishful.

Next I want to add one more expert from scratch and see how little code it takes. Then fix the above, starting with the user id.

Takeaways

• Keep what changes between bots, the behavior, in a small class, and put everything else behind interfaces.
• Build components in factories so memory, vector store and model provider are settings.
• Make ingestion fail loudly on empty input.
• Check the README against the code before trusting either.