From a model call to a configurable chatbot

It started as a Gradio chat

The first version was about as small as a chatbot gets: a Gradio page, a model, and MongoDB so the conversation survived a restart. The changelog for v1.0 even lists what it couldn't do yet. It couldn't handle more than one user, and the history wasn't saved per user id.

v1.0.1 added streaming, a way for the bot to pick up details about the user, and an agent with a few basic tools. Once there was an agent, a Gradio callback was a bad place for everything. Prompt building, the model call and output handling were all tangled together, and an empty answer could come from any of them. So I moved to a FastAPI backend with a Next.js frontend, and the bot became a class the API just calls.

Making the bot a pipeline

Inside the backend, one request is a LangChain chain with a few steps. Load the history for the conversation, optionally anonymize, call the model, optionally put the original values back, then store the exchange. I gave the steps names like LoadHistory and AnonymizeSentence, which turned out to be the most useful small change in the project. In a LangSmith trace you see those names instead of a pile of anonymous lambdas, so when an answer looks wrong you can tell which step handed over the bad input.

History is loaded before anonymizing on purpose. If only the new message were anonymized, a name replaced in one turn would show up in plain text in the next prompt, because it's sitting in the stored history. The prompt templates live on the LangSmith hub, so changing a prompt doesn't mean touching the Python.

history_loader = RunnableMap({
    "input": itemgetter("input"),
    "agent_scratchpad": itemgetter("intermediate_steps") | RunnableLambda(format_log_to_str),
    "history": itemgetter("conversation_id") | RunnableLambda(self.memory.load_history)
}).with_config(run_name="LoadHistory")

if self.config.enable_anonymizer:
    anonymizer_runnable = self.anonymizer.get_runnable_anonymizer().with_config(run_name="AnonymizeSentence")
    de_anonymizer = RunnableLambda(self.anonymizer.anonymizer.deanonymize).with_config(
        run_name="DeAnonymizeResponse")

    agent = (history_loader
             | anonymizer_runnable
             | self.chain.chain
             | de_anonymizer
             | ReActSingleInputOutputParser())
else:
    agent = history_loader | self.chain.chain | ReActSingleInputOutputParser()

Hiding phone numbers from the model

The anonymizer is the part worth describing. If the model is a hosted API, I'd rather it never saw a real phone number. I wrapped LangChain's Presidio reversible anonymizer: it detects whether the message is Vietnamese or English, swaps personal details for realistic fake ones, and after the model replies it swaps the real ones back. The user sees a normal answer and the provider only ever sees the stand-ins.

I picked the reversible kind because a plain [REDACTED] token wouldn't survive a conversation. If someone gives their number and refers to it two messages later, the fake value has to stay the same across turns. The price is that the backend now holds a mapping between real and fake values, which is its own thing to look after.

The traces below are from a Vietnamese chat where I typed a phone number. The first shows the real number going into the anonymize step and a different one coming out. The second is the model call, which only has the fake one. The third is the real number coming back in the reply.

The annoying parts

It's one phone number in one conversation, so I wouldn't call that a test. The extra recognizers I added are a list of US-style identifiers (SSN, passport, driver license and so on), which doesn't help much with Vietnamese text. The anonymizer is also off in the example env file, so by default none of this happens.

The repo also shows its age. There's a Config class and a separate pydantic Settings module that both read configuration, and I never checked that the anonymizer flag actually reaches the bot through the Settings path. I'm also not sure per-user history works end to end, since that was still on the v1.0 pending list.

I never measured latency, answer quality or how many identifiers slip through the anonymizer. Next I'd write a small set of fake conversations with seeded phone numbers and count how many end up in the model input, the stored history and the trace.

Takeaways

• Name the steps of a chain. Traces are far easier to read when LoadHistory and AnonymizeSentence are labelled.
• Anonymize the stored history too, or a replaced name comes back in the next prompt.
• Reversible substitution keeps prompts readable but means you now hold a mapping of real and fake values.
• I tested the anonymizer on one phone number. I haven't measured how much it misses.