Should I train the AI on my company's data, or give it the data when it answers?
The short answer
For knowledge about your business, give it the data when it answers. This method is called retrieval: the agent first searches your own documents, then answers from what it found. Training the model further on your data, called fine-tuning, is in my view better suited to shaping how it writes and behaves. A 2023 study found it a weak way to teach it new facts.
Two ways to give a model your knowledge
Fine-tuning means training the model a little more, this time on your own examples. The model’s internal numbers change, and your material becomes part of the model itself.
Retrieval leaves the model as it is. Your documents are kept in a searchable store. When a question arrives, the agent first searches that store, picks the passages that matter, and places them in front of the model. The model then answers from those passages. The full name is retrieval-augmented generation, usually shortened to RAG: retrieve first, then generate. The method was described in a 2020 research paper, which found that answers produced this way were more specific and more factual than answers from the model’s memory alone.
What the research says about learning new facts
A 2023 study compared the two methods directly, for the job of giving a model new knowledge. It tested one common kind of fine-tuning. Retrieval consistently did better, both for facts the model had seen before and for completely new facts. The authors also found that models struggle to learn new facts through this kind of fine-tuning.
Why retrieval suits a business
- It stays current. When a price or a contract changes, you update the document. Nothing needs retraining.
- It can show its source. The agent can say which document and which passage it used, so the answer can be checked.
- It respects who may see what. You can limit the search to the documents a given person is allowed to read. Facts mixed into a model’s internal numbers cannot be separated out like that.
Where fine-tuning still fits
My own view: if fine-tuning has a place, it is in shaping how the model behaves, not what it knows. For example, a fixed format, a particular style, or a narrow task repeated thousands of times in the same way. It is a later step, if it is needed at all.
What this means for you
Start with retrieval. The real work is then not in the model but in your documents: whether they exist, where they live, and whether they are current. The question Does my company’s data have to be clean before we start with an AI agent? covers where to begin.