Give an AI agent your company documents by building a small AI agent knowledge base. Pick one job, gather only the documents that job needs and clean them up. Then upload them or connect the folder where they live. The agent searches those documents when it answers. You do not need to train a model on them.
This guide is for operations, marketing, sales and support leads, and for the IT person who approves new tools. It compares the ways to give access and sets out what to test before anyone relies on the answers.
What is an AI agent knowledge base?
It is the set of documents an agent is allowed to look things up in. Think of the handbook you would give a new starter for one role. It holds the policies, price lists, product sheets and past answers that role needs, and nothing else.
Start small. A knowledge base for a support job might hold the returns policy and the product guides. A knowledge base for a sales job might hold the current price list and the approved case studies. Files that do not help with that job make wrong answers more likely.
Our guide to building your first agent from a template shows where documents fit in that flow.
What is RAG, in plain words?
RAG stands for retrieval-augmented generation. The agent first searches your documents for the passages that match a question, then writes its answer using those passages. The term is usually traced to a 2020 research paper by Lewis and colleagues. The paper describes models that combine what they learned in training with a searchable index of text.
In 2020, the same paper called two things open research problems for models that hold knowledge only in their trained weights. One is showing the source of a decision. The other is updating what the model knows. The paper proposes retrieval as a way to address both. With retrieval, you usually update the answer by updating the document.
Do you need to train or fine-tune a model on your documents?
For most business uses, no. If you want answers from documents that change, retrieval is usually the better fit. Fine-tuning changes the model itself, so a new price list means another training run. Retrieval only needs the new file, once it is indexed again.
| Retrieval (RAG) | Fine-tuning | |
|---|---|---|
| What changes | The agent searches your files each time | The model’s learned weights are adjusted |
| When a document changes | Replace the file and index it again | Train again |
| Pointing to the source | Possible, because the passage is known | Harder, because the knowledge sits inside the model |
| Typical use | Policies, product facts, internal how-tos | A fixed writing style or a narrow, stable task |
This table is our own summary of the general approach, not a vendor benchmark. Fine-tuning has its place, but it is rarely the first step for “answer from our documents”.
Upload, connect or MCP: which way should you give access?
Choose by where your documents already live and how often they change. There are three common patterns, and a team can mix them.
| Upload to a knowledge base | Connect a folder | MCP server or API | |
|---|---|---|---|
| Best for | Stable files such as handbooks and policies | Files that change often, kept in Drive or OneDrive | Internal systems and databases with no folder to connect |
| When files change | You upload the new version | The tool syncs or reads the folder, so check how often | Depends on the system behind it |
| Setup effort | Lowest | Medium, needs someone to approve the connection | Highest, needs technical help |
| Who decides what it can reach | You, by choosing what to upload | The person who sets up the connection | The person who builds or configures the server |
| Watch out for | Old copies left behind after the source changes | A folder that holds more than the job needs | A connection with broader reach than the task |
MCP is a standard way to connect AI applications to other systems. Our explainer on what MCP is covers it for non-developers.
Whichever route you pick, give the narrowest access that does the job. Google’s guidance on choosing Drive API scopes, as of October 2026, says to choose the most narrowly focused scope possible. It also says to avoid requesting scopes your app does not need. The same habit applies to any connection. Choose a read-only connection and named folders where your setup allows it.
Some platforms also carry source permissions into the answers. Microsoft’s page on managing access to agents in SharePoint, as of October 2026, covers this case. A user who can reach the agent but not the site it references gets answers without content from that site.
How to set it up, step by step
Work through these steps in order, and finish the test before you widen access.
- Pick one job, such as answering staff questions about leave or drafting replies to product enquiries.
- List the documents a new starter would need for that job. Leave out everything else.
- Clean them. Remove drafts, duplicates and expired policies, and take out passwords and personal details that the job does not need.
- Name an owner. One person is responsible for keeping each source current and for removing files that go out of date.
- Connect with the narrowest access that works, as described above.
- Write a test set before anyone uses the agent. Collect a short list of questions with answers you already know, for example 20, drawn from real questions your team gets.
- Run the test set and compare each answer with the known one. Check that each answer matches what the right document says. Fix the source file or the instructions, then run it again.
- Release to a small group and review a sample of answers each week.
Include a few questions the documents cannot answer. A good agent should say it does not know instead of guessing. Include a question whose answer changed recently, to see whether an old file is still being used.
What can go wrong with documents an agent can read?
Four problems are worth planning for: old or conflicting files, sharing too much, personal information and instructions hidden in documents.
Old or conflicting files. If two policies disagree, the agent may quote either. This is why step 4 names an owner.
Sharing too much. An agent may reveal content to someone who could not open the file themselves. Decide who will use the agent and what they are allowed to see before you connect a folder.
Personal information. The Office of the Australian Information Commissioner published guidance on commercially available AI products on 21 October 2024. As a matter of best practice, it recommends that organisations do not enter personal information, and particularly sensitive information, into publicly available generative AI tools. That advice is about public tools. Before you add staff or customer records to any agent, check with your adviser what applies to your business.
Instructions hidden in documents. A file can contain text that tells the model to behave differently. OWASP calls this indirect prompt injection. It happens when a model takes in content from external sources, such as files. That content then changes the model’s behaviour in unexpected ways. Do not add documents from outside your organisation without checking them first, including any hidden text. Keep an agent’s ability to act narrow, so that a bad document can do less.
Company documents in Xagent
Xagent is an enterprise agent platform by Xinference. It has a knowledge base where you upload documents, and agents search them with hybrid search, which combines keyword and semantic matching.
Native connectors include Google Drive and Microsoft OneDrive. Other systems, including internal ones, connect through MCP servers. You can save a task as an agent that keeps its role, model, tools and knowledge. Every tool call and result is visible while a task runs.
To see how this works with your own documents, book a demo.
Questions
Can ChatGPT read my company documents?
General chat tools can usually read files you add to a conversation. Check your provider’s own documentation for what each plan supports.
Is RAG better than fine-tuning?
For answering from documents that change, retrieval is usually the better fit, because you replace a file instead of training again. Fine-tuning suits a fixed style or a narrow, stable task.
How many documents should an AI agent knowledge base start with?
Start with the documents for one job. There is no standard number. If a new starter would not need a file for that job, leave it out.
Should an AI agent see personal information in documents?
Only if the job needs it. As a matter of best practice, the OAIC recommends not entering personal information into publicly available generative AI tools. Check what applies to your business before adding it.
How do I connect an AI agent to a database?
Usually through an API or an MCP server that exposes only the data the job needs. Avoid handing over raw database credentials, and ask a technical colleague to set the connection up.
Try Xagent. See all use cases, or book a demo on your own workflow.
