Organisations connecting an assistant to their documents usually find that answers are poor, and blame the model. The cause is more often the documents, because of how retrieval systems handle them.
Documents are split before they are searched
Retrieval works by dividing files into passages of a few hundred words, indexing each, and fetching the ones matching a question.
The model only sees the fetched passages, so anything not contained within a retrieved fragment is unavailable no matter what the document says elsewhere.
A file that reads well as a whole can perform badly once cut into pieces that individually make no sense.
Each section should stand alone
The practical requirement is that a passage carries enough context to be understood without what precedes it.
That means naming the subject rather than relying on it, avoiding references such as the above process, and repeating the qualifying condition rather than assuming it carries down the page.
Repetition that feels redundant to a human reader is what makes a fragment usable in isolation.
Headings do most of the work
Splitting is usually guided by document structure, so real heading levels produce clean boundaries and bold text pretending to be a heading does not.
Descriptive headings help further, because heading text is often prepended to each fragment and becomes part of what is matched.
A section titled Overview contributes nothing to retrieval, while one titled Refund eligibility for annual subscriptions contributes a great deal.
Tables and layout survive badly
Multi-column layouts, text in images and complex tables are frequently extracted in the wrong order or lost entirely.
Any content that only makes sense visually should also exist as prose, since the extractor reads text and not arrangement.
Scanned documents need a text layer before they contain anything retrievable at all, and a surprising proportion of internal archives do not have one.
Superseded material is the main source of wrong answers
Retrieval has no sense of which version is current and will happily return a policy withdrawn three years ago.
Dating each document, removing obsolete files from the indexed set, and stating the effective period inside the text all reduce this.
Most bad answers from a well-built system turn out to be correct quotations from a document that should not still have been there, which makes archive hygiene a larger factor in quality than any tuning of the retrieval itself.