Adding pages to your assistant's knowledge one at a time is fine for a dozen. For a documentation site or a large help centre it is not. There is a bulk importer for exactly this, and a companion button that re-fetches pages that have gone stale.
Both are straightforward, and both have behaviour that is easy to be surprised by later. This covers what each one does, and the two things worth knowing before you run them on a real site.
Two jobs on one screen
The bulk importer takes a set of URLs, fetches each one, and stores the extracted text as a knowledge source. The refresh button goes back over URLs you imported earlier and fetches them again, replacing the stored text with whatever is there now.
Neither job creates anything a human reads. Sources feed the assistant; they are not help articles and they do not appear in your help center. If you want the same material published for people as well, that is a separate importer on a separate screen — see seeding your knowledge base from pages you already have, which can turn these very sources into articles.
The distinction matters because the stored copy is what your assistant reads. It does not visit your website when a visitor asks a question — it reads text captured at import time. A page you rewrote last week is still answering with last month's wording until something re-fetches it. Importing is the easy half; keeping copies current is the half that decides whether your assistant is accurate a year from now.
What counts as a URL
You can give the importer a pasted list, a sitemap URL, or both together. A pasted list is split on almost anything sensible — new lines, commas, semicolons, spaces, tabs — so you can paste a column out of a spreadsheet without reformatting it.
Two filters then apply, and both are silent:
- Only http and https entries survive. Anything else is dropped without comment. Relative paths, bare domains with no scheme, and a stray heading you copied along with the column all vanish here. If your import comes back smaller than your list, this is the first place to look.
- Duplicates are removed, ignoring case. Supplying both a list and a sitemap that overlap is safe; each page is fetched once.
You can also set a title prefix applied to every page in the run. It is more useful than it sounds. Titles come from the page itself, and a help centre where forty pages are titled with the same site suffix produces a source list you cannot navigate. A prefix like the section name makes the list readable and makes it obvious later which import a row came from.
Where a page has no usable title, the URL is used instead. Titles are trimmed at two hundred characters.
Fifty at a time, on purpose
Each run imports at most fifty URLs. If you supply more, the extra are cut and you re-run with the next batch. Point the importer at a sitemap for a large site and you will be doing several passes.
This is a deliberate limit rather than a performance accident. Each URL is a live HTTP fetch, and an uncapped run against a thousand-page sitemap is a scrape of your own site inside a dashboard request. Fifty keeps each run short enough to watch and to judge.
Failures inside a run do not stop it. Every URL is attempted independently, and the summary separates three outcomes: pages imported, pages skipped because the fetch returned nothing usable, and pages that failed outright. You get a sample of the errors — the first handful, with the reason attached — and the titles of the first pages that landed, which is enough to confirm the extraction found real content rather than a navigation menu.
Read that summary before running the next batch. A run reporting forty-eight imports and two failures is healthy. A run reporting fifty skips means the fetch is reaching something other than your content, and importing another fifty will not fix it.
Imported pages go live immediately
This is the one to internalise. Pages imported in bulk are enabled the moment they land. There is no staging step and no review queue: the next visitor question can be answered from a page you imported thirty seconds ago and have not read.
Most of the time this is convenient. It stops being convenient when a fetch captured the wrong thing — a cookie banner, a login wall, a page of navigation with no article text, or an internal draft that happened to be reachable. That content is now knowledge, and your assistant will use it.
So build the review into your routine rather than relying on the import to hold anything back:
- Import a small first batch — five or ten pages — and open a few of the stored sources to see what was actually extracted before you commit to the whole site.
- After each batch, scan the imported titles in the summary. Repeated or obviously wrong titles are the cheapest signal that extraction is going astray.
- Ask the assistant two or three questions those pages should answer, and check that what comes back resembles the page rather than its chrome.
- Disable anything that should not be there. A source can be switched off individually without deleting it.
The one thing not to do is import several hundred pages across six runs and review none of them. That is how a login wall ends up being quoted to a customer.
Keeping the copies current
The refresh button asks a simple question: which imported pages have not been fetched for a while? The window is a fixed thirty days — the button offers no control for widening or narrowing it — and it takes up to fifty of the most overdue pages, oldest first, and fetches them again. What that button actually does is worth reading before you rely on it, because a fetch that fails still counts as a check.
Two conditions decide what is eligible, and one of them surprises people:
- Only URL sources are considered. Text you pasted and FAQs you wrote have no source to re-fetch and are left alone.
- Only enabled sources are refreshed. A source you switched off is never re-fetched. Switch it back on later and it carries whatever text it had when you disabled it, however old that is. If you park sources rather than deleting them, refresh them after re-enabling.
Because it takes fifty at a time, oldest first, a site with two hundred imported pages needs four runs to come fully current. Running it monthly on a large source list means some pages are always a cycle behind — which is fine as long as you know it, and an argument for keeping the source list to pages that genuinely answer questions rather than everything your sitemap lists.
The quiet failure worth knowing about
When a refresh cannot fetch a page — it has moved, it now returns an error, it sits behind a login — the row is marked with the failure and the reason is recorded against it. That much is good: you can tell a page that failed from a page nobody has refreshed.
But the refresh timestamp is updated too, even though nothing was fetched. That is intentional: it stops a permanently broken URL from consuming a slot in every subsequent run and blocking healthy pages behind it.
The side effect is the thing to watch. A broken page drops out of the stale list for your entire window. With a thirty-day setting, a page that started returning an error today will not be attempted again for thirty days — and in the meantime its old text is still enabled and still being served to visitors. Nothing looks wrong from the outside. The source list shows a recently refreshed row.
The defence takes a minute: after each refresh run, sort or filter your sources by sync status and look at the errors. A page that has failed twice in a row is not stale, it is gone, and the right move is to fix the URL or disable the source. Doing this each time you refresh is what keeps a knowledge base from slowly filling with confident answers drawn from pages that no longer exist.
What to measure
Track the count of sources in an error state. It should be near zero and it should not trend upward. A rising count is your site's structure drifting away from what you imported, and it will show up in answers long before anyone reports it.
Track the age of your oldest successful refresh. That number is the honest answer to how current your assistant's knowledge is, and it is usually older than people assume.
Then keep an eye on the size of the source list itself. Bulk import makes it easy to add hundreds of pages, and a large list is not the same as a good one — every page competes for the limited space in each reply. Pages that answer real questions earn their place; the rest are cost. When in doubt, import fewer pages and read them.
Pages are not the only source you already own. Your resolved tickets contain answers your team wrote by hand and never got round to documenting; which resolved tickets your AI can safely learn from covers how those are shortlisted and what has to come out before they are used.
And if what you hold is a folder of markdown rather than a set of URLs, splitting one markdown file into help articles is the import path built for that shape.