Enchant AI website data source
Enchant AI takes data from multiple sources. One of the sources available is the website data source, which gathers insights from publicly accessible web pages on your website or documentation site.
Automated Content Discovery: Pull content directly from your existing website or documentation without manual entry or duplication
Stay Current: With automatic weekly recrawling, your Enchant AI stays up-to-date with your website changes
Base URLs are the starting points for the web crawler. You can add one URL per line. Only pages beneath these paths will be indexed.
All your base URLs need to be on the same subdomain, so you can't mix the following in the same crawler:
But you can cover multiple sections of your site by listing both of these as base URLs:
If you want to crawl your entire site, use a root path like in the URL https://docs.example.com which will cover all paths under the subdomain.
Exclude paths let you skip specific sections of your website during crawling.
One pattern per line. They support wildcards (*).
/admin excludes the exact path.
/docs/*/private excludes any private folders within your docs section.
/*.pdf to skip any PDF files.
Common use cases for exclude paths are:
excluding non-documentation pages like /blog/* or /news/*
skipping dynamic or user-generated content like /users/* or /*/comments
exclude admin or private section slike /admin* or /private/*
When crawling your website, you can use CSS selectors to control exactly what content gets indexed. Think of this as a two-step filter: first you tell us what to include, then you tell us what to exclude from those included sections.
CSS Includes let you specify which parts of the page contain your actual content. For example, if your documentation is always inside an <article> tag with class documentation, you can specify that selector (article.documentation) and we'll only index content from those areas. This is helpful when you want to skip navigation menus, headers, footers, and sidebars entirely. If you leave this empty, we'll try to automatically detect the main content area of each page.
CSS Excludes work the opposite way - they remove specific elements from whatever content we're indexing. This is useful for removing things like table of contents sidebars, comment sections, or "share this page" buttons that might be nested inside your main content area. Excludes are applied after includes, so they can clean up elements within the sections you've already selected.
You can use any standard CSS selector syntax. Some common examples include:
article or nav - to capture tags like <article>
.content or .sidebar - for classes
#main-content - for ids
[role="main"] - for attributes
Separate multiple selectors with commas.
When sitemap support is enabled (which it is by default), the crawler looks for and uses sitemap files to discover pages more efficiently than just following links. This is especially helpful for large sites or those with JavaScript navigation where not all pages are easily discoverable through regular links.
Query parameters are the URL components that come after the ? symbol, like ?page=2&lang=en. By default, all query parameters are stripped during crawling to avoid indexing duplicate content. However, you can use the Allowed Query Params field to preserve important parameters that actually change the content.
You will typically want to allow for functional query parameters, for example those for version, language, locale, or pagination.
Enter a comma separated list of parameter names. All other parameters will be stripped while crawling to to prevent duplicate indexing.
The Enchant crawler strictly respects your robots.txt file, which is the standard way websites tell crawlers which paths are allowed or disallowed.
At the start of each crawl, we fetch your robots.txt and follow Disallow, Allow and Sitemap directives in the file. Our crawler can be targeted directly in your robots.txt file as Enchant-Crawler.
Each crawler has a maximum limit of 1000 pages.
Each crawler waits a minimum of 5 seconds between page fetches to prevent overwhelming your server
When setting up your crawler, it's usually best to start narrow and then expand. Begin with a specific base URL like /docs/help rather than crawling your entire site. Once the initial crawl is working well, you can add more base URLs to expand coverage.
Use CSS selectors to reduce noise in your indexed content. Include only the main content container and exclude navigation, footers, sidebars, and ads. This results in cleaner, more relevant data for Enchant AI because the AI isn't distracted by the same repeated text that appears on every page.
When testing exclusion patterns, start with broad excludes like /admin, then add more specific patterns like /docs/*/internal as you discover what needs filtering.
After each crawl completes, monitor the indexed page count. If it's too high, tighten your base URLs or add more exclusions. If it's too low, check whether your rules are overly restrictive.