Why are there no records in my website collection?
Our crawler may have encountered issues with your site during initial crawl. If our crawler encounters errors or if you have redirects or canonicals which cause redirect loops, we will abandon crawling.
- The first thing you should do is to add your website’s homepage URL to page debug tool.
- The page debug tool will indicate whether the website crawler was able to crawl your homepage. Read the fetch log to see if there were any redirects before the page was parsed(crawled) or any errors
- There are common issues we see:
- Canonical tags that are all set to your homepage - in this case we may only ever end up indexing your homepage. To resolve, update the canonical tags to the correct page or they should be left blank.
- Website homepage is in a canonical loop (i.e. homepage redirects to a different page, which redirects back to the homepage, and ends in a loop). To resolve, either remove the canonical tag from the website’s homepage or add the correct canonical tag.
- Using path-relative URLs instead of root-relative URLs - Path-relative URLs can cause redirect loops. The crawler does not follow path-relative URLs (i.e. if there is no base path on the page <a href="example-page">). Please make sure you use root-relative (e.g. <a href="/example-page">) or Absolute URLs (e.g. <a href="https://www.acme.com/example-page">) on your site.
- Sajari does not fully crawl third party sites until our pingback code is installed.
- Please don't add "amazon.com", "google.com", or other domains that you do not own as your domains to be crawled - we do not fully crawl third party domains without authentication.
Get started today.