Botify, an France-based SJE Group investment company, has posted an interesting article that can help organisations optimise their “crawl budget”
and increase their website ROI. This is a summary of their key points.
Botify defines “crawl budget” as the maximum number of pages a search engine will crawl on any given website. Given that search engines don’t have unlimited time and resources to crawl all the content on the web all the time, they prioritise what pages they’ll look at based on how healthy and popular a site is.
A limited “crawl budget” is why search engines miss more than 51% of an organisation’s website content – potentially missing out on appearing in web searches.
What to do
Botify suggests doing four things:
1. Use your robots.txt file to tell search engines what not to crawl. By using your site’s robots.txt file, you can tell search engine bots what to crawl and what to ignore. This helps preserve your “crawl budget”.
2. Clean up your internal linking. Don’t waste your “crawl budget” by pointing search engine bots through multiple middlemen (e.g. chains and loops) to find your content. Instead, link to the ultimate destination.
3. Make sure your XML sitemap is up to date. Search engines use a site’s XML sitemap to index and search for content. Avoiding common XML sitemap mistakes helps improve your “crawl budget”. Common mistakes include: Listing non-compliant pages like non-200s, non-canonicals, non-HTML, and no-indexed URLs; Forgetting to update your sitemap after URLs change; and Omitting important pages.
Botify can be engaged to audit your website for inefficiencies. They can also be engaged to help optimise your website for enhanced SEO and hence ROI. Botify’s clients include Marriott, Expedia, Conde Nast, Clarins, Glassdoor, John Lewis, Merkle and more.