In plain English
It's a librarian walking every aisle, noting each new book on the shelves. If a book is in a locked room with no door, the librarian never finds it, however good it is.
Used in a sentence
“Google crawled the new pages within a day of launch, because the sitemap and internal links pointed straight to them.”
“We built a page for every town we serve, but nothing linked to them, so Google never crawled them.”
In detail
Google's crawler, Googlebot, finds pages by following links from pages it already knows and by reading sitemaps. It downloads each page, reads its text and code, and passes it on to be indexed. Pages that nothing links to, pages blocked in robots.txt and pages that are slow or broken are crawled less, or not at all.
Crawling is only the first step. A page can be crawled and still left out of search results if Google decides it isn't useful, is a duplicate, or has been marked noindex. Clear internal links, an up-to-date sitemap and a quick, reliable server make crawling easy.
Why it matters for your business
If Google can't crawl a page, it can't rank it, however good the page is.
- Link every page from somewhere
- Keep your sitemap up to date
- Don't block pages you want found
Easy to mix up
Crawling isn't indexing. Crawling is finding and reading the page; indexing is deciding to store it and show it in results. A page can be crawled and never indexed.
Related terms
Indexing, Sitemap and Robots.txt. Or browse the whole glossary.
