Crawling and indexing are the two steps that have to happen before a page can rank at all. Neither is visible to a visitor, and confusing them is why so many people conclude that search is a mystery.
This article separates them and gives you the six reasons a page usually goes missing.
The difference between crawling and indexing
Crawling is discovery. Software follows links around the web, arrives at your page, and downloads it. That is all. A crawl means the crawler fetched the page, not that anybody decided to keep it.
Indexing is the decision that follows. The search engine reads what it fetched, works out what the page is about, and files it so it can appear in a search. Plenty of crawled pages never reach the index. Ranking is the third step, and how a search engine chooses between indexed pages picks up where this article stops.
So a page can be crawled and absent, which surprises people. Search Console reports the two separately for exactly this reason, and reading the wrong one sends you off fixing something that was never broken.
Six reasons a page never appears
Almost every missing page has one of these causes, and each has a different fix.
- Nothing links to it. Crawlers follow links, so nothing ever reaches a page nobody points at.
- Robots rules block it. A leftover instruction from development commonly disallows a whole section.
- The page says not to index it. A no index tag does exactly what it says, and it is easy to leave on by accident.
- A duplicate wins. The search engine decided another page says the same thing and kept that one instead.
- Quality. Crawlers fetch thin pages and the index drops them again, which is a judgement rather than a technical fault.
- It is too new. Discovery takes time, particularly on a small site with few links.
Notice that only two of those are technical. The others are about links, duplication and quality, which is why a purely technical approach to crawling and indexing often stalls.
Reading the Search Console report
The page indexing report splits your pages into indexed and not indexed, and it gives a reason for each one it excluded. Those reasons are more precise than they look.
"Crawled, currently not indexed" means the crawler fetched the page and the index turned it down. Usually the honest reading is that the page is thin or duplicates something else, and no amount of resubmitting changes that.
"Discovered, currently not indexed" means the search engine found the page and never fetched it. On a large site that often points at crawl budget: the search engine decided not to spend more time here for now, frequently because so many low value addresses exist.
"Duplicate, Google chose a different canonical" means it picked another page over yours. Check whether the other page is yours too, because if it is, this is a filter or parameter problem rather than a content one.
Crawling and indexing on a large site
Small sites rarely have a crawl budget problem. Large ones do, and it is almost always self inflicted.
Filters, sort orders, search result pages, session parameters and printer friendly versions all generate addresses. A shop or a listings site can produce hundreds of thousands, and the crawler spends its time on those instead of your actual pages.
The fix is not to ask for more crawling. It is to stop generating addresses nobody needs, so the attention you already get lands somewhere useful.
What actually speeds up indexing
Two things reliably help and one thing does not.
Links help, both internal and external. A new page linked from a page crawlers visit often turns up quickly. A sitemap helps too, because it lists what exists and when each page last changed. Between them they answer most crawling and indexing delays on a small site.
Repeatedly requesting indexing in Search Console does not help. It is useful once, for a page you have just fixed, and it is not a lever you can pull harder.
A short routine for crawling and indexing problems
When a page is missing, work through it in this order rather than guessing.
- Check the page indexing report and read the specific reason given.
- Open the page in the URL inspection tool to see what the crawler actually got.
- Check that something on your site links to it.
- Check the page for a no index instruction and the robots file for a block.
- Read the page honestly and ask whether it deserves a place.
That sequence answers nearly every case, and it takes about ten minutes per page. Most crawling and indexing problems turn out to be one setting or one missing link rather than anything deep, which is why working through the list beats theorising about it.
Frequently asked questions
What is the difference between crawling and indexing?
Crawling is discovery: software follows links, arrives at your page and downloads it. Indexing is the decision that follows, where the search engine reads the page and files it so it can appear in a search. Plenty of crawled pages never reach the index.
Why is my page crawled but not indexed?
The crawler fetched the page and the index then turned it down, usually because it is thin or says much the same as another page. That is a quality judgement rather than a technical fault, so resubmitting changes nothing. Improve or merge the page instead.
What does "discovered, currently not indexed" mean?
The search engine knows the page exists and has not fetched it yet. On a larger site this usually points at crawl budget, where so many low value addresses exist that the crawler spends its attention elsewhere before reaching the pages you care about.
How long does indexing take for a new page?
Anything from hours to weeks, depending mostly on how well linked the page is and how often the rest of your site gets crawled. A new page linked from a page that is crawled daily appears far faster than one nothing points at.
Does submitting a sitemap help crawling and indexing?
It helps, because it lists what exists and when each page changed, which matters most for pages with few internal links. It does not force indexing. A search engine can still fetch a page listed in a sitemap and turn it down on quality grounds.
What is crawl budget and should I worry about it?
It is the attention a search engine is willing to spend on your site. Small sites rarely need to think about it. Large ones do, and the problem is usually self inflicted, caused by filters and parameters generating far more addresses than there are real pages.
Can I force Google to index a page?
No. Requesting indexing in Search Console is useful once, for a page you have just fixed, and it is not a lever you can pull harder. Links and a well maintained sitemap do more to help crawling and indexing than repeated requests ever will.
How do I find pages nothing links to?
A crawler tool lists them in minutes, because it can only reach pages by following links. Without one, list your important pages and check by hand which you can reach from the home page in three clicks. Anything you cannot reach is effectively invisible.