Search engines such as Google and Bing deliver search results by using software called robot to crawl the servers on the internet looking for new web pages to add to search indices, and for updates to previously crawled pages.
Here are some of the reasons why a website is not indexed by the search engine:
URL of the website is restricted by robots.txt
The website is not well connected through multiple links from other websites on the web.
The design of the website makes it difficult for the search engine to effectively crawl its content.
The website was temporarily unavailable when the search engine tried to crawl it or the search engine received an error when the search engine tried to crawl it.
The website was launched after the search engine’s most recent crawl was completed.
The website is involved in some illicit practices that may lead to a site being removed entirely from the search engine index or otherwise penalized.
