There are many scenarios when at times you may want to stop search engines from crawling your WordPress website or listing it in search results. For example, you are developing your website at a development location and do not want search engines to index it. WordPress comes with a built-in feature that allows you to instruct search engines not to index your site. In this article, we will guide you how to stop search engines from crawling a WordPress website.
Note: When this box is checked, WordPress adds some lines to your website's header and modifies your website's robots.txt file which asks the robots (web crawlers) not to index your pages. However, it is totally up to search engines to accept this request or ignore it. Even though most search engines respect this, some page or random image from your website may get indexed.
Updated on Oct 27, 2016
The techReview is an online magazine by Batoi and publishes articles on current trends in technologies across different industry verticals and areas of research. The objective of the online magazine to provide an insight into cutting-edge technologies in their evolution from labs to market.