Easy methods to Cease Search Engines from Crawling a WordPress Website

  • December 31, 2022
  • elkenz
  • 7 min read

Just lately, one in all our customers requested us how they will cease search engines like google from crawling and indexing their WordPress web site.

There are a number of eventualities if you would wish to cease search engines like google from crawling your web site and itemizing it in search outcomes.

On this article, we’ll present you learn how to cease search engines like google from crawling a WordPress web site.

How to Stop Search Engines From Crawling a WordPress Site

Why Cease Search Engines From Crawling a WordPress Website?

For many WordPress web sites, search engines like google are the largest supply of visitors. Chances are you’ll ask, why would anybody wish to block search engines like google?

Listed here are some conditions if you gained’t need search engines like google to index your web site:

  • When beginning out, chances are you’ll not know learn how to create an area growth setting or a staging web site, and as an alternative develop your web site whereas it’s reside. You gained’t need Google to index your web site when it’s beneath building or in upkeep mode.
  • There are additionally many individuals who use WordPress to create personal blogs. They don’t need them listed by search engines like google as a result of they’re personal.
  • You should use WordPress for mission administration or an intranet. In these instances, you gained’t need your inside paperwork to be publicly accessible.

A typical false impression is that should you don’t have hyperlinks pointing to your area, then search engines like google will in all probability by no means discover your web site. This isn’t utterly true.

For instance, there could also be hyperlinks pointing to your web site as a result of the area title was beforehand owned by another person. Additionally, there are millions of pages on the web that merely record domains. Your web site could seem on a kind of.

With that being stated, let’s check out learn how to cease search engines like google from crawling your web site. We’ll cowl 4 strategies:

Methodology 1: Asking Search Engines to not Crawl Your WordPress Website

That is the only technique however doesn’t totally defend your web site from being crawled.

WordPress comes with a built-in characteristic that lets you instruct search engines like google to not index your web site. All you have to do is go to Settings » Studying and test the field subsequent to ‘Search Engine Visibility’.

Search Engine Visibility Setting in WordPress

When this field is checked, WordPress provides this line to your web site’s header:

<meta title="robots" content material="noindex,observe" />

WordPress additionally modifies your web site’s robots.txt file and provides these strains to it:

Consumer-agent: *
Disallow: /

These strains ask robots (internet crawlers) to not index your pages. Nonetheless, it’s completely as much as search engines like google to just accept this request or ignore it. Though most search engines like google respect this, there’s nonetheless an opportunity that some pages or pictures out of your web site could get listed.

If you wish to make it unimaginable for search engines like google to index or crawl your web site, then you have to to password defend your WordPress web site utilizing Strategies 3 or 4.

Methodology 2: Asking Search Engines to not Crawl Particular person Pages

You may want search engines like google to crawl and index your web site, however not embody sure posts or pages in search outcomes pages.

The simplest method to try this is utilizing the All in One search engine optimisation (AIOSEO) plugin. It’s the finest search engine optimisation software for WordPress and is trusted by over 3 million companies.

For this tutorial, we’ll be utilizing the AIOSEO free model because it consists of the search engine optimisation Evaluation software. There’s additionally a premium model of AIOSEO that gives extra options like sitemap instruments, redirection supervisor, schema markup, robots.txt editor, and extra.

The very first thing you’ll have to do is set up and activate the AIOSEO plugin in your web site. You may discover ways to set up and configure the plugin by following our step-by-step information on learn how to arrange All in One search engine optimisation for WordPress.

As soon as the plugin is ready up, you should utilize it to ask search engines like google to not index sure posts and pages. Once more, it’s completely as much as search engines like google to just accept this request or ignore it.

Merely edit the put up or web page that you simply don’t wish to be listed. It’s worthwhile to scroll all the way down to AIOSEO Settings on the backside of the WordPress editor after which click on the ‘Superior’ tab.

AIOSEO Advanced Settings

Discover that the article is utilizing the default robots settings. To alter this, you have to swap the ‘Use Default Settings’ toggle to the off place.

Asking Search Engines Not to Index a Single Page

Now you possibly can click on the ‘No Index’ checkbox. As soon as the put up is revealed or up to date, search engines like google shall be requested to not index it.

Methodology 3: Password Defending an Whole Website Utilizing cPanel

In case your WordPress internet hosting supplier presents cPanel entry to handle your internet hosting account, then you possibly can defend your complete web site utilizing cPanel. All you need to do is log in to your cPanel dashboard after which click on on the ‘Listing Privateness’ icon within the ‘Recordsdata’ part.

Using cPanel's Directory Privacy Feature

Subsequent, you have to discover the folder the place you put in WordPress. Normally, it’s the public_html folder. After that, you have to click on the ‘Edit’ button subsequent to that folder.

Edit the Root Folder

Be aware: In case you have a number of WordPress websites put in beneath public_html listing, then you have to click on on the public_html hyperlink to browse these websites, after which edit the folder for the web site you wish to password defend.

This brings you to a display the place you possibly can activate password safety.

Merely test the field that claims ‘Password defend this listing’ and click on the ‘Save’ button. When you like, you may also customise the title for the protected listing.

Check the Box to Password Protect This Directory

You will note a affirmation message saying that the entry permissions for the listing have been modified.

Subsequent, it is best to click on the ‘Go Again’ button.

cPanel's Directory Privacy Success Message

You’ll be taken to a display the place you possibly can create a username and password that may have to be used when accessing this listing.

It’s worthwhile to enter a username and password after which verify the password. Be certain that to notice your username and password in a secure place, reminiscent of a password supervisor app.

Add a Username and Password

When you click on the ‘Save’ button you may have efficiently added password safety to your WordPress web site.

Now, every time a person or search engine visits your web site they are going to be prompted to enter the username and password you created earlier to view the location.

A Username and Password Are Now Needed to Access Your Website

Methodology 4: Password Defending WordPress With a Plugin

In case you are utilizing a managed WordPress internet hosting answer, then chances are you’ll not have entry to cPanel. In that case, you should utilize a WordPress plugin to password defend your web site.

Listed here are the 2 hottest options:

  1. SeedProd is the #1 coming quickly and upkeep mode plugin for WordPress, used on over 800,000 web sites. It comes with full entry management and permissions options that you should utilize to cover your web site from everybody together with search engines like google. We’ve got a step-by-step how-to information for SeedProd.
  2. Password Protected is a quite simple option to password defend your WordPress web site with a single password (no person creation wanted). See our step-by-step information on learn how to password defend a WordPress web site.

We hope this text helped you cease search engines like google from crawling or indexing your WordPress web site. You may additionally wish to discover ways to pace up your web site efficiency, or see our final step-by-step WordPress safety information for inexperienced persons.

When you favored this text, then please subscribe to our YouTube Channel for WordPress video tutorials. It’s also possible to discover us on Twitter and Fb.

Leave a Reply

Your email address will not be published. Required fields are marked *