If You’re Looking for the Commonly Searched Term “How To Create the Best Robots.txt Settings For Wordpress?” (or) “How to Create Robots.txt for my Website?”. You are at the Right Place to Get Your Answers & Solve the Issue/Error.
You May Also Like: How to Fix “ERR_GFX_INIT Failed to initialize graphics device. Please reboot or reinstall the latest drivers.”Is the error in Red Dead Redemption 2?
Most People Like to do blogging and most of them Always wanted to get more visitors. Some peoples achieve it quickly But Most of them are not. This is due to low-quality writing, Not Properly using Long-tail Keywords, Good Keywords, and most Importantly spammy Backlinks. Before Going to the Topic, Check your Website to have Enough articles with High-Quality Content and Remove the Spammy Backlinks If anything is present on your site. You Can Check the Spammy Backlinks using SEMrush Backlink Audit Tool or Ahrefs Site Explorer Tool. When all this is fine you can Create a Custom Robots.txt File. A Proper Indexation Can be Done By Using the Proper Robots.txt File.
Robots.txt is a text file that contains a few codes that instruct web search engines about how to index your websites and crawl your website properly. It can be created or written by using notepad itself. It Doesn’t Need Any Coding Knowledge. Basic Knowledge is Enough To Manage It.Robots.txt File has a huge impact on Seo’s point of View. Without the Proper Usage of this code leads to negative impacts on SEO and decreases your Website’s Seo Ratings and Rankings Also.Most Important Things You Must Know Before Using a Robots.txt file. A Robots.txt file should be named as “robots.txt” (not as Robots.txt, robots.TXT – Since it is Case Sensitive). A robots.txt file is usually placed in the top-level directory of a site. You can Block the Crawling and Indexing of Certain Pages, Posts from Search Engines with help of Robots.txt Files. So, You need to Use it Carefully. If your Blocking a URL gets Wrong then it may affect your website SEO.
For WordPress Users, Creating a Robots.txt is So Easy. You can use Popular WordPress Seo Plugins for Creating a Robots.txt File. I Would Recommend Yoast Seo for Creating a Robots.txt.Yoast Seo Plugin usually Creates Default Robots.txt and that is more than Enough For Any Website. But If you Wanted to Stop or Block any Urls from Indexing in Search Engines You Need To Change That Default Code using Yoast.We are here to help you, Just Follow our Method to Create a Perfect Robots.txt File.
Steps For How To Create the Best Robots.txt Settings For WordPress?
Step 1: Goto to your WordPress and log in with your Account and Navigate to Plugins Sections.
Step 2: Now Click the Add Plugin Option, and Search for Yoast Seo and Install and Activate it.
Step 3: After Installing The Yoast Seo Plugin, You can See the Yoast SEO Settings On the Left Side Panel Click On It.
Step 4: Under the Tools Section of Yoast SEO, You’ll Find a Number Of Tools. In That Select Tools Settings And Now Click Edit File Editor.
Step 5: Now Just Copy-Paste The Codes Which we Mentioned below or Use Default Yoast Robots.txt
Here is Our Search Engine Optimised Robots.txt Code that We used given Below. This is the Default Best Robots.txt Code For every Website.
@@ Default Best Settings @@
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://www.website.com/sitemap.xml
@@ CODE 2 @@
User-agent: *
Disallow: /wp-content/plugins/
Disallow: /wp-admin/
User-agent: Mediapartners-Google*
Allow: /
User-agent: Googlebot-Image
Allow: /wp-content/uploads/
User-agent: Adsbot-Google
Allow: /
User-agent: Googlebot-Mobile
Allow: /
Sitemap: https://www.website.com/sitemap.xml
*Important Note: In Sitemap Url, Replace ” website” with your site name.
Step 5: Click Save Changes. That’s All You Are Done Man
How to Check Your Robots.txt File?
You can view your robots.txt file by typing your full URL for the homepage and then just add /robots.txt
Example – https://website.com/robots.txt
Replace ” Website” With Your Site Name
Final Step
How To Inform Google about the Updated The Robots.txt?
You Can Inform Google About The Changes You Made in Robots.txt File because google takes time to update the robots.txt file, So You can Tell Google Quickly To Update it Via Manually Using the Below Given Url and also check it whether the changes are made or not
https://www.google.com/webmasters/tools/robots-testing-tool?
*Important Note: You Must Verify Your Blog in Google Webmasters Tool in order to access the Robots Testing Tool
***
