What is a Robots.txt File?

Contents hide

1 What is a Robots.txt File?

1.1 Why You Need a Robots.txt File

1.1.1 This is How You Make a Robots.txt File

1.1.1.1 That is how you make a robots.txt file

It’s a small text file that tells Google what it should or should not put in its search results.

Why You Need a Robots.txt File

The search engines like it when you have a robots.txt file. It tells the search engines what you want to be indexed (listed) in the search results.

This is How You Make a Robots.txt File

1. Open notepad
2. Copy and paste the bolded text below into the empty notepad file

User-agent: *Allow: /
Disallow: thankyou.html

3. Save it as robots.txt
4. Upload it to the root directory of your hosting account
5. You are done!

For everything you don’t want someone to or the search engines looking up (i.e. = not relevant to your website), type new Disallow line for each thing, like this…

User-agent: *Allow: /
Disallow: thankyou.html
Disallow: /myporncollection
Disallow: /this-old-web-page-i-dont-want-anyone-seeing.html
Disallow: /thispornvideo.mpeg

That is how you make a robots.txt file

If you are using WordPress, I would suggest downloading this XML Generator WordPress Plugin.

Thomas J. Kane

At 16 years old in 1997 I built my first client website for a car dealership. At 17 I was one of the first few people in Kansas to be Cisco Networking Certified and I helped co-build the first computer network for a high school, two elementary schools, a hospital, and a 911 station. By the age of 40 now I have over 25,000 hours in content writing mastery and Google Advertising/SEO and have worked with over 480 different industries, amazing mentors, SEO & PPC corporate team leader at Godaddy.com, trained other SEO company's staff, managed over $37M dollars in advertising spend since 2007 for multiple companies and spent over $1M in A/B testing, created landing pages, sales funnels and massive PPC campaigns for the USA's #1 automotive leads provider, helped expand a global e-commerce business from 5 English sites to 122 sites in 30 countries in 7 languages, owned two #1 top-ranked SEO companies, in both Arizona & Kansas. Won all-expense-paid trip awards from Google for PPC performance, and been to the headquarters in Boston and California, and worked with 3 dedicated senior Google PPC teams at 3 major companies. I run an online advertising service for Montana, Kansas, Arizona & Florida businesses, as well as manage large campaigns for online national stores and services. Into renewable energy, solar, wind energy, crypto currency, Tesla, Nikola, natural health and healing, traveling, psychology, human behavior analysis, lie detection, forward progress, inspirational and educational writing, research, and analysis. Fascinated with AI (https://thomasj.ai/) and how it relates to the future of search and information.