Skip to content
Logo LottoSport
Menu
  • Utilities
  • Animal
  • Life
  • Product
  • Sleeping
  • Technology
Menu

The Proper Way To Use The robots.txt File Update

Posted on 08/28/2022 by

The Proper Way To Use The robots.txt File Update

In my last article about the robots.txt file I had spelled it wrong. It should have been robots.txt instead of robot.txt. The article should read like this:
When optimizing your web site most webmasters don’t consider using the robots.txt file.This is a very important file for your site. It let the spiders and crawlers know what they can and can not index. This is helpful in keeping them out of folders that you do not want index like the admin or stats folder.
Here is a list of variables that you can include in a robot.txt file and there meaning:
1) User-agent: In this field you can specify a specific robot to describe access policy
for or a “*” for all robots more explained in example.
2) Disallow: In the field you specify the files and folders not to include in the crawl.
3) The # is to represent comments
Here are some examples of a robots.txt file
User-agent: *
Disallow:
The above would let all spiders index all content.
Here another example
User-agent: *
Disallow: /cgi-bin/
The above would block all spiders from indexing the cgi-bin directory.
User-agent: googlebot
Disallow:
User-agent: *
Disallow: /admin.php
Disallow: /cgi-bin/
Disallow: /admin/
Disallow: /stats/
In the above example googlebot can index everything while all other spiders can not index admin.php, cgi-bin, admin, and stats directory. Notice that you can block single files like admin.php.

Recent Posts

  • Ruay Lotto Online
  • The Healing Properties Of Aromatherapy Pillows (2)
  • How To Be Different From Other Affiliates!
  • History Of Chinese Astrology
  • Flannel Sheet Set

Recent Comments

No comments to show.

Archives

  • February 2023
  • January 2023
  • December 2022
  • November 2022
  • October 2022
  • September 2022
  • August 2022
  • July 2022
  • June 2022
  • May 2022
  • April 2022
  • March 2022
  • February 2022
  • January 2022
  • December 2021

Categories

  • Animal
  • Life
  • Product
  • Sleeping
  • Technology
  • Utilities

เว็บไซต์พันธมิตร

  • Huaygeneral
  • Ruay80.com
  • Ballhuay.site
  • Sportthailotto
  • Lotteryfifa
  • Verylottoruay
  • Ruayvariety
  • Lottovipmono
  • Lottoslot.info
  • Bloghuay.mobi
© 2023 All interesting things are waiting for you to discover | Powered by Lottosport.Mobi