What is the standard robots.txt file?
A robots.txt file is a file that can control which files on your website so-called crawlers are allowed to access. The robots.txt file is located in the Document Root of the website, that is, in the directory where all files of a website are located. In the example of the website www.meine-domain.ch, the robots.txt file is in the following path: www.meine-domain.ch/robots.txt. In your web server, this would be the path: /var/www/vhosts/meine-domain.ch/httpdocs/robots.txt
The robots.txt file is a plain text file without formatting that complies with the "Robots Exclusion Standard" (see https://de.wikipedia.org/wiki/Robots_Exclusion_Standard). In this file, rules are defined that allow or block a specific crawler from accessing the Document Root of the domain or subdomain. If nothing is specified in the robots.txt file, all files can be crawled.
What is delivered with this file?
If you have not defined your own version of the robots.txt file for your website, the default version from Emubox will be delivered. This defines a crawl delay (the interval between requests in seconds) so that bots cannot send their requests directly in succession but must wait between requests. Additionally, the Emubox default allows access to all files.
(Please note that the crawl delay is not a standard and may not be observed by all bots.)
Another example of a created robots.txt file:
User-agent: *
Disallow: /cgi-bin/
Disallow: /errdocs/
Disallow: /js/
Disallow: /jss/
Disallow: /img/
Disallow: /images/
Disallow: /webalizer/
Disallow: /*.gif$
Disallow: /*.jpg$
Disallow: /*.doc$
Disallow: /*.xls$
Disallow: /*.pdf$
Disallow: /*.txt$
Disallow: /*.zip$
Disallow: /Impressum.*
Disallow: /Widerrufsrecht.*
Disallow: /Datenschutz.*
Disallow: /AGB.*
Disallow: /BGBW.*
Disallow: /Rechtsinformation.*
How can I override the standard version of the robots.txt file?
If you want to create your own robots.txt file, you can create it in the directory of your website and set your own rules as you wish.
Further helpful information about the robots.txt file can be found at the following link: https://moz.com/learn/seo/robotstxt.