How to create or customize the robots.txt?

configuration
robots.txt
configuration-general
configuration-general-design

(Dmitry Fedyuk) #1

«Stores» → «Configuration» → «General» → «Design»→ «Search Engine Robots»:

After saving the configuration the robots.txt file will be generated in the Magento 2 root folder.


The robots.txt file should be located in the Magento 2 root folder
(Muneeb Syed) #2

All great just would like to do one more addition in the post, Custom Instructions like:

Disallow: /lib/
Disallow: /*.php$
Disallow: /pkginfo/
Disallow: /report/
Disallow: /var/
Disallow: /catalog/
Disallow: /customer/
Disallow: /sendfriend/
Disallow: /review/
Disallow: /*SID=