robot.txt updated
This commit is contained in:
@@ -1,3 +1,3 @@
|
|||||||
# allow crawling everything by default
|
# allow crawling everything by default
|
||||||
User-agent: *
|
User-agent: *
|
||||||
Disallow:
|
Disallow: /cdn-cgi/
|
||||||
Reference in New Issue
Block a user