I have already shared my opinion on the variety of various Robots.txt checkers and verifiers : be aware of them but use those with caution . They are full of errors and misinterpretations. However I happened to come across one really good one the other day: robots.txt checker . I found it useful to use for self-education in the first place. Checks are done considering the original 1994 document A Standard for Robot Exclusion ,