Keep in mind that these settings can be read and followed only if crawlers are allowed to access the pages that include these settings.