You Don’t Need Robots.txt On Root Domain, Says Google


Google’s Gary Illyes shares an unconventional but valid method for centralizing robots.txt rules on CDNs.

  • Robots.txt files can be centralized on CDNs, not just root domains.
  • Websites can redirect robots.txt from main domain to CDN.
  • This unorthodox approach complies with updated standards.


Leave a Comment

document.write(`
window.googletag = window.googletag || {cmd: []}; googletag.cmd.push(function() { googletag.defineSlot('/21984817904/www.rankwebtools.com_Billboard', [970, 250], 'gpt-passback2').addService(googletag.pubads()); googletag.enableServices(); googletag.display('gpt-passback2'); });
`)