# See http://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file User-agent: * Disallow: /media_proxy/ Disallow: /interact/ User-agent: BoardReader User-agent: BoardTracker User-agent: Gigabot User-agent: Twiceler User-agent: dotbot User-agent: duckduckgo User-agent: google User-agent: Baidu User-agent: Baiduspider User-agent: Baiduspider-video User-agent: Baiduspider-image User-agent: NaverBot User-agent: Sosospider User-agent: Yandex User-agent: YoudaoBot User-agent: Yeti User-agent: proximic User-agent: sogou User-agent: sogou spider User-agent: Sogou web spider User-agent: archive Disallow: / User-agent: Mediapartners-Google* Disallow: