# small robots.txt # more information about this file can be found at # http://www.robotstxt.org/wc/robots.html # lines beginning with the pound ("#") sign are comments and can be deleted. # to stop a polite robot indexing an exampledir # add a line like (delte the #'s) # user-agent: polite-bot # Disallow: /exampledir/ # a list of known bots can be found at # http://www.robotstxt.org/wc/active/html/index.html # see http://www.sxw.org.uk/computing/robots/check.html # for syntax checking User-agent: * Crawl-Delay: 5 Disallow: /blog/ Disallow: /clients/ Disallow: /flas/ Disallow: /image kit/ Disallow: /kunder/ Disallow: /private/ Disallow: /psds/ Disallow: /temp/ Disallow: /text/ Disallow: /video/ Disallow: /xml/ Disallow: /*/feed/ Disallow: /*/feed/rss/ Disallow: /*/trackback