Main » WWW » 2007 » Proceedings of the 16th International Conference on World Wide Web, WWW 2007, Banff, Alberta, Canada, May 8-12, 2007 »

A large-scale study of robots.txt

Yang Sun, Ziming Zhuang, C. Lee Giles