The Robots Exclusion Protocol is a standardized communication protocol that enables website owners to instruct web crawlers and search engine bots which areas of a site should not be accessed or indexed, thereby regulating automated retrieval of web content.
8
Publications
260
Citations
17
Authors
13
Institutions
Publications per year
2017–2026
0
Variants
RepRobots.txt Protocol
Parents
Publications
8
Bo Cheng, Guobin Shen, Jie Liu et al. · 2014 · 49 citations
Determining Bias to Search Engines from Robots.txt
Yang Sun, Ziming Zhuang, C. Lee Giles · 2007 · 44 citations
A large-scale study of robots.txt
Yang Sun, Ziming Zhuang, C. Lee Giles · 2007 · 43 citations
The Ethicality of Web Crawlers
Yang Sun, C. Lee Giles · 2010 · 39 citations
1–5 of 8
Authors
17
- YS
4
h-index
- CL
4
h-index
- ZZ
2
h-index
- ABArnab Bhattacharjee
Bangladesh University of Engineering and Technology, Bangladesh
1 publications · 17 citations
1
h-index
- BC
1
h-index
1–5 of 17
Institutions
13
Pennsylvania State University
State College, United States
4 h-index
10 publications
336 citations
Yahoo (United States)
Sunnyvale, United States
1 h-index
4 publications
68 citations
Mountain View College
Dallas, United States
1 h-index
1 publications
39 citations
Illinois Institute of Technology
Chicago, United States
1 h-index
2 publications
98 citations
Google (United States)
Mountain View, United States
1 h-index
1 publications
39 citations
University of North Carolina at Charlotte
Charlotte, United States
1 h-index
1 publications
49 citations
Yahoo (United Kingdom)
London, United Kingdom
1 h-index
4 publications
68 citations
University College London
London, United Kingdom
1 h-index
3 publications
78 citations
1–8 of 13
Venues
- Journal of Information Science
JIS · 1 publications · 26 citations
1
h-index
1–1