Path: | lib/spider/robot_rules.rb |
Last Update: | Sat Feb 23 07:20:50 +0000 2019 |
robot_rules.rb
Created by James Edward Gray II on 2006-01-31. Copyright 2006 Gray Productions. All rights reserved. https://github.com/eribertomota/robot_rules.rb https://github.com/johnnagro/spider/issues/1