Robots.txt Simulator

TODO(es): Simplified model: longest matching Allow/Disallow prefix wins within matching User-agent groups. Real crawlers may differ.