Robots.txt Simulator
TODO(es): Simplified model: longest matching Allow/Disallow prefix wins within matching User-agent groups. Real crawlers may differ.
TODO(es): Simplified model: longest matching Allow/Disallow prefix wins within matching User-agent groups. Real crawlers may differ.