December 21, 2005
based on probable cause. ”
This is rather startling, because in a discussion of how robots.txt files work to prevent files from being indexed, someone told me to look at the White House robots file. Pretty darn interesting!!
They basically listed all the directories that had interesting stuff, in order to keep them from being indexed. D’oh!
In case you want to look:
and I’m not making that live link on purpose.
Some of them are perplexing:
- Disalow /911/patriotism/
and some are just stupid:
- Disallow /president/tee-ball/