Archive.org is getting shitter, then
Posted: Sat Jul 30, 2005 12:01 pm
Just tried to look up http://www.mdqnet.net, which maybe a very small number of you would remember was a major Q1 mod development centre back in the day.
Up until mebbe 6 months ago, archive.org would list that site no problem, I could scroll back and see it in (a few of) its original states.
Now, archive.org reports that site is blocked by robots.txt. Problem is, the domain expired about 3 years ago and it's just a link farm domain now, owned by someone completely different.
So archive.org looks like it'll be come practically useless from now on then. I can't think of many sites which, when they expire, don't fall into the hands of link farmers... and if they're blocking with robots.txt, and archive.org is making that apply to the whole history of the site... well... shit.
Up until mebbe 6 months ago, archive.org would list that site no problem, I could scroll back and see it in (a few of) its original states.
Now, archive.org reports that site is blocked by robots.txt. Problem is, the domain expired about 3 years ago and it's just a link farm domain now, owned by someone completely different.
So archive.org looks like it'll be come practically useless from now on then. I can't think of many sites which, when they expire, don't fall into the hands of link farmers... and if they're blocking with robots.txt, and archive.org is making that apply to the whole history of the site... well... shit.