On March 1st the U.S. Controlled Wayback Machine Started Blocking Access to Russian News Agency Tass Articles.

submitted by [deleted] from imgur.com

sorted by:
top
newinsightfulfun
you are viewing a single comment's thread.

view the rest of the comments →

[–]cat256 1 insightful - 1 fun1 insightful - 0 fun2 insightful - 1 fun -  (2 children)

yeah, that's kinda interesting. i need to code tomorrow, side hustle today... i'd ask around a bit.

edit: so a simple google search would show the result: anti crawling sites like quora and people in the open source community are working on this. see: https://github.com/internetarchive/wayback/issues/228

[–]LarrySwinger2 1 insightful - 1 fun1 insightful - 0 fun2 insightful - 1 fun -  (1 child)

So they want to pressure sites to allow crawling and archiving? I'm not sure if that's the way to go. There should be a decentralized archive, so that there are no legal consequences for anyone. That way, if someone simply decides to crawl them anyway, there's nothing they can do, right? I don't know if .htaccess has to be respected. I don't see why it would.

[–]cat256 2 insightful - 1 fun2 insightful - 0 fun3 insightful - 1 fun -  (0 children)

there are other measures too. or just build your own crawler bot and spread the word on sites like hackernews and other hacktivist sites. i'd do it if i got time.