New York Times tried to block a web crawler that was affiliated with the famous Internet Archive, a project whose easy-to-use comparisons of article versions has sometimes led to embarrassment for the newspaper.

submitted by [deleted] from theintercept.com

sorted by:
new (suggested)
topinsightfulfun
you are viewing a single comment's thread.

view the rest of the comments →

[–]Site_rly_sux 1 insightful - 1 fun1 insightful - 0 fun2 insightful - 1 fun -  (6 children)

Go ahead.

Explain it.

It's a giant conspiracy theory and the proof is: six ways to lose your student loan and the "changing tone" on Bernie

Fuck sakes man. This is peurile.

Explain it, go on

[–]brimshae 4 insightful - 3 fun4 insightful - 2 fun5 insightful - 3 fun -  (5 children)

Explain it

The alternative is trusting the New York Times to be honest.

I rest my case.

[–]Site_rly_sux 1 insightful - 1 fun1 insightful - 0 fun2 insightful - 1 fun -  (4 children)

Trusting them with what exactly?

The only statement they've made, that we've seen, is the robots.txt link above. That's a pretty clear statement. Would you like to flesh out how exactly they might be lying within their robots.txt

Are you saying that somehow OP and the Intercept are more trustworthy? Than a technical document ? What

[–]brimshae 1 insightful - 1 fun1 insightful - 0 fun2 insightful - 1 fun -  (3 children)

Trusting them with what exactly?

"trusting the New York Times to be honest."

[–]Site_rly_sux 1 insightful - 1 fun1 insightful - 0 fun2 insightful - 1 fun -  (2 children)

They literally haven't made any statement on this topic that we have seen other than robots.txt

So you sound like a moron.

Do you believe that their robots.txt is probably a true statement? Are you going to tell me that you have low trust in a machine instruction. You sound stupid

[–]brimshae 1 insightful - 1 fun1 insightful - 0 fun2 insightful - 1 fun -  (1 child)

Someone is angry because he can't understand basic words. It's probably the guy screaming "moron", "stupid", and the like who is still trying to figure out a basic dig at the NYT and their general lack of credibility.

[–]Site_rly_sux 1 insightful - 1 fun1 insightful - 0 fun2 insightful - 1 fun -  (0 children)

There's no matter here on which their credibility is up for debate. They said that certain user agents may not crawl certain areas of their site.

You replied that you didn't believe them.

It IS moronic. Robots dot txt is a machine instruction. There is no sense in which credibility enters the argument. To claim that you don't believe their robots dot txt makes you sound like a retard.