CloudFlare check sucks

submitted by binaryblob from self.AskSaidIt

sorted by:
top
newinsightfulfun
you are viewing a single comment's thread.

view the rest of the comments →

[–]magnora7 37 insightful - 5 fun37 insightful - 4 fun38 insightful - 5 fun -  (50 children)

Just FYI this setting is automated by cloudflare and means we're under attack big time. I also noticed another big forum I frequent is also in lock-down mode today as well, so it's probably internet-wide. I wonder if it's election bots/shills getting started for this upcoming election.

It is frustrating, but the alternative is that we get DDOS'd so hard the site goes down. So I know it sucks, but just be happy the site is up at all, because most websites in our position wouldn't be online anymore.

[–]passionflounder 14 insightful - 2 fun14 insightful - 1 fun15 insightful - 2 fun -  (0 children)

Thank you for the explanation.

[–]binaryblob[S] 8 insightful - 1 fun8 insightful - 0 fun9 insightful - 1 fun -  (21 children)

Thanks for the explanation. It makes sense that it is automated.

Can you see how big the attack is on your infrastructure (that is, how many requests you would have to handle per second (alternatively, more specific or different metrics like number of bytes per second or number of connections made per second) if it wasn't for CloudFlare)?

I am just wondering how big the hate is against this website.

I knew that the web was broken, but it's disappointing to see it in action.

[–]magnora7 20 insightful - 2 fun20 insightful - 1 fun21 insightful - 2 fun -  (20 children)

Saidit gets about a third of a million page load requests a month, and probably half of those are fraudulent (originating from malicious DDOS attacks). During a heavy attack day we might have 10x the page load requests of normal, during which probably 98% of the page requests are fraudulent. Cloudflare ensures (most of) those fraudulent connection attempts stop at cloudflare and don't hit our main server, which is how we're able to stay online.

We've been DDOS attacked multiple times daily for 5 years now. As a result we can never turn our DDOS protection off. So someone out there is setting aside resources on the regular to do this. For years.

I remember the day saidit opened to the public, before we had a registration captcha set up, someone used a script to register all the active usernames from the subreddits where I showed people saidit, and then they auto-registered thousands of usernames, and stole everyone's usernames who might've migrated. Just to frustrate migration and growth of this site. They did this literally hours after we opened the website, they had an automated script ready to go.

This place has been under nonstop attack since day 1, via every method possible from hardware attacks to social engineering, it has honestly surprised me how bad it's been over the last 5 years, I would've never guessed it was this bad. Whoever these groups are, they really don't want forums like saidit to exist.

[–]Maniak 10 insightful - 1 fun10 insightful - 0 fun11 insightful - 1 fun -  (7 children)

Well there goes the hypothesis that WotB was fine here (for now) because SaidIt was too small for the 'powers-that-be' to attack it.

It seems like the ones who've been going after you for all this time are a different group than those who have been attacking the Reddit version of WotB for pretty much the same timespan, but those were on the content-side, not attacking the infrastructure (since Reddit... well... yeah, that's a much bigger investment to bring that down, outside of starting a blackout which does it for free apparently).

On one hand I'd love for there to be a page showing the amount of attacks every day, just as a reminder of what's being thrown at any remaining places that allow people to express anything, but this would also motivate the 12-year-old-Kevins out there to participate in it even more.

Though we're way past the point of a few kids using their parents' credit cards to pay for a DDOS bot somewhere just because they were told that Mercy is useless and D.va is for virgins. (sorry, using the first game example that popped into my mind, I don't even know if those are still in there)

[–]magnora7 12 insightful - 1 fun12 insightful - 0 fun13 insightful - 1 fun -  (5 children)

Yeah basically every forum is under attack, by lots of different groups, all the time. Especially now with AI commenting bots that can flood forums with nonsense. It's just how the internet is now, and I'm frustrated by it tbh. A far cry from the internet in 2003.

/u/cunninglingus shared this, this is basically the graph you wanted showing the number of attacks: https://radar.cloudflare.com/security-and-attacks

[–]Maniak 5 insightful - 1 fun5 insightful - 0 fun6 insightful - 1 fun -  (4 children)

Mmh looks like despite going through the 403 errors, the current issues still ate up my reply :'(

So let me try to remember...

It started with something like "Damn, hadnt even thought about the AI thing in relation to this", and yeah, AI-based DDOS attacks are something to have fucking nightmares about. They could bring down pretty much anything and everything just by using the currently available tools for DDOSing anything and unleashing their algorithms on it.

The whole "AI" thing is only the mainstream release of what came out of the "machine learning" push that's been going on for more than a decade already. There's no intelligence behind it. They've been grabbing as much data as they could, which isn't unrelated to the whole NSA/Snowden thing, and throwing brute-force processing at it.

"Machine learning" may be only 10-15 years old, but the data processing has been going on since data could be processed. It was just a marketing term, much like AI is for the current iteration. It only means "we've been storing up everything everybody has been doing everywhere, including you, and we can now predict everything you can and would do, in order to get you to do what you should do to serve our interests".

It's not intelligence, but it's based on enough data to get close enough to it, especially when it comes to doing brute things like DDOS attacks or spamming forums with meaningless diatribes.

Them doing it to fill forums isn't new, and was scary enough, but if they're starting to use it for straight up targeted attacks on the infrastructure, and of course they would, then... well... I'd like to say that I have at least a clue as to what to do against it, but given that they already control the entire infrastructure and that precisely 0% of Internet is controlled by individuals...

Well... didn't wake up today thinking I'd end the day thinking even less of where we're heading, and yet...

Ok, first things first, still expecting Cloudflare to fix this shit up by tomorrow or at least start giving regular updates.

The overall shitfest is on a way bigger scale and outside of always trying to keep talking and discussing about everything, everywhere we can, hoping and trying to get enough people aware of what's going on around them, I'm not quite sure what more we can do. And it's already a lot.

PS: entirely unrelated but I really do need to address this at some point: how often is the source on Git rebuilt and redeployed? Because there are a few WotB-based CSS changes that I'd really like to submit in a PR if only to make the textarea properly resizable. Nothing to do with the topic at hand, sorry, just popped in my head because this next round of fighting against the incoming 403s :)

Edit: looks like the original reply being eaten up was me being impatient, stupid and most certainly both. I've been multiplying SaidIt tabs over the day because it seems to help with the checks (click the checkbox on one tab, it unlocks another one without having to wait for the original tab to reload), and I found the original reply in another tab that was still getting 403s. Must have gotten my wires crossed at some point, or a cat came in and messed everything up, I don't know. Anyway, that one is not entirely on today's mess :)

[–]penelopepnortney 3 insightful - 1 fun3 insightful - 0 fun4 insightful - 1 fun -  (3 children)

a cat came in and messed everything up

Blaming the cat always works for me. They're guilty until proven innocent.

[–]Maniak 3 insightful - 1 fun3 insightful - 0 fun4 insightful - 1 fun -  (2 children)

And yet they somehow always get out of everything, entirely unscathed.

You could say that cats are, in this way, akin to democrats. But that would be an intolerable insult to the whole of catkind. How DARE you even think of this @Sensei?

[–]penelopepnortney 3 insightful - 1 fun3 insightful - 0 fun4 insightful - 1 fun -  (1 child)

Don't make me call Luna.

[–]Maniak 2 insightful - 1 fun2 insightful - 0 fun3 insightful - 1 fun -  (0 children)

https://saidit.net/media/P77fAb0bD2NByxlQx9kcNAxK2W93j87TbOZPY2y7tJ4.png

[–]bucetao6969 3 insightful - 1 fun3 insightful - 0 fun4 insightful - 1 fun -  (0 children)

Well there goes the hypothesis that WotB was fine here (for now) because SaidIt was too small for the 'powers-that-be' to attack it.

If by "fine" you mean the site won't get attacked, it won't be fine nowhere.

This attack proven even more my faith on the website, as no matter what happens its infrastructure is still hanging on. Hell, possibly better than twitter!

[–]jet199 5 insightful - 1 fun5 insightful - 0 fun6 insightful - 1 fun -  (0 children)

It's likely literally just power mods from reddit scared of losing their power

[–]CheeseWizard 4 insightful - 1 fun4 insightful - 0 fun5 insightful - 1 fun -  (6 children)

¯_(ツ)_/¯

I tried to think possible of suspects, just for fun.

Hacktivists. Idk. They love to LARP and believe whatever narrative they are told. Conspiracionists love free speech platforms, so anyone that think thought crimes is a harm to society, including anti vaxx crazies as one of the harms, will try to bring Justice TM around. So it's an easy activist target.

Maybe Reddit and other giant Platforms, are attacking competitors, to keep their quality down. Political attacks for campaigns, like you said. For political reasons, you can try to bring it down, lower the quality and speed by forcing defense mechanisms, or use bots and shills for propaganda. Etc.

[–]magnora7 8 insightful - 1 fun8 insightful - 0 fun9 insightful - 1 fun -  (0 children)

Yeah. I mean literally every big government has an online hired presence, at this point. As do most big corporations.

For example, Monsanto (now Bayer) is known to have a hired shill pool that is running forums 24/7. They look for negative mentions of their company and products, and then their shill network works those threads to make Monsanto and their products look better. Including forum-sliding negative comments and posts.

Now multiply that across hundreds of companies and governments, with thousands of incentives. And basically you get a ton of noise, that is trying to drown out real discussion. And now with AI bots, this process is automated.

[–]bucetao6969 4 insightful - 1 fun4 insightful - 0 fun5 insightful - 1 fun -  (4 children)

I don't think we can really know unless they actually announcing they're attacking us.

[–]magnora7 5 insightful - 1 fun5 insightful - 0 fun6 insightful - 1 fun -  (3 children)

Either that or the ISPs trace the malicious packets back to their sources, which they basically won't do

[–]bucetao6969 3 insightful - 1 fun3 insightful - 0 fun4 insightful - 1 fun -  (2 children)

So you're saying there's a chance that we know who the hackers are?

[–]magnora7 4 insightful - 1 fun4 insightful - 0 fun5 insightful - 1 fun -  (1 child)

I'm saying it's potentially knowable, but any way to actually do that is extremely unlikely unfortunately

[–]bucetao6969 2 insightful - 1 fun2 insightful - 0 fun3 insightful - 1 fun -  (0 children)

If it's potentially knowable it's a good idea to try everything in your power to investigate. These guys could kill the website.

[–]bucetao6969 4 insightful - 1 fun4 insightful - 0 fun5 insightful - 1 fun -  (0 children)

This place has been under nonstop attack since day 1, via every method possible from hardware attacks to social engineering, it has honestly surprised me how bad it's been over the last 5 years, I would've never guessed it was this bad. Whoever these groups are, they really don't want forums like saidit to exist.

Man I really wish I had a system security class with you as the teacher :)

[–]brimshae 3 insightful - 1 fun3 insightful - 0 fun4 insightful - 1 fun -  (0 children)

Hardcore redditors are such cuckfaggots....

[–]binaryblob[S] 2 insightful - 1 fun2 insightful - 0 fun3 insightful - 1 fun -  (0 children)

Those numbers sound really low, even with a DDOS going on. From what you are saying, you are implying that even a single consumer Internet connection could completely crush this website if you didn't have Cloudflare. I think I would really like to see data for the metrics I requested.

Are you a professional in this area (so I can know how I should talk to you)?

I think the only sustainable way to make such a site work (without you begging for donations) is if you host it via a darknet, which would likely require a significant time investment, but it can't be crushed. If I were a state actor and wanted to crush a forum today, I would just deploy a LLM and have it spam the site with content that your spam filters won't recognize as spam. As such, the only way to run a forum in the long run is if you build a web of trust (which has been pioneered in darknet context already also). Still, most social media assumes that the number of bots is lower than the number of users. That assumption might not be true in the future, so that would bring us to the point that there would have to be some mechanisms for only humans to obtain tokens to access the site, which would kill its somewhat anonymous nature.

I think you need to make up your mind at the very least as to what your goals are in the long run.

[–]KleinWolf35 2 insightful - 1 fun2 insightful - 0 fun3 insightful - 1 fun -  (0 children)

Maybe local competitors? As opposed to the regular bastards (Russia or China bots).

[–]hfxB0oyA 7 insightful - 1 fun7 insightful - 0 fun8 insightful - 1 fun -  (3 children)

Certain people don't like certain sites that don't toe certain lines.

[–]magnora7 6 insightful - 1 fun6 insightful - 0 fun7 insightful - 1 fun -  (2 children)

It seems to be hitting all forums, and the only ones that can keep their heads above water without cloudflare these days are the forums that have their own massive server farms and nearly unlimited bandwidth like reddit

[–]magnora7 6 insightful - 1 fun6 insightful - 0 fun7 insightful - 1 fun -  (1 child)

In the future, this means that if cloudflare were to suddenly disappear or go out of business one day, then basically every forum site including saidit would be exposed to endless DDOS and go down forever until a new cloudflare alternative arose and people switched over and got everything set up again. And there just isn't a good cloudflare alternative. They basically have a monopoly. It's a pretty weird and precarious situation. Not great for the future of free speech, or the internet in general. But no one has really made a viable cloudflare alternative, because it requires owning tons of bandwidth which costs a lot of money.

[–]iamonlyoneman 1 insightful - 2 fun1 insightful - 1 fun2 insightful - 2 fun -  (0 children)

Setting up an alternative DDOS protection scheme seems like a good business opportunity for a millionaire who wanted to become a thousandaire and be imprisoned for violating obscenity and porn laws fairly quickly, eh

[–]cunninglingus 7 insightful - 1 fun7 insightful - 0 fun8 insightful - 1 fun -  (1 child)

I wanted to disagree, but there was a spike in DDoS attack volume:

https://radar.cloudflare.com/security-and-attacks

[–]magnora7 5 insightful - 1 fun5 insightful - 0 fun6 insightful - 1 fun -  (0 children)

Nice find

[–]bucetao6969 4 insightful - 2 fun4 insightful - 1 fun5 insightful - 2 fun -  (0 children)

We pissed someone off it seems

[–]Vulptex 4 insightful - 1 fun4 insightful - 0 fun5 insightful - 1 fun -  (0 children)

Yes, all other CloudFlare sites always have the same problems at the same times.

[–]Maniak 4 insightful - 1 fun4 insightful - 0 fun5 insightful - 1 fun -  (12 children)

Thanks, was wondering about that, whether it was linked to using a VPN connection or something.

Some kind of DDOS campaign hitting 'dissenting' sites would explain it all, in which case we can only wait until the wave passes over. If it's linked to the election, more than one year out, it's going to be a slow-ass year...

Does Cloudflare share data on those things when they happen so you can have an idea as to the amount of attacking requests they've been filtering?

Even posting comments tends to return a 403 error, and I'm guessing it's Cloudflare blocking the request as well.

[–]magnora7 4 insightful - 1 fun4 insightful - 0 fun5 insightful - 1 fun -  (11 children)

Yeah I had the same issue with posting comments returning a 403. Refreshing the page and commenting quickly seems to fix it. They must've added some sort of timeout. They must be having a particularly bad attack, I've not seen some of these limitations before

[–]Maniak 5 insightful - 1 fun5 insightful - 0 fun6 insightful - 1 fun -  (10 children)

So... patience and waiting for it to pass or for more information to come in.

Hey it's the weekend, lots of ways to spend time without worrying about a verification checkbox popping up every couple of reloads :)

It's already good to know that it's a more-than-sitewide issue.

A new case of 'shit happens', until proven worse.

[–]magnora7 6 insightful - 1 fun6 insightful - 0 fun7 insightful - 1 fun -  (9 children)

Thanks for being patient.

I went looking for more information and was able to find this: https://www.securityweek.com/cloudflare-users-exposed-to-attacks-launched-from-within-cloudflare-researchers/

Looks like maybe the problem is this recent popularization of a new attack method of generating the DDOS attacks from within cloudflare's trusted network. So cloudflare is having to clamp down to counter this new type of attack, hence all the weird new little authentication stages and whatnot.

[–]Maniak 4 insightful - 1 fun4 insightful - 0 fun5 insightful - 1 fun -  (6 children)

A stress-test for their engineering abilities, hitting on a sunday so the least amount of people are around to deal with it. It would fit.

Given the size of Cloudflare and most importantly the size of their customers (SaidIt being ultra-tiny in comparison) I'd expect them to throw the kitchen sink at it after this.

So a shitty sunday for us (and much worse for you I guess :), but maybe a good thing long-term, because starting tomorrow they'll be highly motivated to make sure that this can be dealt with more easily next time.

[–]magnora7 5 insightful - 1 fun5 insightful - 0 fun6 insightful - 1 fun -  (3 children)

Yes, in the long term either cloudflare will improve, or an alternative to cloudflare will arise. But in the short-term there is a cat-and-mouse game going on which is impacting the whole internet.

The DDOSers are too powerful, and nothing is reigning them in. I think attacking public communication lines repeatedly should be a jailable offense. I mean if someone went and cut the internet fiber lines, they'd rightfully go to jail. So why is jamming them with made-up data packets 24/7 so no one else can use them any different?

Anyway, things will probably be back to normal in a day or two, I'd guess. Have a great Sunday!

[–]Maniak 4 insightful - 1 fun4 insightful - 0 fun5 insightful - 1 fun -  (2 children)

Yeah, this ties to the whole "Internet should be a public utility" topic, which is being fought by establishments all over the world for obvious reasons.

They want and need it to be private so that all the usual weak points are there to be exploited. And of course whenever something like this happens, they'll only use it as a way to justify more censorship laws for their own interests rather than making it a national utility so that attacks against its proper functioning would be considered a criminal offense.

The fact that we're all governed by ancient corrupt fucks who don't understand anything about any technology that came up within the past half-century isn't helping.

Of course there are cons to the public utility approach, but as far as the power play is concerned, it's the one approach that leans towards actual people having some say vs only a few CEOs to put pressure on.

Given that the Internet infrastructure is being more and more bought by the usual monopolies (MS/Google/Amazon), Internet is becoming less distributed at its root, so less weak points but way more impactful. And companies like Cloudflare whose entire business is about dealing with those weak points... well, they become a weak point themselves.

So hopefully they get over this latest round of attacks and improve enough so that the next one won't hit too hard, because the day they break is the day they disappear, and I'm not aware of many alternatives with a solid enough backbone to deal with such things, unless, of course, you end up falling back on... MS/Google/Amazon, the very same corporations that are owning more and more of the infrastructure being attacked.

In the meantime, we get incessant checkboxes to click in order to be able to post random comments such as this in one of the few remaining places where people can still at least try to discuss anything.

Oh well, if that's the price of still being able to post freely, I'll just keep clicking.

Sorry for the extended rant, can't keep the WotB out of the user it seems :)

[–]magnora7 5 insightful - 1 fun5 insightful - 0 fun6 insightful - 1 fun -  (1 child)

Given that the Internet infrastructure is being more and more bought by the usual monopolies (MS/Google/Amazon), Internet is becoming less distributed at its root, so less weak points but way more impactful.

Yes I agree in theory. But in practice, the internet backbones are already owned by a small handful of "Tier 1" Internet Network Service Providers: https://en.wikipedia.org/wiki/Tier_1_network#List_of_Tier_1_networks

L3 Technologies being the biggest, and has been for a long time. Every backbone cable is owned by these big companies, and they all have government connections and so on.

And it's known companies like AT&T basically have "black rooms" set up at the internet backbone connection points, where the three letter agencies have their own equipment set up to sniff and intercept traffic, and AT&T isn't allowed inside. Read about this: https://en.wikipedia.org/wiki/Room_641A

Anyway, things are bad, but it's not futile. I wouldn't be running saidit if it was futile.

But in the short term, things will probably continue to consolidate (or remain consolidated) and it makes me honestly question the usefulness of the internet over the next few decades. It may eventually just turn in to TV, where you can't really interact with it in a serious way, it just pushes ideas at you all day. That seems to be the direction it's heading. And the AI bots now flooding every forum with garbage content isn't helping, so even websites that don't want to move in that direction are being forced to by malicious traffic.

But on the main point, I'm sure the current cloudflare thing will be fine over the next few weeks and months. Cloudflare seems competent and motivated. The long-term direction of the internet as a whole is concerning though, to say the least. Thanks for the chat. Have a good one!

[–]Maniak 4 insightful - 1 fun4 insightful - 0 fun5 insightful - 1 fun -  (0 children)

Fair point. A lot is happening under the radar for most people (myself included, I had even forgotten about L3 given how rarely it appears in any news), and it's all about governments and corporations making deals between themselves as to the control over what everybody else on the planet is now using as their most basic communication system (outside of actual speech). Control that and you control everything.

If there's one positive side to the whole profit motive thing, it's that those corporations are not tied to any specific government. They're buying them out, sure, but ultimately they're solely interested in their own profits, so there's still some wiggle-room for regular people to lean on that. For now.

I'm right there with you as to what Internet may very well become in the nearer future than we'd think. Tor may end up having to pick up the slack, but there's work to do on the engagement aspect, and then we get right back to the infrastructure issue because everything is going through that. You can add has many encryption layers as you want, you're still going through fiber connections that are owned by a couple of corporations who don't care about your freedom of expression.

There's constant nudging going on, it's always going in the same direction, and attacks such as today's on Cloudflare are an integral part of it.

Oh well, we'll keep on doing our thing whenever and wherever we're able to, and when we can't anymore, we'll find other ways to keep doing it anyway. As of right now, count this as one grateful for what you've done here.

Good weekend-end, and hopefully tomorrow (or soon after) Cloudflare will have taken control back, until next time :)

[–]Dragonerne 1 insightful - 1 fun1 insightful - 0 fun2 insightful - 1 fun -  (1 child)

hitting on a sunday

That's how you know it's not Christians behind it. I wonder who

[–]In-the-clouds 2 insightful - 1 fun2 insightful - 0 fun3 insightful - 1 fun -  (0 children)

I'm a Christian and I keep the Sabbath on Saturday, but it doesn't really matter, because a real Christian should not be trying to stop free speech on any day of the week.

[–]iamonlyoneman 2 insightful - 2 fun2 insightful - 1 fun3 insightful - 2 fun -  (1 child)

LOL the phone calls are coming from inside the house!

[–]magnora7 1 insightful - 2 fun1 insightful - 1 fun2 insightful - 2 fun -  (0 children)

Haha you nailed it

[–]Mcheetah 4 insightful - 1 fun4 insightful - 0 fun5 insightful - 1 fun -  (2 children)

What can we do to combat this?

[–]magnora7 5 insightful - 1 fun5 insightful - 0 fun6 insightful - 1 fun -  (0 children)

That's a great question, and I wish I had a great answer. Unfortunately as a user I don't think there's much you can do other than to be patient with the cloudflare stuff, and keep an eye out for bad actors. And help others with these things.

The problem is with the fraudulent traffic, and I don't know how an average user could help that. Push for legislation to prosecute DDOSers? Run a new forum website using the saidit open source code? That's all I can think of. If anyone has other ideas, I'm all ears

[–]Maniak 4 insightful - 1 fun4 insightful - 0 fun5 insightful - 1 fun -  (0 children)

I'm guessing count on Cloudflare to do its job and don't shit on SaidIt for the (ultra) annoying constant connection checks...

Not piling on to this thing that the site has no control over is probably the best, and most, that we can do while waiting for the attack to subside.

Edit: or maybe donate a couple millions bucks to SaidIt so there are many more servers around to take the hit :)

[–]Site_rly_sux_rly_sux 3 insightful - 2 fun3 insightful - 1 fun4 insightful - 2 fun -  (0 children)

I guess it's better than getting a 504 error all day long. It's annoying but necessary against the shill armies. With upcoming elections in Canada and the USA like you mentioned, it'll get more prevalent. Conservatives/Republicans will get the vote regardless unless Dominion has a say in it. Edit: Had to refresh the page to post my comment(error 403).

[–]Clown_Chan 2 insightful - 1 fun2 insightful - 0 fun3 insightful - 1 fun -  (0 children)

Shit, just like Kiwifarms

I bet it's fucking butthurt troons doing it.

[–]SoCo 2 insightful - 1 fun2 insightful - 0 fun3 insightful - 1 fun -  (0 children)

Cloudflare is having a major outage this morning. It is not just this site.