Saidit just upgraded our DDOS protection system

submitted by magnora7 from self.SaidIt

Last night I spent 6 hours building and testing a new DDOS protection system on top of CloudFlare.

Basically now anyone who tries to connect too many times in a short time-span, if done enough times, will automatically be added to our firewall to be blocked. Previously this was done manually, or required payment to cloudflare, which left us a bit vulnerable during a DDOS where IP addresses are numerous and changing rapidly. But now it's automatic and fast, and run by our own free scripts, so this should make the fortress of saidit stronger than ever, without adding a penny to our expenses.

Thanks to u/Marou for the lead on how to get started with this project, and to u/d3rr for the help along the way.

As a side effect, I noticed lots of little connections that have been slowly eating up bandwidth in a repeated and regular way are now gone, and the site itself seems to run much more quickly!

Anyway, just wanted to let everyone know that after this upgrade our defenses are better, and our site is running faster than ever!

all 38 comments
sorted by:
top
newinsightfulfun

[–]Tarrock 7 insightful - 1 fun7 insightful - 0 fun8 insightful - 1 fun -  (6 children)

Is it a temp block or perm block? I got a feeling that someone could accidently trigger this.

[–]magnora7[S] 8 insightful - 2 fun8 insightful - 1 fun9 insightful - 2 fun -  (5 children)

It's set up to be very difficult for a human to accidentally trigger, and even if someone somehow does so, the ban should not last long.

Someone who deliberately goes nuts on it will be banned for much much longer than someone who accidentally oversteps it just slightly (which is still very very difficult to do by accident).

[–][deleted] 5 insightful - 2 fun5 insightful - 1 fun6 insightful - 2 fun -  (1 child)

Nice, glad you were able to get something set up.

[–]magnora7[S] 6 insightful - 2 fun6 insightful - 1 fun7 insightful - 2 fun -  (0 children)

Yeah thanks a lot for your help! Your link and information ended up being exactly what we needed. I pulled an all-nighter and got it working wonderfully from the info you sent.

[–]Drewski 2 insightful - 1 fun2 insightful - 0 fun3 insightful - 1 fun -  (1 child)

Yesterday I was unable to access the site at all on my home IP. Today it seems fine though.

[–]magnora7[S] 1 insightful - 1 fun1 insightful - 0 fun2 insightful - 1 fun -  (0 children)

Ok thanks. We’re still adjusting our new firewall. If you have more problems let me know

[–]realister 5 insightful - 2 fun5 insightful - 1 fun6 insightful - 2 fun -  (0 children)

Please consider donating to their patron

https://saidit.net/s/Ice_Poseidon2/comments/2wtb/since_saidit_has_been_good_to_us_all_and_provided/

[–]theoracle 4 insightful - 2 fun4 insightful - 1 fun5 insightful - 2 fun -  (2 children)

It has been a rough day for DOSing.

Best protection at the site I think is to force a gateway page with an extremely simple captcha. Like you have on the signup, but maybe simpler. What ever will stop bots but not form an attack vector itself. Use it to deny access to the rest of the site until they pass the captcha. You can limit attempts and also only have the page enable once a certain bandwidth or connection limit is passed, so people usually don't see it.

[–]magnora7[S] 6 insightful - 2 fun6 insightful - 1 fun7 insightful - 2 fun -  (1 child)

Good ideas, but cloudflare actually already does most of those exact things for us.

Even during the DDOS I was having to click pictures of umbrellas and airplanes to get past the Cloudflare wall because I had connected so many times, haha

[–]theoracle 4 insightful - 1 fun4 insightful - 0 fun5 insightful - 1 fun -  (0 children)

Yes you are right they do do that! It didn't come to my mind though when I was suggesting it. I guess if you don't use Cloudflare it's an option.

[–]theoracle 4 insightful - 2 fun4 insightful - 1 fun5 insightful - 2 fun -  (1 child)

I was just thinking what about users of vpns or proxies where a few ips can generate a lot of traffic? Maybe this is not a problem for now but the future. Hopefully if you become big enough that this is an issue you will address it then.

[–]magnora7[S] 3 insightful - 2 fun3 insightful - 1 fun4 insightful - 2 fun -  (0 children)

Yeah we're just having to block certain VPN services because they're avenues for attacks, and I'm not sure if there's a way around that. But I think our new system might give us some opportunities to make it more dynamic.

[–]justagent 4 insightful - 2 fun4 insightful - 1 fun5 insightful - 2 fun -  (0 children)

Nice work!

People who DDOS are assholes.

[–]Antifa 3 insightful - 4 fun3 insightful - 3 fun4 insightful - 4 fun -  (0 children)

Outstanding. Now we can watch Reddit die safely.

[–]Zednix 4 insightful - 2 fun4 insightful - 1 fun5 insightful - 2 fun -  (0 children)

Cool. I hope saidit can become more active because of smoother operations

[–][deleted] 4 insightful - 2 fun4 insightful - 1 fun5 insightful - 2 fun -  (0 children)

That's good to hear, appreciate the work you're putting into it, hopefully now those sorts of problems are stifled by the new upgrade.

[–]Riva 3 insightful - 2 fun3 insightful - 1 fun4 insightful - 2 fun -  (0 children)

This is great. Thank you!

[–]LarrySwinger2 3 insightful - 2 fun3 insightful - 1 fun4 insightful - 2 fun -  (11 children)

That's great. I am noticing, however, that the connection gets blocked when I log in via tor (403 or 1020). But, based on your explanation, perhaps I shouldn't click too fast. I hope the compromise in terms of usability won't be too big. I'll report back on this.

[–][deleted] 4 insightful - 2 fun4 insightful - 1 fun5 insightful - 2 fun -  (2 children)

I'll try to loosen the Tor restrictions a bit. We're already looser than default CloudFlare. Some Tor ips are marked as dirty because abuse comes from them.

[–]skiseme 2 insightful - 1 fun2 insightful - 0 fun3 insightful - 1 fun -  (1 child)

What about having an onion address alongside a clearnet address?

[–][deleted] 2 insightful - 1 fun2 insightful - 0 fun3 insightful - 1 fun -  (0 children)

I'm into that. it's a bit of a tech burden, because you have to serve the site as http and not https.

[–]magnora7[S] 3 insightful - 2 fun3 insightful - 1 fun4 insightful - 2 fun -  (6 children)

Yeah it's best not to use a VPN, as we've had to block a lot of VPN traffic because that's where a lot of DDOS attacks originate from. I'm still tweaking the new firewall rules to balance everything, I think it's almost all worked out.

[–][deleted]  (1 child)

[deleted]

    [–]comments 3 insightful - 2 fun3 insightful - 1 fun4 insightful - 2 fun -  (0 children)

    https://notabug.io/t/saidit.all when you're blocked

    [–]theoracle 3 insightful - 2 fun3 insightful - 1 fun4 insightful - 2 fun -  (1 child)

    Yeah we're just having to block certain VPN services because they're avenues for attacks, and I'm not sure if there's a way around that. But I think our new system might give us some opportunities to make it more dynamic.

    It's concerning that you would block VPNs at all, but then maybe there is some particularly bad ones? Then what about the likes of tor?

    Personally I don't like blocking, banning or deleting. It is like the internet's version of segregation, apartheid and genocide....

    Obviously if it's your only solution so be it, I just hope it's not your "Final Solution", because I think better can be done, much better.

    [–]magnora7[S] 4 insightful - 2 fun4 insightful - 1 fun5 insightful - 2 fun -  (0 children)

    I understand your concern, but we're not banning entire VPNs, just certain IP addresses that happen to be on VPNs. We only block those addresses because if we don't, one person can make 100,000 requests per second and take the website down, and those particular IP addresses have displayed that type of behavior already according to connection logs.

    [–]comments 3 insightful - 1 fun3 insightful - 0 fun4 insightful - 1 fun -  (1 child)

    yeah I guess this is what happens when you have to share IPs with people who don't use them respectfully.

    [–]magnora7[S] 2 insightful - 2 fun2 insightful - 1 fun3 insightful - 2 fun -  (0 children)

    Yeah exactly

    [–]theoracle 3 insightful - 1 fun3 insightful - 0 fun4 insightful - 1 fun -  (0 children)

    I just tested tor and it is very slow but works.

    Not good if it is blocked... :-(

    [–]kokolokoNightcrawler 3 insightful - 2 fun3 insightful - 1 fun4 insightful - 2 fun -  (1 child)

    Don't open Saidit from more than 2-3 tabs at the same time..I got IP banned this morning and had to reset my IP

    [–]JasonCarswell 1 insightful - 2 fun1 insightful - 1 fun2 insightful - 2 fun -  (0 children)

    That's terrible. That's how I start my day, open each item in a new tab - never an entire page enough. I'm not sure if I can reset my IP. I would hope they might consider a white list to sign up for or something. Maybe set a more human limit like a dozen or two before blocking, and/or block for only 6 or 12 hours just in case, and/or keep intensifying the blocking with regularity because humans will learn the limits and generally abide within them.

    EDIT: I read later down that M7 anticipated this.

    [–]Wahwah 2 insightful - 3 fun2 insightful - 2 fun3 insightful - 3 fun -  (0 children)

    🙏

    [–]adultmanhwa 2 insightful - 2 fun2 insightful - 1 fun3 insightful - 2 fun -  (0 children)

    👍👍👍 gokuro sama deshita 👍👍👍

    [–]teelo 1 insightful - 3 fun1 insightful - 2 fun2 insightful - 3 fun -  (0 children)

    Yeah, wouldn't want to get any of those Damned Disk Operating Systems.

    [–]m68k 2 insightful - 1 fun2 insightful - 0 fun3 insightful - 1 fun -  (0 children)

    Nice, I appreciate the hard work being put into the site. :)

    [–]comments 1 insightful - 1 fun1 insightful - 0 fun2 insightful - 1 fun -  (3 children)

    having a harder and harder time accessing saidit. glad the site's up but these changes may mean some people have difficulty accessing the site.

    [–]magnora7[S] 2 insightful - 1 fun2 insightful - 0 fun3 insightful - 1 fun -  (2 children)

    It's only temporary until they stop actively DDOSing us. Usually the worst of it just lasts a couple hours. Normally we'd be completely down right now.

    [–]comments 2 insightful - 1 fun2 insightful - 0 fun3 insightful - 1 fun -  (1 child)

    wow. this has been going on a few days now hasn't it?

    [–]magnora7[S] 2 insightful - 1 fun2 insightful - 0 fun3 insightful - 1 fun -  (0 children)

    Yeah we didn't have a DDOS for months, then we had one the other day, and then the one tonight. Oh well, they'll probably give up soon since they're not really getting any traction this time with our new anti-DDOS software I made.