you are viewing a single comment's thread.

view the rest of the comments →

[–]magnora7 2 insightful - 1 fun2 insightful - 0 fun3 insightful - 1 fun -  (9 children)

This guy lost it and deleted every thread in this entire sub to censor everyone because I made him follow the mod rules, and then changed the custom CSS to hide the whole page. So he's lost his mod privileges forever on saidit because it's obvious he's here to control people instead of spread info, given how many mod rules he's broken and how much he's censored. I added an admin note to his account. Just wanted to let you know

[–][deleted] 2 insightful - 1 fun2 insightful - 0 fun3 insightful - 1 fun -  (8 children)

Thanks. Some of these extremists have such thin skins, I'm kind of shocked. If they won't get down in the mud and debate, they'll never win anyone over.

[–]magnora7 2 insightful - 1 fun2 insightful - 0 fun3 insightful - 1 fun -  (0 children)

No prob. Yes, I am also surprised how often these extremists have such thin skin too. You'd think they'd be debate-hardened veterans!

I think they don't argue because deep down they know they're being overly-extreme, so they know they won't win debates if it comes down to it, so they have to resort to cheap tactics to push narratives. Because those narratives don't hold up to the scrutiny of reality. Thank goodness for our mod rules. Mod fiefdoms and pushing narratives seem to go hand-in-hand much more often than not.

[–]magnora7 2 insightful - 1 fun2 insightful - 0 fun3 insightful - 1 fun -  (6 children)

Hey our Alexa rank broke above 60k, congrats to us! We must be doing something right :)

[–][deleted] 2 insightful - 1 fun2 insightful - 0 fun3 insightful - 1 fun -  (5 children)

Heck yeah! Is this our all time best rank? I forgot where we peaked last time. This US rank of 15,700 is pretty amazing too.

[–]magnora7 3 insightful - 1 fun3 insightful - 0 fun4 insightful - 1 fun -  (4 children)

I think it is our best ever, I think our last peak a few months ago was 62k. And we're still going up! We're going to pass notabug soon. Our time-on-site per user is excellent, as is our bounce rates and our average pages per visitor. Things look really good. We're a reddit exodus or two away from being on the same tier as voat and steemit, which is very exciting.

If we were to suddenly get say 5x the number of users, what would our hardware bottleneck be, do you think? The CPU?

[–][deleted] 2 insightful - 2 fun2 insightful - 1 fun3 insightful - 2 fun -  (3 children)

Shit I'm scared and making a backup now... hahaaaa.

If we were to suddenly get say 5x the number of users, what would our hardware bottleneck be, do you think? The CPU?

No clue. If I had to guess I'd say yeah probably CPU limited since we have 8GB of RAM and aren't hosting chat on the same server anymore. But CPU limited at the nginx level, like there's not enough CPU to handle all of the incoming requests and requests have to wait. (we can technically only serve 4 requests at once right now). I think a natural progression of scaling is for us to host all of the thumbnails on a separate domain and server, so those requests don't hit the main server anymore. Also custom sub CSS. So any url starting with 'static' or 'media' could be handled on a separate server.

Cassandra and postgres can also both be on their own servers in the far future, if local disk space becomes the issue. It's all figured out aside from being stuck on Ubuntu 14. I hope to get back to that challenge sometime soon.

[–]magnora7 2 insightful - 1 fun2 insightful - 0 fun3 insightful - 1 fun -  (2 children)

Awesome, great points and great analysis.

I just noticed when we had our most recent DDOS our cloudflare hourly traffic went from about 3k per hour to 6k per hour, and that was enough to cause the site to crawl. However I'm not sure if that extra 3k from the DDOS was in a 5 minute span (for instance) or across the whole hour. This similar situation happened twice. They probably did it over a shorter timespan like 10 minutes rather than the whole hour, so perhaps it represents more like 6x-20x normal traffic, but anyway perhaps this is an indicator we should start looking at our bottlenecks.

I think if all the things you recommended were done, with the thumbnails and static and css being on other servers, that'd probably give us like 4x-6x the traffic capacity for the main server. And we can probably handle about 4x normal traffic on our current setup without changing anything before it really slows down. So combined that'd be 16x-24x our current traffic. That's a decent buffer from where we are. Then once we hit those limits we can start upgrading the main server to have more CPUs, of course. Maybe research what plan you would upgrade to next, so we can do it quickly if necessary.

Do you think our bandwidth or our number of connections allowed per second could be a future bottleneck as well? Or nah.

(we can technically only serve 4 requests at once right now)

Can you tell me more about this? Is it a setting in nginx? Can this number be expanded by increasing the number of CPUs? Is that the only way?

I think you'll crack the Ubuntu 18 thing. I think you're about 75% of the way there by the sounds of it. I understand the need for a breather, it's a monumental task and sometimes it's best to just walk away for a while and come back to it with a fresh perspective. The fact you got a pageload to work is a huge sign, imo.

Shit I'm scared and making a backup now... hahaaaa.

Haha always a good idea.

[–][deleted] 2 insightful - 1 fun2 insightful - 0 fun3 insightful - 1 fun -  (1 child)

I think if all the things you recommended were done, with the thumbnails and static and css being on other servers, that'd probably give us like 4x-6x the traffic capacity for the main server. And we can probably handle about 4x normal traffic on our current setup without changing anything before it really slows down. So combined that'd be 16x-24x our current traffic.

Agreed

Then once we hit those limits we can start upgrading the main server to have more CPUs, of course. Maybe research what plan you would upgrade to next, so we can do it quickly if necessary.

I'll keep this in mind, but I think we need to plan on horizontal scaling (more servers) rather than vertical scaling (more powerful server). For example with popular hosting company Linode ;) 4 cores is $40 a month, but 8 cores is $160/mo. So we could have 4 servers and 16 cores for the price of a single 8 core server. Splitting the traffic to different servers can be done with a load balancer server in front of the other servers.

"only serve 4 requests at once right now" Can you tell me more about this? Is it a setting in nginx? Can this number be expanded by increasing the number of CPUs? Is that the only way?

This is the number of CPU cores we have. Yes I believe more cores is the only way to serve insane traffic levels, but then you get into maxing out what a single network card can handle as well.

[–]magnora7 1 insightful - 1 fun1 insightful - 0 fun2 insightful - 1 fun -  (0 children)

Cool, thanks for the info. I'm all about the horizontal scaling, sounds good to me.