32 replies

1. Zach Leatherman 1 min later

   For those managing servers: what kind of concurrent request count spikes do y’all prepare for?

   Looks like the Locust load testing tool’s “first test” starts with 100 concurrent users: https://docs.locust.io/en/stable/quickstart.html

   1. Zach Leatherman 16 min later

      Found a pretty hefty blog post about concurrency benchmarks using the Nginx web server.

      https://blog.nginx.org/blog/testing-the-performance-of-nginx-and-nginx-plus-web-servers

      Looks like the slowest metric (1 CPU, HTTPS) handled 428 connections per second.

      1. William O'Connell 44 min later

         @zachleat Yeah, even if all ~30,000 mastodon instances hit the server within 60 seconds (unlikely), that's still only 500 requests per second. That seems pretty manageable? At least for public content that's identical for every user. A real DDoS by an attacker can be *millions* of requests per second.

         1. Zach Leatherman 1 hr later

            @williamoconnell yeah, that third blog post notes a hypothetical of 6.7k simultaneous requests—which seems like a lot!

            1. William O'Connell 1 hr later

               @zachleat It says they're supposed to be spread over 60 seconds though. In practice probably more, since it should take some time for all the instances to pick up the post in the first place. The author says "I'm yet to see that work for me" but I'm unclear of that means they have evidence that it isn't happening or their site is just still crashing regardless. The lack of specific data makes it hard to draw conclusions.

               1. Zach Leatherman 1 hr later

                  @williamoconnell I’d assume that is just a timing issue between writing of the blog post and the rollout of the 60s “jitter” addition—a temporary mitigation for sure.

                  Mastodon *could* upload the preview as an asset sidecar’d with the post, similar to any image upload.

      2. John Hobbs 6 hr later

         @zachleat I would think with session reuse you wouldn't pay that TLS connection cost constantly. Also I imagine OpenSSL/BoringSSL has improved in the last seven years, and more cryptographic operations are done in CPU now then back then.

         That said I don't think I'd run anything without a caching server in front anymore, unless it was very server interactive and would miss constantly. Varnish did amazing things at Flywheel.

         1. Zach Leatherman 2 days later

            @jmhobbs great insight, thank you!

            As a side note I think I’ve seen some of these caching defaults play out at the serverless platform level recently: https://www.zachleat.com/web/serverless-cost/ Some platforms are born from use cases that are intensely personalized, not heavily cached.

   2. William O'Connell 18 min later

      @zachleat I've never been lucky enough to have something that was this popular so I can't really speak from experience, but I'm a little confused by why it's causing problems for so many sites. I would think any CDN should be able to solve this, even a really short TTL like 30 seconds would reduce the flood to just a few requests.

      Usually when testing an application I'm happy if it can handle 100-200 requests per second, but for public content I would always have a CDN in front.

   3. cthos 🐱 54 min later

      @zachleat if you’re load testing, I start with typical traffic patterns and use a multiplier.

      If stress testing, start small and ramp it up until it falls over.

2. Dave 🧱 2 min later

   @zachleat I think a vast part of my career is about making things someone else's problem 😂

   1. Zach Leatherman 3 min later

      @DavidDarnes for the right things, it makes sense!

3. Sara Joy 10 min later

   @zachleat yeah I haven't noticed this with Netlify but then I also don't pay attention haha, I just post and scoot

   1. Zach Leatherman 17 min later

      @sarajw post and scoot 🏆

      1. Noah Liebman 22 min later

         @zachleat @sarajw um, toot ’n’ scoot 😏

         1. Zach Leatherman 22 min later

            @noleli @sarajw toot scoot booooogie ♩♪♫♬

            1. Sara Joy 41 min later

               Who's that tootin' in the feeds?
               It's two devrels and they're on fleek
               Blogs and demos and typin' hands
               What's your language? JS I am!

               Instances come to play, the server's not fast enough
               They'd best stay away when the websites get a shove

               Toot, scoot, riot (riot!)
               Leave a link and run away
               Toot, scoot, riot (riot!)
               No one visits my site anyway...

               https://youtu.be/JZm1krVKzBY

               @zachleat @noleli

               1. Noah Liebman 47 min later

                  @sarajw @zachleat oh em gee. Amazing ✨

                  1. Sara Joy 51 min later

                     @noleli couldn't resist 😅

                     @zachleat

                     1. Zach Leatherman 56 min later

                        @sarajw @noleli for some reason I’m now thinking of Jim Carrey in The Mask 😅

4. Dustin Rue 1 hr later

   @zachleat this is _easily_ solved with a caching strategy which I have mentioned a few times to people who metion this issue. You can read about it at https://dustinrue.com/2023/02/avoiding-stampeding-mastodons/. It is not necessarily something only Cloudflare can solve, but can be used to inform people on how to fix the issue using any caching solution and some effort.

   1. Zach Leatherman 1 hr later

      @dustinrue WordPress always needs one more thing to work like it should by default

      1. Dustin Rue 1 hr later

         @zachleat Generally speaking I disagree. Any the site that is Next.js, Drupal or any other countless number of systems that generate content on the fly and isn't a static site will fall victim to this same issue.

         1. Zach Leatherman 1 hr later

            @dustinrue the nuance I’d contribute is that (in my opinion) static sites are a better default for most web sites and dynamism should be an additive architectural layer.

            Removes an entire class of problems for most folks

5. Emelia 👸🏻 1 hr later

   @zachleat there's ongoing work to improve the situation

   1. Zach Leatherman 1 hr later

      @thisismissem great!

      For the record I’m mostly surprised that sites are being taken down by this level of concurrency

      1. Flaki 1 hr later

         @thisismissem would you happen to have a link to the issue/discussions perhaps, I'd be interested to see what's happening to try to address this?
         @zachleat

         1. Zach Leatherman 1 hr later

            @flaki @thisismissem https://github.com/mastodon/mastodon/issues/23662 is the one you want, I think!

            1. Emelia 👸🏻 2 hr later

               @zachleat @flaki yeah, there's that, but the discussion I know of is happening internally within the core team. Current direction is to be publishing preview data via ActivityPub.

6. Private reply 1 hr later
   1. Private reply 1 hr later
      1. Private reply 1 hr later