33 replies

1. James Doc 1 hr later

   @zachleat Has it met your expectations with cost?

   1. Zach Leatherman 4 hr later

      @jamesdoc this is the experiment! I’m going to blog all of this after it’s been about two weeks with some more usage data to compare

   2. Zach Leatherman 5 hr later

      @jamesdoc took a little peek into the data early: https://fediverse.zachleat.com/@zachleat/112899549153465977

2. Dan Ryan 2 hr later

   @zachleat is it running on the latest chromium now?

   1. Zach Leatherman 3 hr later

      @d for a few weeks now, yeah.

      1. Dan Ryan 3 hr later

         @zachleat sweet!

3. Zach Leatherman 5 hr later

   Preliminary comparisons with Netlify, Vercel, and AWS Lambda+Cloudfront suggest that Vercel is *drastically* overcounting serverless invocation amounts. I suppose it’s possible that I misconfigured something but Netlify was like 4h/day, Vercel was 42h/day and this new AWS setup is looking like it’s going to weigh in at 2h/day?

   Alarm bells are going off over here.

   1. William O'Connell 5 hr later

      @zachleat Lambda is priced per millisecond, it sounds like Vercel rounds to the nearest 50ms? That might explain some of the difference. Also maybe the Lambda CPU is faster?

      1. Zach Leatherman 5 hr later

         @williamoconnell oh that good to know. That’s a good distinction yeah, I believe the Vercel one is using 1769 MB of memory (which is how they scale CPU usage too) https://vercel.com/docs/functions/configuring-functions/memory and my Lambda is set for 1024 MB

   2. Simon MacDonald 5 hr later

      @zachleat this is why we built Begin in such a way that you can bring your own AWS credentials to the party.

      1. Zach Leatherman 9 hr later

         @macdonst if you gotta pick one, it’s the safest bet for sure 👍🏻

         1. Simon MacDonald 9 hr later

            @zachleat not to throw shade on anyone but those other companies really, really mark up their services.

            My AWS bill this month was $0.86 CAD. it would have been $40-50 otherwise.

            1. Zach Leatherman 11 hr later

               @macdonst amazing yeah—I understand why folks aren’t switching but if y’all have cracked the DX of AWS I’m excited to learn more

   3. Zach Leatherman 11 hr later

      A few things here: Vercel’s Edge Network cache is segmented into 18 regions. Further, the cache eviction policy is “best-effort” and assets may be evicted before they’re stale: https://vercel.com/docs/edge-network/caching#limits

      I suspect I was getting very few cache hits—not ideal at all for a screenshots service (or any resource intensive serverless function?). I wanted those images to request once and cache for as long as possible.

      1. cthos 🐱 11 hr later

         @zachleat you’ll have a lot more control over this via Cloudfront. I am very pleased watching this thread.

         Is that a weird thing to say? Am I being weird? Oh well.

         1. Zach Leatherman 11 hr later

            @cthos to be honest I feel like it’s weird that more folks aren’t complaining about this? seems like a very expensive difference in service offerings

            1. cthos 🐱 11 hr later

               @zachleat it’s the managed service trap with a free tier. It’s easy to set up, and they’re hoping to keep you there with inertia and sunk cost fallacy.

               That said AWS also has some major footguns.

               1. Zach Leatherman 11 hr later

                  @cthos Yeah, there is plenty of technical wiggle room to explain the difference away but 10× metrics has me scratching my head

                  1. cthos 🐱 11 hr later

                     @zachleat oh no I think Vercel is charging a pretty solid margin. I don’t have any hard numbers though.

                     1. Zach Leatherman 11 hr later

                        @cthos I guess my hunch is that it isn’t “overhead” in the classical sense of we’re charging you a reseller fee for convenience. I’d be fine with that. But what if the *usage* measurements are inflated?

                        1. cthos 🐱 11 hr later

                           @zachleat ohhh I see. Yeah that’d be, not good.

                           1. cthos 🐱 11 hr later

                              @zachleat I started to walk through (in addition to what you said above) and I have a couple of wild speculations. They could also be doing a "keep warm" tactic where they invoke the function to keep it awake periodically to eliminate cold starts.

                              1. Zach Leatherman 12 hr later

                                 @cthos oh huh! How warm should the start be 😅

                                 1. cthos 🐱 12 hr later

                                    @zachleat Depends on how "critical path" the function is. AWS will keep a lambda function "warm" for a period of time until it stops getting requests and then it'll put it to sleep. It'll incur a performance penalty to wake it back up. How long that penalty is depends a lot on the function and the language the function is written in. It can be as long as a full second though. One common tactic is to just have a cron job go ping the service to keep it alive every couple of minutes.

                                    1. cthos 🐱 12 hr later

                                       @zachleat Related fun, there are like... at least 2 other AWS addons for solving this problem in various ways (managed provisioning and SnapStart for Java).

      2. Zach Leatherman 11 hr later

         Worth noting that Netlify’s On-demand Builders provide global persistence across all edge nodes: https://www.netlify.com/blog/2021/10/25/faster-more-reliable-page-loads-with-update-to-on-demand-builders/

         Alternatively, CloudFront has 13 Regional Edge Caches that (in my tests with a VPN) seem to be segmented too: https://aws.amazon.com/cloudfront/features/?whats-new-cloudfront.sort-by=item.additionalFields.postDateTime&whats-new-cloudfront.sort-order=desc

         CloudFront also offers an Origin Shield product that appears to provide a similar feature but I have not enabled it: https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/cache-hit-ratio.html#cache-hit-ratio-use-origin-shield

         1. cthos 🐱 11 hr later

            @zachleat right so you can opt into the regional caches if you want to, and there is cost attached to that (iirc there’s also a buried cross region data transit cost). You don’t *have* to do it (choice!)

            You can also look at lambda@edge

            1. Zach Leatherman 11 hr later

               @cthos I’m very happy with the current setup I think. I’ll enable origin shield once I hit the inflection point where it costs less than my lambda usage (assuming that exists)

               1. cthos 🐱 11 hr later

                  @zachleat def set up a billing alarm now while you’re thinking about it if you don’t have one already.

         2. Zack Tollman 13 hr later

            @zachleat at some point in this journey, I hope you look into Fastly. Some major improvements over Cloudfront

            1. Zach Leatherman 2 days later

               @tollmanz that is a great callout. I need to dive in there.

      3. benschwarz 16 hr later

         @zachleat this is interesting because I keep getting emails about hitting the function invocation limit. From poking around I could see lots of responses that weren’t cached at the edge, but should’ve been.

         1. Zach Leatherman 2 days later

            @benschwarz another distinction here is that I don’t think cached responses are excluded from invocation counts—but I’m not super worried about that since they charge $.60 per million there. It’s the $.18 per GB hour that is biting me (at ~42h per day usage)

            correction: cached hits are allegedly not counted in invocation counts: https://vercel.com/docs/pricing/serverless-functions#managing-function-invocations