18 replies

1. Tim Severien 6 min later

   @zachleat Your timing is perfect. I actually wasn't sure to include the strawberry example in my blog post I published earlier today. Guess that's still a thing!

   1. Zach Leatherman 10 min later

      @timsev yeah, for the official record this screenshot is from a search performed just a few minutes before posting

2. fromjason.xyz ❤️ 💻 ✍️ 🥐 🇵🇷 6 min later

   @zachleat omg

3. Thomas Randolph 8 min later

   @zachleat You're absolutely correct, and it was smart to question this.

   Yes, this stuff IS getting better or no! Good catch!

   1. Zach Leatherman 12 min later

      @rockerest “Yes, this stuff IS getting better or no!” had me rolling

4. Zach Leatherman 16 min later

   why do I get the feeling that they’re starting to hard-code answers to these common tests in newer models

   1. cthos 🐱 26 min later

      @zachleat because they are, essentially?

      1. cthos 🐱 29 min later

         @zachleat and before an LLM enjoyer snipes me, the reinforcement learning is the thing doing the work here and the additional data from the rest of us dunking on the strawberry thing for years results in this.

         Now, coding benchmarks are closer to “directly cheating on the test”

   2. fromjason.xyz ❤️ 💻 ✍️ 🥐 🇵🇷 31 min later

      @zachleat *Just* thinking the same thing when I googled the strawberry question after seeing your post.

      Boilerplate with a dash of randomness to the output prose

   3. Amelia Bellamy-Royds 18 hr later

      @zachleat It could be, or it could be just that they're plagiarizing human authors' discussion of the problem from the past couple years, now in their updated training sets.

      1. Amelia Bellamy-Royds 18 hr later

         @zachleat (Try giving it a different word, like bumbleberry, that isn't likely to be in those blog posts & has a different answer, and see how it does.)

         1. Zach Leatherman 19 hr later

            @AmeliaBR I particularly like that the source cited there https://discuss.ai.google.dev/t/how-many-rs-in-strawberrry/64565 contains the prompt sample “How many "r"s are in "strawberrry"?” (note the intentional misspelling)

            1. bkardell 19 hr later

               @zachleat @AmeliaBR 😂😂😂😂

5. Simon Cox 36 min later

   @zachleat

   Depends how much you are looking to invest today…

   **rubs hands**

6. 👾 Attila Gonda 1 hr later

   @zachleat I think it would be a great coding challenge to see how many ways people could implement a letter count program in different languages as an answer for this facepalm output...

   I'm confident that even the most complex approaches (like firing up a Win98 with Excel97 and execute a VBScript on a virtual machine) would still cost less, consume less energy and be far more accurate than these... I don't even have words for them anymore.

7. Jason Garber 3 hr later

   @zachleat just another hundred billion dollars bro we’re on the verge of agi bro i am inevitable bro bro

   massive bong rip

8. Vale 5 hr later

   @zachleat @zachleat Google’s AI Overview in Search is despised even by the most diehard AI lovers because of how atrocious the model is. Complete waste of compute. I can’t fathom how much misinformation it has spread. DuckDuckGo is no better by the seems.

9. Eddy Bear February 27, 2026

   @zachleat So, I just checked this out. the first time I worded it differently and it got the answer correct. then second time I worded it like you have here and it was incorrect. I love oddities like this! Thanks for the chuckle.