7 replies

1. cthos 🐱 9 min later

   @zachleat because they are, essentially?

   1. cthos 🐱 12 min later

      @zachleat and before an LLM enjoyer snipes me, the reinforcement learning is the thing doing the work here and the additional data from the rest of us dunking on the strawberry thing for years results in this.

      Now, coding benchmarks are closer to “directly cheating on the test”

2. fromjason.xyz ❤️ 💻 ✍️ 🥐 🇵🇷 14 min later

   @zachleat *Just* thinking the same thing when I googled the strawberry question after seeing your post.

   Boilerplate with a dash of randomness to the output prose

3. Amelia Bellamy-Royds 17 hr later

   @zachleat It could be, or it could be just that they're plagiarizing human authors' discussion of the problem from the past couple years, now in their updated training sets.

   1. Amelia Bellamy-Royds 18 hr later

      @zachleat (Try giving it a different word, like bumbleberry, that isn't likely to be in those blog posts & has a different answer, and see how it does.)

      1. Zach Leatherman 18 hr later

         @AmeliaBR I particularly like that the source cited there https://discuss.ai.google.dev/t/how-many-rs-in-strawberrry/64565 contains the prompt sample “How many "r"s are in "strawberrry"?” (note the intentional misspelling)

         1. bkardell 18 hr later

            @zachleat @AmeliaBR 😂😂😂😂