27 replies

1. Jan <3 4 min later

   Oh wow that’s cringe

2. Private reply 4 min later
   1. Zach Leatherman 5 min later

      it’s .too-obscure

   2. Luke 15 hr later

      Why isn't it a .md file too...

      1. Zach Leatherman 18 hr later

         hear me out: what if we call it sitemap.txt and then we put xml in it

         hear me out again: what if we put markdown in robots.txt

         1. Michael 18 hr later

            Decades of research and development into software that can spider and map the World Wide Web, and the new hotness needs a special file to understand your blog.

            1. Zach Leatherman 19 hr later

               “Converting complex HTML pages with navigation, ads, and JavaScript into LLM-friendly plain text is both difficult and imprecise.” on llmstxt.org 😅

               1. Michael 19 hr later

                  Weird how my phone can do it instantly when I click the Safari Reader button.

                  1. Zach Leatherman 19 hr later

                     haha 😅

                     I personally believe it’s a SPA thing — spidering with JavaScript support is far more expensive

                     1. Michael 19 hr later

                        BRB spinning up a saas business to use LLMs to generate llms.txt.

                        1. Anthony Whitford 21 hr later

                           Isn’t that what FireCrawl does?

3. Matt Kane 15 hr later

   I'm not sure the existence of a Starlight plugin for it really counts as "full steam ahead". It makes it easier for people who want it to add it to their own docs sites. If it *does* get wider adoption and would be helpful to users then maybe we'll add it to the Astro docs site but not right now.

   1. Zach Leatherman 18 hr later

      a starlight plugin from a core maintainer of astro, that shows up as result #2 on a search for "llms.txt" on npm? if y’all are hesitant to have it framed this way, I’d be curious to learn more about why!

      1. Matt Kane 18 hr later

         I've nothing against llms.txt. The only reason I don't want it framed that way is because it's not true! Most Astro core maintainers have lots of our own projects. You could just as well say that Astro has gone all-in on Airtable because I made a loader for it.

         1. Zach Leatherman 18 hr later

            I’ve now had three different Astro employees talk to me about this post on two different social networks so something isn’t quite matching up

            1. Matt Kane 18 hr later

               Are we saying different things?

      2. Wes Souza 18 hr later

         When I read “full steam” I expected astro.build/llms.txt to exist.

         1. Matt Kane 18 hr later

            Yes, that's what I would consider to be the main criterion. Even then it's a bit of a small thing to really qualify as "full steam ahead". We're also full steam ahead on webfinger, sitemap.xml and robots.txt

4. Alexander Lichter 18 hr later

   IMO it is good to have an easy integration for people building pages (and our own @nuxt.com docs for example). It is an extra module on purpose right now, especially because the llms.txt "proposal" is still pending.

5. zeu 18 hr later

   software developers are by far the biggest users for AI chatbots and one of the biggest criticisms against using LLMs is hallucinations

   if ppl are gonna use LLMs to learn about web dev, i’d rather the data be accurate, and /llms.txt gives library authors control on what is being said by the bots

   1. Griffin Wiebel 18 hr later

      Maybe it's unclear to me, but haven't these popular LLMs already been trained on public data? And what's to stop them from continuing to scrape and index public data rather than using these generated txt files? Sounds like just another thing to train on.

      1. definitelymayb3.bsky.social 1 day later

         At a guess, even the scraped public data was further processed to remove noise. The process likely imperfect.

6. Angelika Cathor 19 hr later

   Your post gave me the impression that all three domains are hosting an llms.txt file but I only found it on the svelte.dev domain

   1. Zach Leatherman 19 hr later

      github.com/nuxtlabs/nux... and delucis.github.io/starlight-ll... from a core astro maintainer are the sources for the above

      1. Angelika Cathor 19 hr later

         Thank you

7. Soufiane 19 hr later

   First Astro doesn't have a /llms.txt page. Also, making docs more accessible to llm will prevent more hallucination for those who use it. Finally hopefully some legislation will pass to stop companies from scrapping websites that don't opt-in this llms.txt stuff.

8. Sean Roberts 1 day later

   I'm not sold on /llms.txt for a lot of reasons - not least of which is the cost for digital services. That said, the same input for humans is not the same as what's needed for LLMs and the format for providing it is also diff - therefore I'm not certain that this is a matter of using the same HTML