I saw several people on LinkedIn recommending the use of the llms.txt file.
But at the same time, I saw several experts who were against using it. A bit torn between the opinions of supporters and opponents, I decided I needed to form my own opinion—and it just so happens that the Rank Math and Yoast SEO plugins have integrated the llms.txt file into their tools.
I've decided to alternate between the two extensions.
What is an llms.txt file? And, more importantly, what is it used for?
Before I explain in detail how I went about it, I’ll explain what the llms.txt file is and what benefits it’s said to offer.
Simply put, the llms.txt file is a document placed in the root directory of a website to inform language models—such as Gemini, ChatGPT, or Perplexity—about the content they can explore and understand. It works somewhat like a robots.txt file, but for artificial intelligence systems. Some recommend using it because it may improve a site’s visibility to LLMs, make it easier for them to understand the content, and give the site owner some control over what the models can access or ignore.
To put it simply, that's how it can be summarized. If you want to learn more, the Abondance website published an article on this topic .
Using the Yoast SEO and Rank Math plugins to configure the llms.txt file
Now that you know pretty much everything, let’s get back to my tests. I first used Yoast SEO, then Rank Math, and I finished my tests by manually creating an llms.txt file, which I uploaded via an FTP client to the root directory of my site, guycharles-ahondjo.com.
I began my testing by enabling the feature through Yoast SEO. As expected, Yoast SEO limits your ability to choose which pages you want to appear in the file. It has an automatic mode that, when enabled, selects the pages it deems relevant to include in the llms.txt file. So I enabled automatic mode since manual mode didn’t work the way I wanted. That’s an understatement—it’s basically useless.
Here is the content of my llms.txt file, automatically generated by Yoast SEO:
Generated by Yoast SEO v26.0, this is an llms.txt file intended for use by LLMs.
The XML sitemap for this website can be found by following [this link](https://guycharles-ahondjo.com/sitemap_index.xml).
# Guy-Charles Ahondjo’s Blog: The Cultural Adventurer
## Pages
– [Black Forest Cake Recipe](https://guycharles-ahondjo.com/recette-foret-noire/)
– [AI Bot Test](https://guycharles-ahondjo.com/test-bots-ia/)
– [Contact Guy-Charles Ahondjo](https://guycharles-ahondjo.com/on-se-contacte/)
– [MY UNCONVENTIONAL BIO](https://guycharles-ahondjo.com/ma-bio-decalee/)
– [AGC, the Revolutionary Pair of Sneakers](https://guycharles-ahondjo.com/agc-la-paire-de-basket-revolutionnaire/)
## Articles
– [ChatGPT for Dummies](https://guycharles-ahondjo.com/livres/chatgpt-pour-les-nuls/)
– [The Dangers of Sports Betting in Ivory Coast](https://guycharles-ahondjo.com/blablater/dangers-paris-sportif-en-cote-divoire/)
– [Artificial Intelligence for Dummies](https://guycharles-ahondjo.com/livres/lintelligence-artificielle-pour-les-nuls/)
– [Was It Really Better Before?](https://guycharles-ahondjo.com/blablater/cetait-vraiment-mieux-avant/)
– [New Year, New Resolutions](https://guycharles-ahondjo.com/blablater/nouvelle-annee-nouvelles-resolutions/)
## Categories
– [BLABLATER](https://guycharles-ahondjo.com/category/blablater/)
– [CULTURISSIME](https://guycharles-ahondjo.com/category/culturissime/)
– [BOOKS](https://guycharles-ahondjo.com/category/livres/)
– [DIGITECH](https://guycharles-ahondjo.com/category/digitech/)
## Tags
– [cognitive bias](https://guycharles-ahondjo.com/tag/biais-cognitif/)
– [inspiration](https://guycharles-ahondjo.com/tag/inspiration/)
– [morning](https://guycharles-ahondjo.com/tag/morning/)
– [Gaming](https://guycharles-ahondjo.com/tag/gaming/)
– [beauty](https://guycharles-ahondjo.com/tag/beaute/)
After using Yoast SEO for almost two weeks and seeing that it wasn’t producing any results, I switched to Rank Math, which is much more comprehensive than Yoast SEO. In automatic mode, it indexed 47 posts and 38 pages for me. I won’t list them all here because the list is long. It even indexed pages that I had set to “no-index.”
While my robots.txt file is crawled regularly by both Googlebot and GPTBot, among others, the llms.txt file is completely ignored.
Tired of waiting, I disabled the feature in Rank Math, created my own file, and listed the pages I wanted to select. To make sure I did things right, I followed (almost) to the letter the recommendations on the website https://llmstxt.org/, which provides precise instructions on how to proceed.
How do I add an llms.txt file to the root directory of the website?
The llmstxt.org website aims to improve LLMs’ understanding of content by recommending, on the one hand, placing a structured /llms.txt file (in Markdown format) in the root directory of the site and providing an overview of the articles to be visited—including the title, summary, sections, and links; and second, to create a Markdown (.md) version for each relevant HTML page so that the models can read them without the “noise” of the code.
To complement these steps, we recommend creating another global file—which could be named llms-full.md—that will include all the content you want visitors to see.
As mentioned, I created .md versions of the pages I had selected to make it easier for AI systems to index and understand them, and to enable them to extract relevant information more efficiently without having to parse the HTML or JavaScript.
I followed the procedure partially: I added 6 files, including /llms.txt, /llms-full.md, and 4 articles I like. That’s when I noticed that creating a page with the .md extension replaced the existing web page. In fact, a Markdown version with the same name takes the place of the original page. Since the .md pages were causing display and user experience issues, I renamed my files by adding -llm.md, for example: https://guycharles-ahondjo.com/ma-bio-decalee-llm.md. This left me with two pages for each article: one HTML page and one Markdown page.
Here's an example to help you better understand my explanation:
My HTML web page:
Here is the Markdown (.md) version of the page. I'm sure you'll agree that, from a user experience perspective, the page below is not suitable.
The last point—which several people have already noted, and with which I’d like to conclude—is that despite everything I’ve done, I haven’t detected a single hit from an LLM. Every day, I checked my log files, hoping to see that my llms.txt file had been crawled, but nothing. No .md pages were crawled either.
What can we take away from all these tests?
For nearly two months, I’ve been working on these tests, and so far, I haven’t noticed anything unusual: no LLMs have regularly visited my HTML pages, so I can say that this doesn’t boost traffic to the web pages listed in the llms.txt file. I haven’t received any hits on the Markdown pages or on the llms.txt file.
So none of my tests have yielded any concrete results. I can say that, for now, the llms.txt file and the .md files are useless!! Maybe there will be changes in the coming weeks or months, but for now, those who think this file is useless aren’t wrong—especially since the setup process takes time.