Have the LLM call you
Taking the time to get things right
Sitting down at her desk, the women extracts her notebook (the one for writing stuff in), takes out her lucky fountain pen (it's been with her longer than she can remember) and starts making notes about what tasks she has to do today. These tasks aren't going to be fun but the ritual involved gives her time to think about what can be achieved, what can be delayed and know what she needs help with. Only then does she start up her mac, launch Xcode and get to work.
This is a ritual I have been doing time and time and time again over the years when I approach work. It's also super familiar to anyone that uses an LLM to assist them with coding. They set out the work required, they note down the different tasks and they establish the open questions that need answering before getting stuck in.
For people, they learn over many many years via experience, asking questions, researching questions, discovering answers, having adventures, being wrong and finding new and innovative ways to solve problems.
But what about an LLM? How does it come to solve and to know what an answer to a question is? Well, lets slow down a bit, put away the LLM harness and think through what's going on a bit. So grab a cup of tea, relax and learn about how an LLM will use internet searches to find an answer. More importantly, how we can help the LLM find the write answer by providing context around the content of our websites.
A text file like no other
For years, the way things were found on the web was having a web crawler index your site and discover what's there through robots.txt and sitemap.xml. These are still super useful for SEO, but they don't help an LLM understand your site. What's required is a list of instructions about your site and its content. Enter the llms.txt file and it's expanded counterpart llms-full.txt. These instructions are what guides the model to understand your site. Make good use of them.
Markdown
Your sites content is in a combination of html, javascript and css. It's heavily marked up with extra details that make it great for human navigation and interaction. These extras are expensive and complicated for an LLM to consume. What is required here is to provide the textual content from your site so that it can be consumed easily by the LLM.
For the benefit of the LLM
This is where it's all about communication. The content you provide to the LLM needs to be in a form that the llm understands. In the llm.txt files you provide details about what your site provides, who it is for and who it isn't for. You can also extend the structured data in JSON-LD on your pages to tell the llm about who wrote the site and what the purpose of the site is.
A poisoned cup of tea
It's great that there are academic papers out there which cover how best to engage in Generative Engine Optimization, it's something that can be easily poisoned as a language model relies on statistic generation and can't distinguish between a legitimate source and one that is exploitive. An interesting article on this is here. This needs to be considered in depth, but is beyond this article. I'm not touching that argument just yet until I'm sure about what I want to say.
Be involved in the community
So, can we as software engineers escape this whole landscape? Sadly not. What we can do is make the most of it so that the software we build, the articles we write are all surfaced in the way we want by an LLM.