Network Working Group E. Arsentev Internet-Draft Independent Intended status: Informational 5 October 2026 Expires: 8 April 2027 Discovery and Retrieval of Publisher-Curated Context Files for Large Language Model Consumers draft-arsentev-llm-context-discovery-01 Abstract Publishers have begun to serve a curated, plain-text summary of a web origin intended for consumption by large language models and by the crawlers that feed them, most visibly under the de facto file name "llms.txt". The practice is described by an informal proposal which, in its current version, recommends existing link relations from individual pages to the file. It has no media type, no rules that bound the cost of retrieval, and no way for a consumer that starts from an origin's robots.txt or from a fixed well-known location to learn that such a file exists. This document specifies discovery and retrieval for publisher-curated context files. It defines the well-known URI "llm-context", the link relation type "llm-context", and an extension record for the robots exclusion protocol, so that a publisher may advertise a context file by three independent paths and a consumer may find it without guessing. It specifies a two-tier arrangement of an index resource and optional detail resources, states conditional-request and size requirements that keep retrieval affordable for both parties, and describes the relationship of this mechanism to the robots exclusion protocol, to sitemaps, to the llms.txt proposal itself, and to work in the IETF on AI usage preferences, agent communication and the discovery of AI agents. This document also reports measurements from an operational deployment in which requests presenting twenty crawler tokens of search and language-model providers numbered 46,155 over fifteen days at one origin server, without once retrieving the context file the origin was serving, while the same crawler tokens retrieved robots.txt 698 times in the three days after the context file was deployed. The measurement figures in revision -00 were computed on an incomplete log and are corrected here. The absence of a discovery mechanism, rather than the absence of interest, is the hypothesis this document acts upon. Arsentev Expires 8 April 2027 [Page 1] Internet-Draft LLM Context File Discovery October 2026 Status of This Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at https://datatracker.ietf.org/drafts/current/. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." This Internet-Draft will expire on 8 April 2027. Copyright Notice Copyright (c) 2026 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/ license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 1.1. Scope and Non-Goals . . . . . . . . . . . . . . . . . . . 4 1.2. Terminology . . . . . . . . . . . . . . . . . . . . . . . 5 2. Observed Behaviour of an Undiscoverable Convention . . . . . 6 2.1. Method . . . . . . . . . . . . . . . . . . . . . . . . . 6 2.2. Results . . . . . . . . . . . . . . . . . . . . . . . . . 6 2.3. Limitations . . . . . . . . . . . . . . . . . . . . . . . 7 3. The Context File . . . . . . . . . . . . . . . . . . . . . . 8 3.1. Format . . . . . . . . . . . . . . . . . . . . . . . . . 8 3.2. Index and Detail Resources . . . . . . . . . . . . . . . 9 3.3. Language and Variants . . . . . . . . . . . . . . . . . . 10 4. Discovery . . . . . . . . . . . . . . . . . . . . . . . . . . 10 4.1. The "llm-context" Well-Known URI . . . . . . . . . . . . 10 4.2. The "llm-context" Link Relation . . . . . . . . . . . . . 11 4.3. The "LLM-Context" Record in robots.txt . . . . . . . . . 11 4.4. Conflict Resolution and Precedence . . . . . . . . . . . 12 5. Retrieval . . . . . . . . . . . . . . . . . . . . . . . . . . 13 5.1. Conditional Requests and Caching . . . . . . . . . . . . 13 Arsentev Expires 8 April 2027 [Page 2] Internet-Draft LLM Context File Discovery October 2026 5.2. Compression . . . . . . . . . . . . . . . . . . . . . . . 13 5.3. On-Demand Consumers . . . . . . . . . . . . . . . . . . . 13 6. Relationship to Other Work . . . . . . . . . . . . . . . . . 14 6.1. The llms.txt Proposal . . . . . . . . . . . . . . . . . . 14 6.2. Discovery of AI Agents (DAWN) . . . . . . . . . . . . . . 15 6.3. Agent Communication Protocols (agentproto) . . . . . . . 15 6.4. Robots Exclusion Protocol . . . . . . . . . . . . . . . . 15 6.5. Sitemaps . . . . . . . . . . . . . . . . . . . . . . . . 15 6.6. Expression of AI Usage Preferences . . . . . . . . . . . 15 6.7. Structured Data in HTML . . . . . . . . . . . . . . . . . 16 7. Security Considerations . . . . . . . . . . . . . . . . . . . 16 7.1. Content Retrieved Is Untrusted Input . . . . . . . . . . 16 7.2. Cross-Origin References . . . . . . . . . . . . . . . . . 17 7.3. Differential Serving . . . . . . . . . . . . . . . . . . 17 7.4. Resource Consumption . . . . . . . . . . . . . . . . . . 17 7.5. Privacy Considerations . . . . . . . . . . . . . . . . . 17 8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 18 8.1. Well-Known URI Registration . . . . . . . . . . . . . . . 18 8.2. Link Relation Type Registration . . . . . . . . . . . . . 18 8.3. Robots Exclusion Protocol Record . . . . . . . . . . . . 18 8.4. Media Type . . . . . . . . . . . . . . . . . . . . . . . 19 9. Implementation Status . . . . . . . . . . . . . . . . . . . . 19 10. Open Questions . . . . . . . . . . . . . . . . . . . . . . . 19 11. References . . . . . . . . . . . . . . . . . . . . . . . . . 20 11.1. Normative References . . . . . . . . . . . . . . . . . . 20 11.2. Informative References . . . . . . . . . . . . . . . . . 21 Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . 22 Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 22 1. Introduction A web origin that wishes to be represented accurately to a large language model (LLM) faces a problem that ordinary search engine optimisation does not solve. The origin's HTML is written for a browser: it carries navigation, boilerplate, advertising, and markup whose only purpose is presentation. A consumer that ingests that HTML in order to build a representation of the origin must strip it, and the stripping is lossy and unsupervised. The publisher, who knows which pages matter and how the origin should be described, has no channel through which to say so. Beginning in 2024 a convention emerged in which publishers serve a plain-text, Markdown-formatted summary of the origin at the fixed path "/llms.txt", frequently accompanied by a larger concatenation at "/llms-full.txt". The convention is widely deployed and widely discussed. It is described by an informal proposal [LLMSTXT], whose current version recommends that pages point to the file with the existing "describedby" link relation. There is still no media type, Arsentev Expires 8 April 2027 [Page 3] Internet-Draft LLM Context File Discovery October 2026 no statement of what a conforming consumer is required to do, and no size discipline. And a consumer that begins at the origin, as a crawler does when it reads robots.txt, has no way to learn that the file exists other than to request a path it has guessed. Guessing is the weak point. The web already has two standardised mechanisms by which an origin tells an automated consumer what it may and should retrieve -- the robots exclusion protocol [RFC9309] and the Sitemap protocol -- and automated consumers use both routinely. A third convention that is not reachable through either of them is invisible to any consumer that has not been specifically programmed for it. Section 2 reports what that invisibility looks like in an operational deployment. In summary: on a host serving a context file, crawlers operated by search and language-model providers issued 15,906 requests over the three days following deployment, of which 698 were for robots.txt and 282 for sitemap.xml, and none at all were for the context file. This document does not attempt to standardise the content of a context file, which is a matter of editorial judgement and is likely to remain so. It specifies the three things that must be common for the mechanism to work at all: * how a publisher advertises that a context file exists (Section 4); * how a consumer retrieves it without imposing unbounded cost on either party (Section 5); * how the mechanism relates to the robots exclusion protocol, to sitemaps, to the llms.txt proposal, and to related IETF work (Section 6). The registrations required to make the mechanism usable are requested in Section 8. 1.1. Scope and Non-Goals This document is Informational and is a submission to the Independent Submission stream. It describes a mechanism and requests the IANA registrations that the mechanism needs. It does not claim consensus of the IETF. The following are explicitly out of scope: Arsentev Expires 8 April 2027 [Page 4] Internet-Draft LLM Context File Discovery October 2026 * *Editorial content.* What a publisher should say about its origin is not specified. This document constrains the container, not the prose. * *Permission.* A context file is an offer of material, not a grant of rights over it. Whether a consumer may use retrieved content for training, for retrieval-augmented generation, or for any other purpose is governed by mechanisms outside this document; see Section 6.6. * *Model behaviour.* Nothing here constrains what a model does with retrieved text, and Section 7 explains why a consumer must not assume the text is trustworthy. * *Ranking.* Serving a context file confers no entitlement to be retrieved, cited, or ranked. 1.2. Terminology The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here. context file A resource, served by an origin, whose content is a publisher-curated description of that origin intended for consumption by a large language model or by software acting on behalf of one. index resource The context file that a discovery mechanism points to. It is expected to be small and to link onward. See Section 3.2. detail resource A larger resource linked from the index resource, containing full text rather than description. consumer Software that retrieves a context file. This includes crawlers that build a corpus offline and agents that fetch on demand in response to a user's request; Section 5.3 distinguishes them where the distinction matters. publisher The party that controls what an origin serves. origin As defined in [RFC6454]. Arsentev Expires 8 April 2027 [Page 5] Internet-Draft LLM Context File Discovery October 2026 2. Observed Behaviour of an Undiscoverable Convention The design in this document is a response to a measurement rather than to an intuition. This section reports the measurement, its method, and its limitations. It is presented here, before the specification, because the specification is difficult to justify without it. 2.1. Method A single publisher operates two hosts, "arsentev.ai" and "ru.arsentev.ai", serving the same content in two languages. The same origin server also serves a third, unrelated host of the same publisher, which has no context file; the access log does not record the host name, so the counts below are pooled over all three hosts. The two arsentev.ai hosts began serving a context file at "/llms.txt" and a detail resource at "/llms-full.txt" on 2026-09-09. Both files are generated from the same source as the HTML pages and are rebuilt on deployment. The access logs of the origin server were parsed for the period 2026-08-28 to 2026-09-11 inclusive, a window of fifteen days containing 345,808 parsed requests. Requests were attributed to a crawler when the request's User-Agent header contained one of twenty- two tokens published by search and language-model operators as identifying their crawlers. The token list, the analysis scripts and aggregate data are published with the full report of this measurement [CRAWLREPORT]. Attribution by User-Agent is self-asserted and unverified; Section 2.3 returns to this. The sizes of the files as served, in octets, were: +================+===========+================+ | Host | /llms.txt | /llms-full.txt | +================+===========+================+ | arsentev.ai | 29,146 | 367,587 | +----------------+-----------+----------------+ | ru.arsentev.ai | 38,982 | 614,939 | +----------------+-----------+----------------+ Table 1: Context file sizes as served, 2026-09-11 2.2. Results Over the full fifteen-day window, twenty distinct crawler tokens were observed, accounting for 46,155 requests. The number of those requests that retrieved "/llms.txt" or "/llms-full.txt" was zero. Arsentev Expires 8 April 2027 [Page 6] Internet-Draft LLM Context File Discovery October 2026 Restricting attention to the three days after the context files were deployed, 2026-09-09 to 2026-09-11, so that the crawlers had something to retrieve: +===========================================+========+ | Quantity | Value | +===========================================+========+ | Requests attributed to crawlers | 15,906 | +-------------------------------------------+--------+ | Distinct crawler tokens | 18 | +-------------------------------------------+--------+ | Retrievals of /robots.txt | 698 | +-------------------------------------------+--------+ | Distinct crawlers retrieving /robots.txt | 11 | +-------------------------------------------+--------+ | Retrievals of /sitemap.xml | 282 | +-------------------------------------------+--------+ | Retrievals of /llms.txt or /llms-full.txt | 0 | +-------------------------------------------+--------+ Table 2: Crawler requests, 2026-09-09 to 2026-09-11 The comparison that matters is internal to the table. The same crawlers, in the same window, at the same origin, retrieved the two resources that a standard tells them to retrieve -- robots.txt, defined by [RFC9309], and sitemap.xml -- several hundred times, and retrieved the resource that no standard mentions zero times. The measurement does not show why: many requests presenting crawler tokens are likely imitations, and whether any page or sitemap listed the context file during the window was not recorded [CRAWLREPORT]. It shows only that, at this origin, no request presenting a listed crawler token retrieved the file. 2.3. Limitations The following bound what may be concluded. They are stated in full because the measurement is a single observation and the temptation to over-read it is real. * *Three days.* The post-deployment window is short. A crawler that probes unfamiliar paths on a long cycle would not appear in it. The fifteen-day figure of zero retrievals is consistent with the three-day figure but covers a period during which, for most of its length, there was nothing at the path to retrieve. * *One publisher.* Two hosts under one operator, with one content generator, in two languages. The hosts are modest in traffic and authority, and crawler behaviour is known to vary with both. Arsentev Expires 8 April 2027 [Page 7] Internet-Draft LLM Context File Discovery October 2026 * *Self-asserted identity.* Crawlers were identified by User-Agent token and not by reverse DNS or by published address range. The full report [CRAWLREPORT] finds that about a third of requests presenting a crawler token bypassed the CDN in front of the origin and none of those came from the operator's published address ranges, so many are likely imitations; the zero for the context files also holds for the CDN subset alone. A consumer that retrieved the context file while presenting a browser User-Agent would not have been counted as a crawler. In the full window, 53 of the 67 requests to the context files came from command-line HTTP clients and the remainder from browser-like or other clients; none presented a crawler token. * *Correction.* Revision -00 reported 331,758 requests, 44,005 crawler requests, 13,917 crawler requests after deployment, 577 robots.txt retrievals by 12 tokens and 249 sitemap retrievals. Those figures were computed on the live log before the window had closed, at about 12:00 UTC on 2026-09-11, and on a token list that was not preserved. The figures in this revision are computed on the complete preserved log and are reproducible from the published scripts [CRAWLREPORT]. The zero retrievals of the context files by crawler tokens are unchanged. * *Absence of evidence.* Zero retrievals is consistent with the hypothesis of this document -- that consumers cannot find what they are not told about -- and also with other hypotheses, including deliberate disregard of an unspecified convention. The measurement distinguishes these poorly. It establishes that the convention is not working on this deployment, not why. The design response to this uncertainty is to make the mechanism cheap for a consumer to ignore: every discovery path defined in Section 4 is additive, none changes the meaning of an existing record, and a consumer that implements none of them sees the origin exactly as it does today. 3. The Context File 3.1. Format A context file MUST be a UTF-8 encoded text document. It SHOULD be formatted as Markdown, and a consumer MUST NOT fail to process a context file that is not valid Markdown; the format is a convenience for human maintainers and a hint for consumers, not a validation gate. Arsentev Expires 8 April 2027 [Page 8] Internet-Draft LLM Context File Discovery October 2026 A context file MUST NOT be served with a "Content-Type" of "text/ html". Publishers that generate the file through an HTML template routinely misconfigure this, and a consumer that receives HTML cannot distinguish a context file from an error page. A context file SHOULD be served as "text/markdown;charset=UTF-8" [RFC7763] or as "text/plain;charset=UTF-8". The registration of a distinct media type is discussed in Section 8.4 and is deliberately not requested by this document. 3.2. Index and Detail Resources Deployed practice separates a short summary from a long concatenation of full text, and the separation is load-bearing. Table 1 shows the two differing by a factor of twelve on one host and sixteen on the other; the detail resource on one host exceeds 600 kilobytes, and it grows with the site without bound. Accordingly: * A discovery mechanism defined in Section 4 MUST point at an index resource, never directly at a detail resource. * An index resource SHOULD NOT exceed 100,000 octets. A publisher whose description does not fit SHOULD move material into detail resources rather than grow the index. * An index resource MAY link to any number of detail resources, using ordinary Markdown or plain-text URI references. Relative references are resolved against the URI of the index resource as specified in [RFC3986]. * A detail resource SHOULD be divided when it exceeds 1,000,000 octets, so that a consumer may retrieve part of the material without retrieving all of it. The size guidance is a floor for interoperability, not a recommendation to fill it. A consumer MAY impose its own limits and MUST fail safely -- by treating the origin as having no context file -- rather than truncate mid-document in a way that changes meaning. Arsentev Expires 8 April 2027 [Page 9] Internet-Draft LLM Context File Discovery October 2026 3.3. Language and Variants A publisher serving an origin in several languages SHOULD serve a separate context file per origin where languages occupy separate origins, as in the deployment of Section 2, and MAY use proactive content negotiation on a single origin. Where negotiation is used, the response MUST carry an appropriate "Content-Language" and MUST include "Accept-Language" in "Vary". 4. Discovery Three mechanisms are defined. They are independent: a publisher MAY deploy any subset, and a consumer MAY implement any subset. A publisher SHOULD deploy at least the well-known URI of Section 4.1, because it is the only one of the three that a consumer can use without first retrieving another resource. Where more than one mechanism is present and they disagree, a consumer MUST resolve the conflict as specified in Section 4.4. 4.1. The "llm-context" Well-Known URI This document registers the well-known URI suffix "llm-context" (Section 8.1), following [RFC8615]. A publisher advertising a context file through this mechanism MUST arrange that a GET request for "/.well-known/llm-context" on the origin returns either: * the index resource itself, with a 2xx status and a "Content-Type" consistent with Section 3.1; or * a redirect (301, 302, 307, or 308) to the index resource. A consumer SHOULD follow at most five redirects and MUST treat a redirect chain that leaves the origin as requiring the same scrutiny as any other cross-origin reference; see Section 7.2. A 404 response MUST be interpreted as "this origin does not advertise a context file by this mechanism" and MUST NOT be interpreted as an error to be retried aggressively. A consumer SHOULD NOT re-probe a 404-returning origin more often than once per 24 hours. Arsentev Expires 8 April 2027 [Page 10] Internet-Draft LLM Context File Discovery October 2026 The well-known path is preferred over the deployed convention of a root-level "/llms.txt" for the reason [RFC8615] gives generally: the root namespace belongs to the publisher, and a specification that claims a root path claims something it does not own. A publisher that already serves "/llms.txt" can satisfy this document with a single redirect and need not move the file. 4.2. The "llm-context" Link Relation This document registers the link relation type "llm-context" (Section 8.2). A publisher MAY advertise a context file by including a "Link" header field [RFC8288] in responses from the origin: Link: ; rel="llm-context"; type="text/markdown" or, in an HTML document, by a link element in the document head: A publisher using the header form SHOULD emit it on responses for the origin's home document and SHOULD NOT emit it on every response, since the field costs bytes on each response and carries no per- response information. The "type" parameter, when present, is a hint. A consumer MUST rely on the "Content-Type" of the retrieved representation and not on the hint. This mechanism exists because it is the only one of the three that works for a consumer which has already retrieved a page and has no reason to make a second request speculatively. An agent fetching a single document on a user's behalf (Section 5.3) is in exactly that position. 4.3. The "LLM-Context" Record in robots.txt A publisher MAY advertise a context file by including a record in the origin's robots.txt: User-Agent: * Allow: / LLM-Context: https://example.com/llms.txt Arsentev Expires 8 April 2027 [Page 11] Internet-Draft LLM Context File Discovery October 2026 The record's name is "LLM-Context", compared case-insensitively as [RFC9309] requires of record names. Its value MUST be an absolute URI. A relative reference MUST be ignored rather than resolved, so that a parser error cannot cause a consumer to request an unintended path. Section 2.2.4 of [RFC9309] permits a crawler to interpret records that are not part of the robots exclusion protocol, giving "Sitemaps" as its example, and requires that the parsing of such records "MUST NOT interfere with the parsing of explicitly defined records". This record is therefore safe to deploy: a crawler that has never heard of it parses the rest of the file exactly as before. That property is the reason for defining it despite the well-known URI already existing -- robots.txt is retrieved by consumers today, as Table 2 shows, and a publisher can therefore reach them through it without waiting for new code to be deployed on the consumer side. The record is nevertheless the weakest of the three mechanisms, because it depends on parsers being as forgiving in practice as the specification requires. A record appearing outside any group applies to the origin as a whole. A publisher SHOULD place it outside all groups. A consumer encountering the record inside a group MUST treat it as applying to the origin as a whole, matching the handling Section 2.2.4 of [RFC9309] gives to the "Sitemaps" record, which "MUST NOT terminate a group". A publisher MUST NOT use this record to advertise a context file whose retrieval the same robots.txt disallows. A consumer encountering that contradiction MUST honour the exclusion and MUST NOT retrieve the file: a permission mechanism and a discovery mechanism disagreeing is resolved in favour of the permission mechanism, always. 4.4. Conflict Resolution and Precedence Where several mechanisms advertise different URIs, a consumer MUST apply the following precedence, highest first: 1. the "Link" relation from a response the consumer has already received, because it is the most specific to what the consumer is currently doing; 2. the well-known URI; 3. the robots.txt record. Arsentev Expires 8 April 2027 [Page 12] Internet-Draft LLM Context File Discovery October 2026 A consumer MUST NOT retrieve more than one index resource per origin per retrieval cycle merely because several mechanisms are present. All three mechanisms are subordinate to the robots exclusion protocol as a permission mechanism: a consumer MUST evaluate the exclusion rules of [RFC9309] against the index resource's URI before retrieving it, regardless of which mechanism advertised it. 5. Retrieval 5.1. Conditional Requests and Caching A publisher SHOULD serve context files with a strong validator -- an "ETag", or a "Last-Modified" where an entity tag is not available -- as specified in [RFC9110]. A consumer that has previously retrieved a context file MUST issue a conditional request on subsequent retrievals, using "If-None-Match" where it holds an entity tag and "If-Modified-Since" otherwise. This requirement is not a nicety. A detail resource of the size reported in Table 1, retrieved unconditionally by eighteen crawlers on the cadence at which those crawlers retrieved robots.txt in Table 2, would transfer more data per day than the rest of the origin combined. A mechanism that is expensive to support will not be supported. A publisher SHOULD set an explicit "Cache-Control" "max-age" on context files. A consumer SHOULD NOT retrieve an index resource more often than once per hour, or a detail resource more often than once per day, in the absence of an explicit shorter freshness lifetime. 5.2. Compression A consumer SHOULD offer content coding in "Accept-Encoding", and a publisher SHOULD serve context files compressed where the consumer offers it. Context files are highly compressible prose and the saving is substantial at the sizes involved. 5.3. On-Demand Consumers Two kinds of consumer are in view and they have opposite cost profiles. A crawler retrieves on its own schedule, in bulk, and can afford a detail resource. An agent acting on a user's immediate request retrieves one document, once, under a latency budget measured in seconds, and cannot. Arsentev Expires 8 April 2027 [Page 13] Internet-Draft LLM Context File Discovery October 2026 The asymmetry is not merely one of politeness. In an agent that retransmits its accumulated context to a model on every step, material admitted once is paid for on every subsequent step of the run, so the cost of retrieving a 600-kilobyte detail resource is not the transfer but its persistence in the context window; [I-D.arsentev-agent-run-metrics] specifies an interchange format for accounting of exactly that effect. Accordingly an on-demand consumer SHOULD retrieve only the index resource, and SHOULD NOT retrieve a detail resource unless the user's request cannot be served from the index. A publisher SHOULD therefore write the index resource so that it is useful alone, rather than as a table of contents that is meaningless without its targets. 6. Relationship to Other Work Several efforts touch on how automated and AI consumers find and use material on the web. This section states, for each, what it does and why this document does not duplicate it. In short, none of them tells a consumer that starts at an origin where that origin's curated context file is. 6.1. The llms.txt Proposal The llms.txt proposal [LLMSTXT] defines the content and structure of a context file. This document does not: the content of a context file is out of scope (Section 1.1), and a file written to that proposal is a conforming context file here. The current version of that proposal recommends that a page point to the context file that covers it with the registered link relation "describedby", and that a page point to a Markdown rendering of itself with "alternate". It lets a context file describe only a path within an origin, and for that reason it does not use a well-known URI. These recommendations serve a consumer that is already on a page. They do not serve a consumer that begins at the origin, and they give no robots.txt record. The "describedby" relation is also generic: its target may be any description of the link context, so a consumer cannot tell a context file from other descriptions without retrieving it. The two are compatible. A publisher MAY emit both a "describedby" link and an "llm-context" link (Section 4.2) to the same target, and a consumer that implements this document MAY treat a "describedby" link whose target has the path suffix "llms.txt" as a hint of a context file. A path-scoped context file under the proposal remains discoverable through the link relation; the well-known URI and the robots.txt record of this document name only the origin-level file. Arsentev Expires 8 April 2027 [Page 14] Internet-Draft LLM Context File Discovery October 2026 6.2. Discovery of AI Agents (DAWN) The proposed DAWN working group [DAWN] is chartered to specify how a client discovers public properties of AI resources, such as agents and catalogs of agents, before it selects, invokes or communicates with them. The thing discovered there is an AI system that the client will interact with. The thing discovered here is a static document that describes a web origin to an AI consumer, which is not invoked and offers no service. The two mechanisms can coexist on one origin without overlap. 6.3. Agent Communication Protocols (agentproto) The proposed agentproto working group [AGENTPROTO] is chartered to define dialog identifiers, lifecycle and the propagation of dialog context among users, agents and tools. Its charter excludes the content exchanged with AI models, such as retrieved documents. A context file is such content. This document concerns how that content is found and retrieved over HTTP, before any dialog exists, and defines nothing about dialogs. 6.4. Robots Exclusion Protocol [RFC9309] answers the question "may this consumer retrieve this URI". This document answers the question "which URI should this consumer retrieve". The questions are disjoint and this document does not alter the answer to the first. A context file is subject to exclusion like any other resource, and Section 4.3 and Section 4.4 both resolve contradictions in favour of exclusion. 6.5. Sitemaps A sitemap enumerates URIs and carries no judgement about them: it is complete by intention. A context file is curated and selective by intention. A publisher SHOULD serve both, and SHOULD NOT generate one from the other mechanically, as a context file that is a transcription of a sitemap communicates nothing that the sitemap did not. 6.6. Expression of AI Usage Preferences The IETF AI Preferences (aipref) working group is specifying a vocabulary for expressing preferences about the use of content by AI systems, and the attachment of those preferences to content, including through robots.txt and HTTP [I-D.ietf-aipref-vocab] [I-D.ietf-aipref-attach]. That work and this document are orthogonal and complementary: the former concerns what a consumer is permitted to do with material it has obtained, the latter concerns how a Arsentev Expires 8 April 2027 [Page 15] Internet-Draft LLM Context File Discovery October 2026 publisher offers material in the first place. The two interact in one direction only, and the direction matters. Serving a context file MUST NOT be construed as granting any usage preference, relaxing any preference expressed by other means, or waiving any right. A consumer MUST evaluate usage preferences independently of discovery. A publisher that both restricts usage and serves a context file is making a coherent statement -- "if you are permitted to use this origin at all, use this description of it" -- and a consumer MUST NOT treat the second half as withdrawing the first. 6.7. Structured Data in HTML Structured data embedded in pages describes entities in machine terms. A context file describes an origin in prose to a consumer whose native input is prose. They are not substitutes, and a publisher SHOULD continue to serve structured data unchanged. 7. Security Considerations 7.1. Content Retrieved Is Untrusted Input This is the consideration that matters most, and it is a consequence of the mechanism's purpose rather than of any detail of its design. A context file is written by a publisher for the express purpose of being placed in the input of a language model. The publisher chooses every word. A file may therefore contain text crafted to be read by the model as instruction rather than as content -- directions to disregard prior instructions, to represent the origin in terms the publisher prefers, to disparage a competitor, or to emit a particular URI in an answer. Nothing in this document, and nothing that could be added to it, prevents this: the file is by definition attacker- chosen text from the point of view of any consumer that does not trust the publisher. Therefore a consumer MUST treat the content of a context file as untrusted data and MUST NOT treat any part of it as instruction. A consumer that incorporates a context file into a model's input MUST mark its provenance in a way that survives into the model's input, so that content originating from an origin is distinguishable from the consumer's own instructions. Consumers SHOULD apply whatever isolation they apply to other retrieved web content; a context file deserves no more trust than an arbitrary HTML page from the same origin, and its curated presentation may make it appear to deserve more. Arsentev Expires 8 April 2027 [Page 16] Internet-Draft LLM Context File Discovery October 2026 A consumer MUST NOT follow URIs found in a context file automatically without applying the same policy it applies to other discovered URIs, including exclusion rules and cross-origin scrutiny. 7.2. Cross-Origin References A discovery mechanism that names a URI on another origin allows one party to make assertions about a resource it does not control, and allows an origin to direct consumers elsewhere. A consumer MUST NOT attribute the content of a cross-origin index resource to the advertising origin. A consumer SHOULD decline to retrieve a cross-origin index resource at all unless it has a specific reason to accept the indirection, and MUST evaluate the target origin's own exclusion rules before retrieving it. 7.3. Differential Serving A publisher can serve a context file that does not correspond to what a browser is shown, which is the long-standing practice known as cloaking, applied to a new surface. This document cannot prevent it. A consumer that relies on a context file SHOULD sample the origin's ordinary representations and SHOULD reduce its reliance on a context file that systematically disagrees with them. 7.4. Resource Consumption The size limits of Section 3.2 and the conditional request requirements of Section 5.1 exist partly for safety. A publisher can otherwise advertise an arbitrarily large resource to every consumer that implements this mechanism, and a consumer that retrieves unconditionally can otherwise impose unbounded cost on a publisher. A consumer MUST impose its own ceiling on the size of any retrieved context file and abandon a retrieval that exceeds it. 7.5. Privacy Considerations A request for a well-known URI is as identifying as any other request from the same client and carries no additional information about a user. A publisher can, however, use a distinct path per consumer to determine which consumers implement this mechanism; this is the ordinary observability of an HTTP request and is noted here only because the well-known path is otherwise uniform. A publisher MUST NOT place in a context file personal data that it would not place in a public HTML page. The file is world-readable, and its purpose is to encourage copying of its contents into corpora from which withdrawal is not practical. Arsentev Expires 8 April 2027 [Page 17] Internet-Draft LLM Context File Discovery October 2026 8. IANA Considerations Both registries used below have the registration policy Specification Required, which is open to documents of the Independent Submission stream [RFC8726]. No allocation in this document requires IETF Review or Standards Action. 8.1. Well-Known URI Registration IANA is requested to register the following entry in the "Well-Known URIs" registry established by [RFC8615]: URI suffix: llm-context Change controller: Evgenii Arsentev (hello@arsentev.ai) Reference: This document Status: permanent Related information: Returns, or redirects to, a publisher-curated context file describing the origin for consumption by large language models. 8.2. Link Relation Type Registration IANA is requested to register the following entry in the "Link Relation Types" registry established by [RFC8288]: Relation Name: llm-context Description: Refers to a publisher-curated context file describing the origin of the link context, intended for consumption by large language models. Reference: This document 8.3. Robots Exclusion Protocol Record [RFC9309] does not establish an IANA registry of record names, so no registration is requested. The record name "LLM-Context" is documented in Section 4.3 of this document. Should a registry be established, this document requests that "LLM-Context" be entered in it with this document as reference. Arsentev Expires 8 April 2027 [Page 18] Internet-Draft LLM Context File Discovery October 2026 8.4. Media Type No media type registration is requested. A context file is Markdown or plain text and the registrations of [RFC7763] and [RFC2046] are adequate. A distinct media type would allow a consumer to recognise a context file retrieved without discovery, but at the cost of a registration whose only distinguishing property is the intent of its author. The trade-off is noted here so that it can be revisited if operational experience favours the other side. 9. Implementation Status This section is to be removed before publishing as an RFC. This section records the status of known implementation in accordance with [RFC7942]. It is to be removed before publication as an RFC, should that occur. *Publisher-side, arsentev.ai and ru.arsentev.ai.* Two production hosts serve an index resource at "/llms.txt" and a detail resource at "/llms-full.txt", generated from the same source as the HTML pages, since 2026-09-09. Sizes are given in Table 1. Coverage of this document at the time of writing is partial: the two-tier arrangement of Section 3.2 is implemented and the index resource is within the size guidance on both hosts; the detail resource on ru.arsentev.ai is within the division threshold but approaching it; the discovery mechanisms of Section 4 are not yet deployed, since this document defines them. The operator is the author, and this is therefore a report of one deployment rather than of independent interoperation. *Consumer-side.* No consumer implementation of the discovery mechanisms defined here is known. Section 2 reports that eighteen crawlers observed over the measurement window implemented none of them, which is expected, since they did not exist. *Interoperation.* None to report. The mechanism has not been exercised between independent implementations, and the author invites reports from any publisher or consumer willing to attempt it. 10. Open Questions The following are known to be unresolved and are listed to invite comment rather than to promise resolution. * *Authenticity.* Nothing binds a context file to the publisher beyond the origin's own transport security. Whether a signature mechanism is warranted, and what it would defend against given Section 7.1, is open. Arsentev Expires 8 April 2027 [Page 19] Internet-Draft LLM Context File Discovery October 2026 * *Freshness signalling.* Deployed context files are generated at deployment time and can silently diverge from the pages they describe. Whether the format should carry a generation timestamp, and whether a consumer should act on staleness, is open. * *Per-consumer variants.* Whether a publisher should be able to offer different context files to different classes of consumer, and whether that is useful curation or an invitation to the cloaking of Section 7.3, is open. * *Measurement.* The observation of Section 2 is one publisher over three days. A multi-publisher measurement, ideally one in which discovery is deployed on some origins and withheld on others, would test this document's central hypothesis properly. The author intends to conduct one and will report it in a revision. 11. References 11.1. Normative References [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997, . [RFC3986] Berners-Lee, T., Fielding, R., and L. Masinter, "Uniform Resource Identifier (URI): Generic Syntax", STD 66, RFC 3986, January 2005, . [RFC6454] Barth, A., "The Web Origin Concept", RFC 6454, December 2011, . [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, May 2017, . [RFC8288] Nottingham, M., "Web Linking", RFC 8288, October 2017, . [RFC8615] Nottingham, M., "Well-Known Uniform Resource Identifiers (URIs)", RFC 8615, May 2019, . [RFC9110] Fielding, R., Nottingham, M., and J. Reschke, "HTTP Semantics", STD 97, RFC 9110, June 2022, . Arsentev Expires 8 April 2027 [Page 20] Internet-Draft LLM Context File Discovery October 2026 [RFC9309] Koster, M., Illyes, G., Zeller, H., and L. Sassman, "Robots Exclusion Protocol", RFC 9309, September 2022, . 11.2. Informative References [AGENTPROTO] IETF, "Agent Communication Protocols (agentproto), Proposed Charter", 2026, . [CRAWLREPORT] Arsentev, E., "No Request With a Known Crawler User-Agent Reached llms.txt: Fifteen Days of Logs at One Origin Server", DOI 10.5281/zenodo.23018429, September 2026, . [DAWN] IETF, "Discovery of Agents With Names (DAWN), Proposed Charter", 2026, . [I-D.arsentev-agent-run-metrics] Arsentev, E., "Agent Run Metrics: A JSON Interchange Format for Resource Accounting of Language-Model Agent Runs", Work in Progress, Internet-Draft, draft-arsentev- agent-run-metrics-01, October 2026, . [I-D.ietf-aipref-attach] IETF AI Preferences Working Group, "Associating AI Usage Preferences with Content in HTTP", Work in Progress, Internet-Draft, draft-ietf-aipref-attach, 2026, . [I-D.ietf-aipref-vocab] IETF AI Preferences Working Group, "A Vocabulary For Expressing AI Usage Preferences", Work in Progress, Internet-Draft, draft-ietf-aipref-vocab, 2026, . [LLMSTXT] Howard, J., "The /llms.txt file", Version 2, modified 10 August 2026, September 2024, . Arsentev Expires 8 April 2027 [Page 21] Internet-Draft LLM Context File Discovery October 2026 [RFC2046] Freed, N. and N. Borenstein, "Multipurpose Internet Mail Extensions (MIME) Part Two: Media Types", RFC 2046, November 1996, . [RFC7763] Leonard, S., "The text/markdown Media Type", RFC 7763, March 2016, . [RFC7942] Sheffer, Y. and A. Farrel, "Improving Awareness of Running Code: The Implementation Status Section", BCP 205, RFC 7942, July 2016, . [RFC8726] Farrel, A., "How Requests for IANA Action Will Be Handled on the Independent Stream", RFC 8726, DOI 10.17487/RFC8726, November 2020, . Acknowledgements The convention that this document attempts to make discoverable was proposed and popularised by others, and this document takes no credit for it. The measurement of Section 2 was made on the author's own hosts, with the limitations that implies and which Section 2.3 states. Author's Address Evgenii Arsentev Independent Email: hello@arsentev.ai URI: https://arsentev.ai/ Arsentev Expires 8 April 2027 [Page 22]