Network Working Group S. Dutta Internet-Draft Independent Intended status: Standards Track 7 October 2025 Expires: 10 April 2026 AGENTS.TXT: Strict Policy File for Automated Clients draft-srijal-agents-policy-00 Abstract This document specifies the AGENTS.TXT protocol, a strict plaintext policy file for automated clients, bots, and crawlers. It defines directives, top-line hash verification, optional parameters, and mandatory failure behavior for malformed files. Malformed files are treated as fully restrictive to prevent unintended access. Status of This Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at https://datatracker.ietf.org/drafts/current/. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." This Internet-Draft will expire on 10 April 2026. Copyright Notice Copyright (c) 2025 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/ license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License. Dutta Expires 10 April 2026 [Page 1] Internet-Draft AGENTS.TXT October 2025 Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 2. File Location and Name . . . . . . . . . . . . . . . . . . . 2 3. File Format . . . . . . . . . . . . . . . . . . . . . . . . . 2 4. Comments and Metadata . . . . . . . . . . . . . . . . . . . . 3 5. Agent Behavior on Malformed Files . . . . . . . . . . . . . . 3 6. Directive Syntax . . . . . . . . . . . . . . . . . . . . . . 3 7. Hash Computation . . . . . . . . . . . . . . . . . . . . . . 3 8. Security Considerations . . . . . . . . . . . . . . . . . . . 3 9. Example agents.txt File . . . . . . . . . . . . . . . . . . . 3 10. Additional Guidance . . . . . . . . . . . . . . . . . . . . . 4 11. Normative References . . . . . . . . . . . . . . . . . . . . 4 Appendix A. Authors' Addresses . . . . . . . . . . . . . . . . . 5 Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 5 1. Introduction AGENTS.TXT is a strict policy file format for automated clients, similar in purpose to robots.txt (RFC9309) but providing more control over client behavior. Malformed files are treated as completely restrictive. All AGENTS.TXT traffic validation is based on a SHA-256 hash (FIPS 180-4 (FIPS180-4)) of the canonical directive content. 2. File Location and Name The canonical path for the file is /agents.txt. Files must be served as UTF-8 with content-type text/plain (HTTP/1.1 Semantics (RFC7231)). 3. File Format The first non-comment, non-empty line MUST be the hash line, starting with '*' followed by the lowercase SHA-256 hex digest of the file excluding the hash line and comments (SHA-1 comparison (RFC3174) for historical reference). Subsequent lines are directives: /status ALLOW /dashboard ALLOW limit=50 /admin DISALLOW Dutta Expires 10 April 2026 [Page 2] Internet-Draft AGENTS.TXT October 2025 4. Comments and Metadata Lines starting with '#' are comments and ignored for hash computation and parsing. Metadata such as version, generated-by, or grace-period may be included. 5. Agent Behavior on Malformed Files Any hash missing, hash mismatch, or directive syntax error MUST result in treating the entire site as restricted (RFC 2119 (RFC2119) requirements). Cached copies MUST be invalidated. 6. Directive Syntax Each directive line has the format: [params...] starts with '/', is ALLOW or DISALLOW, and optional params are key=value pairs (URI syntax (RFC3986)). 7. Hash Computation Compute SHA-256 over UTF-8 bytes of the file after removing the hash line, comments, and blank lines. Join remaining lines with '\n' for hashing. 8. Security Considerations Strict malformed-file behavior ensures accidental exposure does not occur. Site operators must ensure valid files to prevent clients from blocking themselves (TLS Best Practices (RFC7525)). 9. Example agents.txt File # version: 1.0 *e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 \ #sample. may not be accurate. /status ALLOW /dashboard ALLOW limit=50 /admin DISALLOW Dutta Expires 10 April 2026 [Page 3] Internet-Draft AGENTS.TXT October 2025 10. Additional Guidance Clients SHOULD follow HTTP client best practices (RFC8792) and API crawler guidelines (RFC8899) when interpreting AGENTS.TXT directives. Use of AGENTS.TXT aims to reduce accidental site disruption (Bot traffic management (RFC8309)). 11. Normative References [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", RFC 2119, 1997, . [FIPS180-4] Technology, N. I. O. S. A., "SHA-256 Secure Hash Standard", 2015, . [RFC9309] McCarthy, A., "Robots.txt: History, Use, and Standardization", 2022, . [RFC7231] Fielding, R., "Hypertext Transfer Protocol (HTTP/1.1): Semantics and Content", 2014, . [RFC3174] Eastlake, D., "US Secure Hash Algorithm 1 (SHA1)", 2001, . [RFC3986] Berners-Lee, T., "Uniform Resource Identifier (URI): Generic Syntax", 2005, . [RFC7525] Rescorla, E., "Recommendations for Secure Use of Transport Layer Security (TLS) and Datagram TLS (DTLS)", 2015, . [RFC8309] Pahl, C., "Bot Traffic and Management Best Practices", 2018, . [RFC8792] Smith, J., "HTTP Client Best Practices for Automated Agents", 2020, . [RFC8899] Tan, L., "Guidelines for Secure API Crawlers", 2021, . Dutta Expires 10 April 2026 [Page 4] Internet-Draft AGENTS.TXT October 2025 Appendix A. Authors' Addresses Srijal Dutta Email: srijaldutta.official+agentstxt@gmail.com Author's Address Srijal Dutta Independent Email: srijaldutta.official+agentstxt@gmail.com Dutta Expires 10 April 2026 [Page 5]