intarea D. K. Gillmor Internet-Draft M. K. Gerchick Intended status: Informational ACLU Expires: 7 June 2026 4 December 2025 Automated Discovery Of Audit Reports (audits.json) draft-dkg-audits-json-00 Abstract This document describes a mechanism that an organization can use to enable automatic discovery of documents associated with regulatory compliance. It is motivated by regulations that require, for example, publicly accessible audits of automated decision-making processes in hiring. About This Document This note is to be removed before publishing as an RFC. The latest revision of this draft can be found at https://dkg.github.io/audits.json/. Status information for this document may be found at https://datatracker.ietf.org/doc/draft-dkg- audits-json/. Source for this draft and an issue tracker can be found at https://github.com/dkg/audits.json/. Status of This Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at https://datatracker.ietf.org/drafts/current/. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." This Internet-Draft will expire on 7 June 2026. Gillmor & Gerchick Expires 7 June 2026 [Page 1] Internet-Draft audits.json December 2025 Copyright Notice Copyright (c) 2025 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/ license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License. Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 1.1. Requirements Language . . . . . . . . . . . . . . . . . . 3 1.2. Terminology . . . . . . . . . . . . . . . . . . . . . . . 3 1.3. Goals . . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.4. Non-Goals . . . . . . . . . . . . . . . . . . . . . . . . 3 2. Locating The audits.json Summary . . . . . . . . . . . . . . 4 3. audits.json Structure . . . . . . . . . . . . . . . . . . . . 4 3.1. audits Content . . . . . . . . . . . . . . . . . . . . . 5 3.2. operations Content . . . . . . . . . . . . . . . . . . . 5 4. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 5 5. References . . . . . . . . . . . . . . . . . . . . . . . . . 5 5.1. Normative References . . . . . . . . . . . . . . . . . . 6 5.2. Informative References . . . . . . . . . . . . . . . . . 6 Appendix A. Test Vectors . . . . . . . . . . . . . . . . . . . . 6 A.1. Example audits.json . . . . . . . . . . . . . . . . . . . 7 Appendix B. JSON Schema For audits.json . . . . . . . . . . . . 7 Appendix C. Examples of Regulatory Regimes . . . . . . . . . . . 9 C.1. Example Regulatory Text . . . . . . . . . . . . . . . . . 10 Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . 10 Document History . . . . . . . . . . . . . . . . . . . . . . . . 10 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 10 1. Introduction An increasing number of regulatory regimes require organizations involved in certain business practices to provide a degree of transparency in their business operations by posting reports of audits in an accessible location. For example, New York City's 2021 Local Law 144 ([NYC-LL144]) established auditing requirements for employers who use automated decision-making tools in the employment process, but it has proved challenging to even find these audits effectively (see [Auditing-the-Audits]). Gillmor & Gerchick Expires 7 June 2026 [Page 2] Internet-Draft audits.json December 2025 For a business that has a website, the natural place to provide access to the audit is on that website. This document describes a standard mechanism that can be used to point to any audit posted associated with regulatory compliance. The mechanism is an audits.json summary document, found at a well- known URL, which allows the hosting domain to refer to specific business operations, relevant regulatory regimes, and their associated audit reports. Adoption of this mechanism should make it easier for businesses to comply with these requirements, while also making it easier for researchers, analysts, and regulators to assess compliance and evaluate the overall effectiveness of the regulations. 1.1. Requirements Language The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here. 1.2. Terminology * An "audit report" is a document produced by an organization or an independent entity commissioned by the organization to describe -- according to some kind of regulatory compliance -- a part of the organization's business operations. 1.3. Goals * Provide a standardized way for an organization to publicize audit reports associated with given regulatory requirements. * Provide a standardized way for a reviewer of audit reports associated with a regulatory requirement to find those audit reports. 1.4. Non-Goals Discovering an audit report is merely the first step in a process of having an effective regulatory regime based on audits. This specification is focused merely on this satisfying this necessary but insufficient stage of larger work. Gillmor & Gerchick Expires 7 June 2026 [Page 3] Internet-Draft audits.json December 2025 * This specification makes no attempt to describe the syntax or semantics of any particular audit report. The underlying assumption is that the audit reports for different regulations or compliance measures will have different requirements for content, structure, syntax, and so on. * This specification also does not associate a real-world organization with a particular domain name. If a researcher wants to find the audits associated with Example Corp, they need to associate Example Corp with the example.com domain name independently in order to use this specification. Likewise, any regulation making use of this mechanism needs to explicitly describe the link between the regulated entity and the domain name or domain names on which it is expected to publish this summary. * This specification does not describe a way for a casual visitor to a given website to find these audit reports. A regulation that requires easy public accessibility of a report may need to offer additional user interface or user experience guidance in addition to requiring the use of this mechanism. 2. Locating The audits.json Summary This specification uses the .well-known URL space defined by [RFC8615]. A given domain hosts the audits.json summary in /.well-known/ audits.json within the website operated by the organization. If Example Corporation operats https://example.com/, then the summary report would be found at https://example.com/.well-known/audits.json. 3. audits.json Structure The object served from the URL described in Section 2 will have Content-Type: application/json, and will consist of a single dictionary object with at least two top-level keys: operations and audits. The overall object relationship looks like this: +------------+ +--------+ | operations | .-->| audits | +------------+ | +--------+ | urls | | | title | | regs | | | urls | | audits +-' +--------+ +------------+ Gillmor & Gerchick Expires 7 June 2026 [Page 4] Internet-Draft audits.json December 2025 Figure 1: Data Structure For `audits.json` 3.1. audits Content The audits member is an object where each member is an object with: * title, simple textual string describing the audit, and * date, date object describing the calendar date (year, month, and day) of the audit's publication, and * urls, an array of URLs that point to the relevant materials for the audit 3.2. operations Content The operations member is a list of objects, each of which describes some set of business operations, via the following members: * urls, an array of URLs, each of which describes a business operation covered by this object (for example, a job listing), * regs, an array of URLs, each of which refers to a piece of regulatory guidance, and * audits, an array of keys which can be used to point to specific audits 4. IANA Considerations IANA should register audits.json in the "Well-Known URIs" registry, with the following values: * URI suffix: audits.json * Change controller: IETF (is this the right choice, if this is an informational draft? is there a better group to control updates to this specification?) * Specification document(s): This document * Status: provisional (unless we see wider adoption, in which case we should ask for permanent) * Related information: Any other examples we want to point to? 5. References Gillmor & Gerchick Expires 7 June 2026 [Page 5] Internet-Draft audits.json December 2025 5.1. Normative References [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, . [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, May 2017, . [RFC8615] Nottingham, M., "Well-Known Uniform Resource Identifiers (URIs)", RFC 8615, DOI 10.17487/RFC8615, May 2019, . 5.2. Informative References [Auditing-the-Audits] Gerchick, M., Encarnación, R., Tanigawa-Lau, C., Armstrong, L., Gutiérrez, A., and D. Metaxa, "Auditing the Audits: Lessons for Algorithmic Accountability from Local Law 144's Bias Audits", ACM, Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency pp. 29-44, DOI 10.1145/3715275.3732004, June 2025, . [CO-SB205] "Colorado Senate Bill 24-205", May 2024, . [I-D.bhutton-json-schema-01] Wright, A., Andrews, H., Hutton, B., and G. Dennis, "JSON Schema: A Media Type for Describing JSON Documents", Work in Progress, Internet-Draft, draft-bhutton-json-schema-01, 10 June 2022, . [JV] Tekuri, S. K., "JSONSchema Validation using Go", n.d., . [NYC-LL144] "Rules of the City of New York: Automated Employment Decision Tools", April 2023, . Appendix A. Test Vectors Gillmor & Gerchick Expires 7 June 2026 [Page 6] Internet-Draft audits.json December 2025 A.1. Example audits.json { "operations": [ { "urls": [ "https://example.com/jobs/sysadmin", "https://example.com/jobs/clerk" ], "regs": [ "https://codelibrary.amlegal.com/codes/newyorkcity/latest/NYCrules/0-0-0-138391" ], "audits": [ "fizz", "sparkle" ] }, { "urls": [ "https://example.com/jobs/cfo", "https://example.com/jobs/cto" ], "regs": [ "https://codelibrary.amlegal.com/codes/newyorkcity/latest/NYCrules/0-0-0-138391" ], "audits": [ "fizz" ] } ], "audits": { "fizz": { "title": "Fizz Resume Prioritization Service", "urls": [ "https://fizz.example/bias_audits/aedt_audit_2024.pdf", "https://fizz.example/bias_audits/aedt_2024_data.tgz" ], "date": "2024-12-13" }, "sparkle": { "title": "Sparkle Video Analytics Assessment", "urls": [ "https://sparkle.example/video_analytics_2023_audit.pdf" ], "date": "2024-01-15" } } } Appendix B. JSON Schema For audits.json The following JSON Schema (see [I-D.bhutton-json-schema-01]) can be used to validate an audits.json summary file. Gillmor & Gerchick Expires 7 June 2026 [Page 7] Internet-Draft audits.json December 2025 { "$schema": "https://json-schema.org/draft/2020-12/schema", "$id": "http://ietf.org/json_schemas/audit_schema.json", "title": "Audit Report Summary", "description": "Automated Discovery of Audit Reports", "type": "object", "properties": { "operations": { "type": "array", "items": { "type": "object", "properties": { "urls": { "type": "array", "items": { "type": "string", "format": "uri" }, "uniqueItems": true, "minItems": 1 }, "regs": { "type": "array", "items": { "type": "string", "format": "uri" }, "uniqueItems": true, "minItems": 1 }, "audits": { "type": "array", "items": { "type": "string" }, "description": "property of top-level 'audits' object", "uniqueItems": true, "minItems": 1 } }, "required": [ "urls", "regs", "audits" ] } }, "audits": { Gillmor & Gerchick Expires 7 June 2026 [Page 8] Internet-Draft audits.json December 2025 "type": "object", "additionalProperties": { "type": "object", "properties": { "date": { "type": "string", "format": "date" }, "title": { "type": "string" }, "urls": { "type": "array", "items": { "type": "string", "format": "uri" }, "uniqueItems": true, "minItems": 1 } }, "required": [ "date", "title", "urls" ] } } }, "required": [ "operations", "audits" ] } For example, you can validate this using with [JV]: jv audits-schema.json audits.json Note that JSON Schema cannot represent foreign key constraints, so the fact that each /operations/*/audits must reference a named member of /audits needs to be enforced separately. Appendix C. Examples of Regulatory Regimes Please propose more pointers for this subsection! Gillmor & Gerchick Expires 7 June 2026 [Page 9] Internet-Draft audits.json December 2025 * [NYC-LL144] established requirements for employers in New York City to publish audits of automated decision-making tools used for hiring * Colorado's SB 205 [CO-SB205] requires developers of some artificial intelligence systems to publish reports about the design and deployment of such systems on their websites C.1. Example Regulatory Text A regulation or law that relies on this mechanism to point toward relevant audits can cite it using text similar to the following: Any Covered Organization that has a public-facing website shall publicly point to the Required Audits associated with each position using a widely understood mechanism such as the audits.json specification. The Required Audits associated with this act shall refer to the act as https://springfield.example/citycouncil/acts/2025/114 (note that this text presumes that the regulation has defined the terms "Covered Organization" and "Required Audits") Acknowledgements Document History This section is to be removed before publishing as an RFC. Authors' Addresses Daniel Kahn Gillmor American Civil Liberties Union Email: dkg@fifthhorseman.net Marissa Kumar Gerchick American Civil Liberties Union Email: mgerchick@aclu.org Gillmor & Gerchick Expires 7 June 2026 [Page 10]