DetNet Q. Xiong, Ed. Internet-Draft B. Tan Intended status: Standards Track ZTE Corporation Expires: 11 January 2024 10 July 2023 Traffic Engineering Extensions for Enhanced DetNet draft-xiong-detnet-teas-te-extensions-00 Abstract As per [I-D.ietf-teas-rfc3272bis], DetNet can also be seen as a specialized branch of TE. As it is required to provide enhancements for data plane in scaling networks, this document proposes a set of extensions for traffic engineering to achieve the differentiated DetNet QoS in enhanced DetNet. Status of This Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at https://datatracker.ietf.org/drafts/current/. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." This Internet-Draft will expire on 11 January 2024. Copyright Notice Copyright (c) 2023 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/ license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License. Xiong & Tan Expires 11 January 2024 [Page 1] Internet-Draft Traffic Engineering Extensions for Enhan July 2023 Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 1.1. Requirements Language . . . . . . . . . . . . . . . . . . 4 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 4 3. Differentiated DetNet QoS for Enhanced DetNet . . . . . . . . 4 4. Traffic Engineering for Differentiated DetNet QoS . . . . . . 5 5. Layers Model of DD-TE . . . . . . . . . . . . . . . . . . . . 6 5.1. Deterministic Links . . . . . . . . . . . . . . . . . . . 7 5.1.1. Deterministic Links Model . . . . . . . . . . . . . . 7 5.1.2. Classfication of deterministic Links . . . . . . . . 8 5.1.3. Deterministic Links Resources . . . . . . . . . . . . 9 5.1.4. Deterministic Links Attributes . . . . . . . . . . . 9 5.2. Deterministic Paths . . . . . . . . . . . . . . . . . . . 10 5.3. Deterministic Services . . . . . . . . . . . . . . . . . 10 6. Control Plane Extensions for DD-TE . . . . . . . . . . . . . 10 6.1. Configuration of Queuing Mechanisms . . . . . . . . . . . 11 6.2. Deterministic Resource Collection . . . . . . . . . . . . 11 6.3. Distributed Deterministic Path . . . . . . . . . . . . . 11 6.4. Inter-domain Deterministic Path . . . . . . . . . . . . . 12 6.5. Deterministic Path Computation and Resource Planning . . 12 6.6. Configuration of Flow Mapping . . . . . . . . . . . . . . 12 7. Security Considerations . . . . . . . . . . . . . . . . . . . 12 8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 12 9. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 12 10. References . . . . . . . . . . . . . . . . . . . . . . . . . 12 10.1. Normative References . . . . . . . . . . . . . . . . . . 12 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 17 1. Introduction As defined in [I-D.ietf-teas-rfc3272bis], Traffic Engineering (TE) is mainly focus on the control and optimization of routing and forwarding functions to steer traffic through the network. TE can deal with the issues with performance evaluation and performance optimization of operational IP networks and address the traffic oriented performance requirements including delay, delay variation, packet loss, and throughput while utilizing network resources. According to [RFC8655], Deterministic Networking (DetNet) operates at the IP layer and delivers service which provides extremely low data loss rates and bounded latency within a network domain. The DetNet QoS includes the bounded latency indicating the minimum and maximum end-to-end latency from source to destination and bounded jitter (packet delay variation). Three techniques are used by DetNet to provide these qualities of service including service protection, explicit routes and resource allocation. Xiong & Tan Expires 11 January 2024 [Page 2] Internet-Draft Traffic Engineering Extensions for Enhan July 2023 As per [I-D.ietf-teas-rfc3272bis], DetNet can also be seen as a specialized branch of TE. The DetNet forwarding sub-layer provides resource allocations and explicit routes to guarantee the bounded latency, using existing TE mechanisms such as SR-TE, MPLS-TE and so on. But the enhanced DetNet is required to provide the packet treatment for data plane to achieve the DetNet QoS in large-scale networks. [I-D.ietf-detnet-scaling-requirements] has described the enhanced requirements for DetNet enhanced data plane including the deterministic latency guarantees. [I-D.xiong-detnet-large-scale-enhancements] has proposed the packet treatment which should support new functions such as queuing mechanisms to ensure the deterministic latency. A common data fields can be defined as per [I-D.xiong-detnet-data-fields-edp] and a Deterministic Latency Action (DLA) option has been proposed to carry DetNet-specific metadata. The existing TE mechanisms for resource allocations and explicit routes are not sufficient for enhanced DetNet. For example, the explicit routes should consider the queuing information when selecting and distributing the explicit path. And the resource management should be provisioned including the resource reservations and allocations. The TE mechanisms should consider the queuing-based or time-based resources. Moreover, as per [I-D.ietf-teas-rfc3272bis], DetNet is required to maintain per-flow state information and provide resource reservation for individual flows. As discussed in [I-D.xiong-detnet-enhanced-detnet-gap-analysis], it should deal with large-scale dynamic deterministic flows and large-scale network topology in enhanced DetNet. It may be challenging for network operations in large-scale networks even if the flow aggregation may be supported. As discussed in [I-D.xiong-detnet-large-scale-enhancements], it may provide traffic scheduling instead of the flow scheduling and support the TE control at traffic-aggregate level than the per-flow or flow-aggregate level. Moreover, as per [I-D.xiong-detnet-enhanced-detnet-gap-analysis], multiple deterministic services may demand different set of SLAs and it should define more than one DetNet QoS levels according to different application scenarios. The TE mechanisms in enhanced DetNet should support the the Differentiated DetNet QoS of Multiple Services while utilizing network resources. As per [I-D.ietf-teas-rfc3272bis], DetNet can also be seen as a specialized branch of TE. As it is required to provide enhancements for data plane in scaling networks, this document proposes a set of extensions for traffic engineering to achieve the differentiated DetNet QoS in enhanced DetNet. Xiong & Tan Expires 11 January 2024 [Page 3] Internet-Draft Traffic Engineering Extensions for Enhan July 2023 1.1. Requirements Language The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [RFC2119]. 2. Terminology The terminology is defined as [RFC8655]. DD-QoS: Differentiated DetNet QoS DD-TE: Differentiated DetNet-aware Traffic Engineering DC: DetNet Traffic Class DT: Deterministic Class-Type TRC: Time-based Resources Container 3. Differentiated DetNet QoS for Enhanced DetNet As per [RFC8655], an important goal of the DetNet QoS is the bounded latency including the minimum and maximum end-to-end latency from source to destination, and bounded jitter. From the services requirements in [I-D.xiong-detnet-enhanced-detnet-gap-analysis] section 3.1.1, a scaling network in enhanced DetNet needs to provide the deterministic services for various applications. The deterministic services may demand differentiated SLAs and different bounded latency guarantees. So multiple DetNet QoS levels should be supported according to different application scenarios. Moreover, as per [RFC8938], the aggregation of individual flows may be still challenging for network operations with a large number of deterministic flows and network nodes in large-scale networks. It may provide traffic class scheduling than the flow scheduling. As per [I-D.xiong-detnet-large-scale-enhancements], the enhanced DetNet data plane should support the traffic scheduling based on traffic class and consider the differentiated DetNet QoS for each DetNet flow. This document proposed the DetNet Traffic Class (DC) to indicate the traffic classes of Differentiated DetNet QoS (DD-QoS). The DetNet traffic class may be divided into 4 types: Xiong & Tan Expires 11 January 2024 [Page 4] Internet-Draft Traffic Engineering Extensions for Enhan July 2023 +--------------+-----------+----------+----------+-----------+-----------+ |Differentiated| Bandwidth | Jitter | Delay | Low | Ultra-low | |DetNet QoS | Guarantee | Guarantee| Guarantee| delay and | delay and| | | | | | and jitter| jitter | +--------------+-----------+----------+----------+-----------+-----------+ | DetNet | | | | | | | Traffic |Best-effort| DC-1 | DC-2 | DC-3 | DC-4 | | Class | | | | | | +--------------+-----------+----------+----------+-----------+-----------+ | Applications | Email | Voice | Audio and| AR/VR | Industrial| | Examples | | | Video | | | +--------------+-----------+----------+----------+-----------+-----------+ Figure 1: Traffic class for Differentiated DetNet QoS 4. Traffic Engineering for Differentiated DetNet QoS As per [I-D.ietf-teas-rfc3272bis], DetNet can be viewed as a TE mechanism to achieve DetNet QoS. DetNet performs the per-flow or flow-aggregate scheduling in service sub-layer and uses resource allocations and explicit route mechanisms in forwarding sub-layer. And DetNet can be applied in existing TE data plane mechanisms such as IP, MPLS-TE and SR-TE. As the enhanced DetNet should support the differentiated DetNet QoS, the document proposes a set of extensions for traffic engineering to achieve differentiated DetNet QoS in enhanced DetNet called Differentiated DetNet-aware Traffic Engineering (DD-TE). DD-TE can be used to achieve multiple classes of deterministic services and optimize the resources utilization in scaling networks. The key elements required in DD-TE solution are as follows: 1. Policy As per [I-D.ietf-teas-rfc3272bis], policy allows for the selection of paths (including next hops) based on information beyond basic reachability. The routing policy including bounded latency constraint-based routing can be considered when selecting and distributing the candidate paths. As per [I-D.peng-lsr-flex-algo-deterministic-routing], deterministic routes can be established along the constraint-based paths within a Flex- Algorithm topology. As per [I-D.xiong-pce-detnet-bounded-latency], deterministic paths can be computed in PCE or controller with the deterministic latency constraints. As defined in [I-D.xiong-idr-detnet-flow-mapping], the BGP flowspec can be used to apply the DetNet flows mapping policy. Xiong & Tan Expires 11 January 2024 [Page 5] Internet-Draft Traffic Engineering Extensions for Enhan July 2023 2. Path steering As per [I-D.ietf-teas-rfc3272bis], path steering is the ability to forward packets using more information than just knowledge of the next hop. The per-flow or flow-aggregate scheduling is not applicable since it requires a large amount of control signaling to establish and maintain DetNet flows when it will be large-scale dynamic deterministic flows and large-scale network topology in scaling networks of enhanced DetNet. As discussed in [I-D.xiong-detnet-large-scale-enhancements], it may provide traffic scheduling in enhanced DetNet data plane and provide 4 DetNet traffic classes for Differentiated DetNet QoS. So the DD-TE mechanism should use the traffic class information to forward packets at traffic- aggregate level instead of the per-flow or flow-aggregate level. As per [I-D.xiong-detnet-large-scale-enhancements], in scaling networks of enhanced DetNet data plane, the enhanced QoS-related functions and metadata has been proposed to guarantee the bounded latency such as the queuing-based mechanisms and metadata. The deterministic latency information may be provided to forward packets for path steering. DD-TE can be applied in TE data plane such as IPv6 [I-D.xiong-detnet-6man-queuing-option], MPLS [I-D.sx-detnet-mpls-queue] and SRv6 [I-D.xiong-detnet-spring-srh-extensions]. 3. Resource management As per [I-D.xiong-detnet-large-scale-enhancements], the resource management should support the time-based resource-aware control and forwarding including resource reservations and allocations. The time-based resource should cover the queuing and scheduling mechanisms based on the capability of end-to-end delay, jitter and loss. To guarantee the time-based resource, the resource control in layers model section 5 may be provided to avoid the conflict between DetNet flows to achieve differentiated DetNet QoS and high resources utilization. 5. Layers Model of DD-TE The resource control of DD-TE is important to regulate the traffic, deliver different levels of services and alleviate congestion issues to guarantee the bounded latency. It needs to resolve competition for network resources between traffic flows belonging to the same service class (intra-class contention resolution) and traffic flows belonging to different classes (inter-class contention resolution). Xiong & Tan Expires 11 January 2024 [Page 6] Internet-Draft Traffic Engineering Extensions for Enhan July 2023 This document proposes the layers model for enhanced DetNet control plane to configure the deterministic services to achieve differentiated DetNet QoS. The DetNet TE domains in control plane can be divided into three layers including deterministic links, deterministic paths and deterministic services as shown in Figure 1. Deterministic Services:|~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~>| Deterministic Paths: +.............................................>+ +.............................................>+ Deterministic Links: O---------------->O O--------------->O O---------------->O O--------------->O O---------------->O O--------------->O +-----+ +-----+ +-----+ DetNet Domain: | A |--------------| B |--------------| C | +--+--+ +--+--+ +--+--+ Figure 2: The DD-TE Layers Model The Layers Model of DD-TE has the following characteristics: * The deterministic links are designed to resolve resources competition among different traffic classes and provide deterministic forwarding capabilities at multiple levels. * The deterministic paths are designed to resolve resources competition among different paths within the same traffic class. * The deterministic services are designed to resolve resources competition among different flows on the same path. 5.1. Deterministic Links 5.1.1. Deterministic Links Model This document proposes the deterministic links to provide a one- dimensional deterministic metric to guarantee the deterministic forwarding capabilities at different levels. The deterministic links can shield the differences from underlying forwarding and queuing mechanisms. Xiong & Tan Expires 11 January 2024 [Page 7] Internet-Draft Traffic Engineering Extensions for Enhan July 2023 DetNet transit node A DetNet transit node B +-------------------------+ +------------------------+ | Queuing | | Queuing | | Regulator subsystem | | Regulator subsystem | | +-+-+-+-+ +-+-+-+-+ | | +-+-+-+-+ +-+-+-+-+ | -->+ | | | | | | | | | + +------------>+ | | | | | | | | | + +---> | +-+-+-+-+ +-+-+-+-+ | | +-+-+-+-+ +-+-+-+-+ | | | | | +-------------------------+ +------------------------+ |-->|------->|------->|-->|------------>|-->|------->|------>|-->|-->| 2,3 4 5 6 1 2,3 4 5 6 1 2,3 |---- Deterministic Link Delay ---->| Deterministic Links Deterministic Node | A |---------------------------------->| B |----------------------->| Figure 3: Deterministic Links Model As per [RFC9320], 6 types of delays are defined in timing Model of DetNet. And the DetNet domain can also be modeling as deterministic links and nodes as shown in Figure 2. The deterministic node delay is constant and the deterministic link delay is variable within bounded latency. The end-to-end bounded latency depends on the sum of the deterministic link delay. * Deterministic Link Delay = Regulation delay + Queuing subsystem delay + Output delay + Link delay + Frame preemption delay * Deterministic Node Delay = Processing delay 5.1.2. Classfication of deterministic Links There are a number of deterministic links between deterministic nodes. And each deterministic link provides different level of deterministic forwarding capabilities indicated by Deterministic Class-Type (DT). Deterministic Class-Type (DT): indicate the set of Traffic Trunks crossing a deterministic link that is governed by a specific set of bounded latency constraints. DT is used for the purposes of deterministic link resource planning, reservation and allocation, deterministic link resource constraint-based routing and admission control. A given Traffic Trunk belongs to the same DT on all links. For example, three deterministic links with guaranteed jitter are supported between the Node A and Node B as following shown. Xiong & Tan Expires 11 January 2024 [Page 8] Internet-Draft Traffic Engineering Extensions for Enhan July 2023 deterministic link 1, DT=1 (Jitter Guarantee), 10us. deterministic link 2, DT=2 (Jitter Guarantee), 20us. deterministic link 3, DT=3 (Jitter Guarantee), 30us. 5.1.3. Deterministic Links Resources The traditional resource reservation method only considers the bandwidth availability of the BE (Best Effort) flow, which means that the reserved bandwidth meets the peak information rate (PIR) of the business flow at the macro level. As per [I-D.ietf-detnet-scaling-requirements], the enhanced DetNet need to support multiple queuing mechanisms to provide deterministic latency. For such scheduling mechanisms, even the bandwidth resources meet the transmission requirements at the macro level, there may not be enough resources in a specific timeslot, cycle or authorization time zone, so bounded delay and jitter cannot be guaranteed. So it is required to provide provisioning of fine-grained reservation for time-based resources. Time-based Resources Container (TRC): the entity which is used for deterministic link to provide the time-based resources with deterministic capabilities by resolving resource conflicts between different levels. The container indicates the transmitting bits per scheduling timeslot and contains the corresponding scheduling resources reserved to guarantee the capability of deterministic link and it may include queuing, buffer or bandwidth. 5.1.4. Deterministic Links Attributes The deterministic link has the following attributes: * Link ID: an identifier that uniquely identifies a deterministic link within DetNet domain. * DT type: indicate the level of deterministic link. * MaxReservedBandwidth: the maximum bandwidth of the deterministic link. * TRC Parameters: carry the TRC ID and the capacity of the time- based resources which is reserved for the link. Xiong & Tan Expires 11 January 2024 [Page 9] Internet-Draft Traffic Engineering Extensions for Enhan July 2023 5.2. Deterministic Paths When DetNet services with different SLA requirements requested to transmit, one or more deterministic paths may be calculated based on the deterministic links. The deterministic paths may be co-existed with the same DT and the time-based resources should be planned when each path is established. The deterministic paths has the following attributes: * Path ID: an identifier that uniquely identifies a deterministic path within DetNet domain. * DT type: indicate the level of deterministic link. * Source and Destination Nodes: indicate the head and end address of requested service. * Path Parameters: used to describe the result of deterministic paths including the seleted deterministic links and other information guiding deterministic forwarding behavior for each hop such as mapping function. * TRC Parameters: carry the TRC ID and the time-based resources which is planned for the path. 5.3. Deterministic Services The deterministic services may be configured to map the DetNet flows to the corresponding path. The deterministic services has the following attributes: * Service ID: an identifier that uniquely identifies a deterministic services on a deterministic path. * Service Level : the indicator of the service requirement. * Path ID: a deterministic path to deliver the service. * Policy: indicate the admission control and traffic policy for the DetNet flows within the same traffic class on the path. * TRC Parameters: carry the TRC ID and the time-based resources which is allocated for the traffic class. 6. Control Plane Extensions for DD-TE Xiong & Tan Expires 11 January 2024 [Page 10] Internet-Draft Traffic Engineering Extensions for Enhan July 2023 +----------+ 3-Service Request-->|Controller|-->4-Deterministic Path Planning +---+--+---+ | ^ 2-Deterministic Links Resource Report | | | | | | 5-Path Distribution V | ................................................. . . . 1-Resoure Collection . . . Flow . +---+ +---+ +---+ . +---> . | A |------------| B |------------| C | . | . +---+ +---+ +---+ . | . DetNet Domain . | . . | . 6-Path Establishment and Resource Allocation . | . . | ................................................. | |-->7-Admision Control and Traffic Policy of Deterministic service Figure 4: The Control Plane for DD-TE 6.1. Configuration of Queuing Mechanisms As described in [I-D.ietf-detnet-scaling-requirements], it is required to support the configuration of multiple queuing mechanisms. Different queuing mechanisms may be supported at different levels of latency, jitter and other guarantees. The enhancement for controller plane should be provided such as configuration information model as defined in [I-D.guo-detnet-vpfc-planning]. 6.2. Deterministic Resource Collection And the type of queuing mechanism and the related queuing parameters should be advertised and configured. For example, the deterministic links with queuing resource could be distributed by IGP protocol as per [I-D.peng-lsr-deterministic-traffic-engineering]. 6.3. Distributed Deterministic Path The deterministic routes may be loose routes in distributed scenarios. It is required to support the distributed deterministic routes which are established by distributed protocols such as IGP as defined in [I-D.peng-lsr-flex-algo-deterministic-routing]. Xiong & Tan Expires 11 January 2024 [Page 11] Internet-Draft Traffic Engineering Extensions for Enhan July 2023 6.4. Inter-domain Deterministic Path In scaling deterministic networks, it may across multiple network domains, it is required to support the inter-domain deterministic routes to achieve the end-to-end latency, bounded jitter. And the deadline of latency and jitter of each domain and segment should be determined and controlled. The inter-domain mechanism MUST be considered at the boundary nodes such as BGP configurations defined in [I-D.peng-idr-bgp-metric-credit] and PCEP solution [I-D.bernardos-detnet-multidomain]. 6.5. Deterministic Path Computation and Resource Planning As defined in [I-D.xiong-pce-detnet-bounded-latency], the deterministic latency constraints can be carried in PCEP extensions and the end-to-end deterministic path computation should be achieved for DetNet service. 6.6. Configuration of Flow Mapping As defined in [I-D.xiong-idr-detnet-flow-mapping], the BGP flowspec can be used for the filtering of the packets that match the DetNet networks and the mapping between TSN streams and DetNet flows in the control plane. 7. Security Considerations TBA 8. IANA Considerations TBA 9. Acknowledgements TBA 10. References 10.1. Normative References [I-D.bernardos-detnet-multidomain] Bernardos, C. J. and A. Mourad, "DETNET multidomain extensions", Work in Progress, Internet-Draft, draft- bernardos-detnet-multidomain-01, 12 January 2023, . Xiong & Tan Expires 11 January 2024 [Page 12] Internet-Draft Traffic Engineering Extensions for Enhan July 2023 [I-D.dang-queuing-with-multiple-cyclic-buffers] Liu, B. and J. Dang, "A Queuing Mechanism with Multiple Cyclic Buffers", Work in Progress, Internet-Draft, draft- dang-queuing-with-multiple-cyclic-buffers-00, 22 February 2021, . [I-D.guo-detnet-vpfc-planning] Guo, D., Wen, G., Yao, K., Xiong, Q., and G. Peng, "Deterministic Networking (DetNet) Controller Plane - VPFC Planning Information Model Based on VPFP in Scaling Deterministic Networks", Work in Progress, Internet-Draft, draft-guo-detnet-vpfc-planning-02, 9 July 2023, . [I-D.ietf-detnet-controller-plane-framework] Malis, A. G., Geng, X., Chen, M., Qin, F., Varga, B., and C. J. Bernardos, "Deterministic Networking (DetNet) Controller Plane Framework", Work in Progress, Internet- Draft, draft-ietf-detnet-controller-plane-framework-04, 13 March 2023, . [I-D.ietf-detnet-scaling-requirements] Liu, P., Li, Y., Eckert, T. T., Xiong, Q., Ryoo, J., zhushiyin, and X. Geng, "Requirements for Scaling Deterministic Networks", Work in Progress, Internet-Draft, draft-ietf-detnet-scaling-requirements-03, 7 July 2023, . [I-D.ietf-teas-rfc3272bis] Farrel, A., "Overview and Principles of Internet Traffic Engineering", Work in Progress, Internet-Draft, draft- ietf-teas-rfc3272bis-24, 15 June 2023, . [I-D.peng-6man-deadline-option] Peng, S., Tan, B., and P. Liu, "Deadline Option", Work in Progress, Internet-Draft, draft-peng-6man-deadline-option- 01, 11 July 2022, . [I-D.peng-detnet-deadline-based-forwarding] Peng, S., Du, Z., Basu, K., cheng, and D. Yang, "Deadline Based Deterministic Forwarding", Work in Progress, Xiong & Tan Expires 11 January 2024 [Page 13] Internet-Draft Traffic Engineering Extensions for Enhan July 2023 Internet-Draft, draft-peng-detnet-deadline-based- forwarding-06, 7 July 2023, . [I-D.peng-idr-bgp-metric-credit] Peng, S. and B. Tan, "BGP Metric Credit Based Routing", Work in Progress, Internet-Draft, draft-peng-idr-bgp- metric-credit-00, 28 December 2021, . [I-D.peng-lsr-deterministic-traffic-engineering] Peng, S., "IGP Extensions for Deterministic Traffic Engineering", Work in Progress, Internet-Draft, draft- peng-lsr-deterministic-traffic-engineering-01, 4 July 2023, . [I-D.peng-lsr-flex-algo-deterministic-routing] Peng, S. and T. Li, "IGP Flexible Algorithm with Deterministic Routing", Work in Progress, Internet-Draft, draft-peng-lsr-flex-algo-deterministic-routing-03, 24 August 2022, . [I-D.sx-detnet-mpls-queue] Song, X., Xiong, Q., and R. Gandhi, "MPLS Sub-Stack Encapsulation for Deterministic Latency Action", Work in Progress, Internet-Draft, draft-sx-detnet-mpls-queue-06, 26 April 2023, . [I-D.xiong-detnet-6man-queuing-option] Xiong, Q. and J. Zhao, "IPv6 Option for DetNet Data Fields", Work in Progress, Internet-Draft, draft-xiong- detnet-6man-queuing-option-04, 10 March 2023, . [I-D.xiong-detnet-data-fields-edp] Xiong, Q. and D. Yang, "Data Fields for DetNet Enhanced Data Plane", Work in Progress, Internet-Draft, draft- xiong-detnet-data-fields-edp-00, 10 March 2023, . Xiong & Tan Expires 11 January 2024 [Page 14] Internet-Draft Traffic Engineering Extensions for Enhan July 2023 [I-D.xiong-detnet-enhanced-detnet-gap-analysis] Xiong, Q., "Gap Analysis for Enhanced DetNet Data Plane", Work in Progress, Internet-Draft, draft-xiong-detnet- enhanced-detnet-gap-analysis-01, 6 July 2023, . [I-D.xiong-detnet-large-scale-enhancements] Xiong, Q., Du, Z., Zhao, J., and D. Yang, "Enhanced DetNet Data Plane (EDP) Framework for Scaling Deterministic Networks", Work in Progress, Internet-Draft, draft-xiong- detnet-large-scale-enhancements-02, 13 March 2023, . [I-D.xiong-detnet-spring-srh-extensions] Xiong, Q., Wu, H., and D. Yang, "Segment Routing Header Extensions for DetNet Data Fields", Work in Progress, Internet-Draft, draft-xiong-detnet-spring-srh-extensions- 00, 10 March 2023, . [I-D.xiong-idr-detnet-flow-mapping] Xiong, Q., Wu, H., Zhao, J., and D. Yang, "BGP Flow Specification for DetNet and TSN Flow Mapping", Work in Progress, Internet-Draft, draft-xiong-idr-detnet-flow- mapping-04, 12 March 2023, . [I-D.xiong-pce-detnet-bounded-latency] Xiong, Q., Liu, P., and R. Gandhi, "PCEP Extension for DetNet Bounded Latency", Work in Progress, Internet-Draft, draft-xiong-pce-detnet-bounded-latency-03, 8 June 2023, . [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, . [RFC4655] Farrel, A., Vasseur, J.-P., and J. Ash, "A Path Computation Element (PCE)-Based Architecture", RFC 4655, DOI 10.17487/RFC4655, August 2006, . Xiong & Tan Expires 11 January 2024 [Page 15] Internet-Draft Traffic Engineering Extensions for Enhan July 2023 [RFC4915] Psenak, P., Mirtorabi, S., Roy, A., Nguyen, L., and P. Pillay-Esnault, "Multi-Topology (MT) Routing in OSPF", RFC 4915, DOI 10.17487/RFC4915, June 2007, . [RFC5120] Przygienda, T., Shen, N., and N. Sheth, "M-ISIS: Multi Topology (MT) Routing in Intermediate System to Intermediate Systems (IS-ISs)", RFC 5120, DOI 10.17487/RFC5120, February 2008, . [RFC5440] Vasseur, JP., Ed. and JL. Le Roux, Ed., "Path Computation Element (PCE) Communication Protocol (PCEP)", RFC 5440, DOI 10.17487/RFC5440, March 2009, . [RFC6549] Lindem, A., Roy, A., and S. Mirtorabi, "OSPFv2 Multi- Instance Extensions", RFC 6549, DOI 10.17487/RFC6549, March 2012, . [RFC7752] Gredler, H., Ed., Medved, J., Previdi, S., Farrel, A., and S. Ray, "North-Bound Distribution of Link-State and Traffic Engineering (TE) Information Using BGP", RFC 7752, DOI 10.17487/RFC7752, March 2016, . [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, May 2017, . [RFC8231] Crabbe, E., Minei, I., Medved, J., and R. Varga, "Path Computation Element Communication Protocol (PCEP) Extensions for Stateful PCE", RFC 8231, DOI 10.17487/RFC8231, September 2017, . [RFC8233] Dhody, D., Wu, Q., Manral, V., Ali, Z., and K. Kumaki, "Extensions to the Path Computation Element Communication Protocol (PCEP) to Compute Service-Aware Label Switched Paths (LSPs)", RFC 8233, DOI 10.17487/RFC8233, September 2017, . [RFC8655] Finn, N., Thubert, P., Varga, B., and J. Farkas, "Deterministic Networking Architecture", RFC 8655, DOI 10.17487/RFC8655, October 2019, . Xiong & Tan Expires 11 January 2024 [Page 16] Internet-Draft Traffic Engineering Extensions for Enhan July 2023 [RFC8664] Sivabalan, S., Filsfils, C., Tantsura, J., Henderickx, W., and J. Hardwick, "Path Computation Element Communication Protocol (PCEP) Extensions for Segment Routing", RFC 8664, DOI 10.17487/RFC8664, December 2019, . [RFC8938] Varga, B., Ed., Farkas, J., Berger, L., Malis, A., and S. Bryant, "Deterministic Networking (DetNet) Data Plane Framework", RFC 8938, DOI 10.17487/RFC8938, November 2020, . [RFC9320] Finn, N., Le Boudec, J.-Y., Mohammadpour, E., Zhang, J., and B. Varga, "Deterministic Networking (DetNet) Bounded Latency", RFC 9320, DOI 10.17487/RFC9320, November 2022, . [RFC9357] Xiong, Q., "Label Switched Path (LSP) Object Flag Extension for Stateful PCE", RFC 9357, DOI 10.17487/RFC9357, February 2023, . Authors' Addresses Quan Xiong (editor) ZTE Corporation China Email: xiong.quan@zte.com.cn Bin Tan ZTE Corporation China Email: tan.bin@zte.com.cn Xiong & Tan Expires 11 January 2024 [Page 17]