ALTO Working Group D. Lachos Internet-Draft I. Poese Intended status: Standards Track Benocs Expires: 11 January 2024 M. Lassnig CERN A. Gu Y. Yang Yale University J. Ros Giralt Qualcomm 10 July 2023 ALTO Multi-Domain Use Cases and Services draft-yang-alto-multi-domain-03 Abstract Application-Layer Traffic Optimization (ALTO) provides means for network applications to obtain network information. Although ALTO is inherently multi-domain, in that the ALTO server representing the network and the ALTO client requesting the network information belong to different trust domains, there are more general cases where the path from the source and the destination spans multiple autonomous networks, which we call multi-domain settings. This document first gives three multi-domain use cases, and the challenges to address the challenges. It then gives a brief update on the implementation solutions that we explored to address the challenges. Requirements Language The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here. Status of This Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at https://datatracker.ietf.org/drafts/current/. Lachos, et al. Expires 11 January 2024 [Page 1] Internet-Draft ALTO Multi-Domain July 2023 Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." This Internet-Draft will expire on 11 January 2024. Copyright Notice Copyright (c) 2023 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/ license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License. Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 2. Multi-domain Use Cases . . . . . . . . . . . . . . . . . . . 3 2.1. Multi-domain Path Distance/Ranking for Rucio . . . . . . 3 2.2. Multi-domain Path -> Link Mapping for FTS . . . . . . . . 5 2.3. Multi-domain Resource Discovery for NOTED/SENSE . . . . . 6 3. Multi-domain Challenges . . . . . . . . . . . . . . . . . . . 6 3.1. Challenge: Distributed Information . . . . . . . . . . . 6 3.2. Challenge: Partial Deployment . . . . . . . . . . . . . . 7 4. Solution Space Exploration . . . . . . . . . . . . . . . . . 7 4.1. Solution Space to Obtain Forwarding Information Bases (FIBs) . . . . . . . . . . . . . . . . . . . . . . . . . 7 4.2. FIB-Compute Based Design . . . . . . . . . . . . . . . . 8 4.3. FIB-Apply Based Design . . . . . . . . . . . . . . . . . 8 4.4. Multi-Domain Cascading ALTO . . . . . . . . . . . . . . . 8 4.5. General-Path Model Supporting Partial Deployment . . . . 9 5. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 10 6. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 10 7. References . . . . . . . . . . . . . . . . . . . . . . . . . 10 7.1. Normative References . . . . . . . . . . . . . . . . . . 10 7.2. Informative References . . . . . . . . . . . . . . . . . 11 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 11 Lachos, et al. Expires 11 January 2024 [Page 2] Internet-Draft ALTO Multi-Domain July 2023 1. Introduction Application-Layer Traffic Optimization (ALTO) provides means for network applications to obtain network information. For example, the Endpoint Cost Service (ECS) and the Cost Map Service (CMS) defined by ALTO in [RFC7285] can provide the network-path properties (e.g., routing costs, or relative ranking) of data transmissions from a set of source network locations to a set of destination network locations. The Path Vector Service [RFC9275] allows ALTO to provide bandwidth availability (bottlenecks) for a given set of flows defined by a set of source-destination pairs. Although such services can provide values when the network path from a given source to a given destination spans only a single network, called a single-domain setting, provides these services, there are many use cases where the path may span multiple networks, called multi-domains settings. [RFC7971] states that "The ALTO protocol is designed for use cases where the ALTO server and client can be located in different organizations or trust domains. ALTO is inherently designed for use in multi-domain environments. Most importantly, ALTO is designed to enable deployments in which the ALTO server and the ALTO client are not located within the same administrative domain.” This document first specifies three multi-domain use cases. It then discusses the two challenges when deploying ALTO in multi-domain settings. At the end, this document provides initial designs based on current implementation experiences to start the design conversation. 2. Multi-domain Use Cases To be concrete, this document uses three use cases from the context of data-intensive sciences for CERN/WLCG. 2.1. Multi-domain Path Distance/Ranking for Rucio The data orchestration system of CERN/WLCG is Rucio, which selects a source for a given downloading client when multiple sources provide the dataset. Rucio also conducts destination selection, when multiple destinations can be the target locations for replication. Lachos, et al. Expires 11 January 2024 [Page 3] Internet-Draft ALTO Multi-Domain July 2023 The main mechanism of source/destination selection in Rucio is distance, which is a perfect match for ALTO cost services; see appendix for a review of the full source/destination mecahnisms of Rucio. Specifically, consider the following ECS query realizing ALTO ECS for Rucio to do destination selection. The source is located at CERN and the destination candidates are at multiple locations of the LHCONE network (AGLT, BNL, and KIT, for example). POST /endpointcost/lookup HTTP/1.1 Host: alto.example.com Content-Length: 248 Content-Type: application/alto-endpointcostparams+json Accept: application/alto-endpointcost+json,application/alto-error+json { "cost-type": {"cost-mode" : "numerical", "cost-metric" : "routingcost"}, "endpoints" : { "srcs": [ "ipv4:188.184.9.234" ], "dsts": [ "ipv4:192.41.231.81", "ipv4:104.18.24.74", "ipv4:141.3.128.6" ] } } HTTP/1.1 200 OK Content-Length: 274 Content-Type: application/alto-endpointcost+json { "meta" : { "cost-type": {"cost-mode" : "numerical", "cost-metric" : "routingcost" } }, "endpoint-cost-map" : { "ipv4:188.184.9.234": { "ipv4:192.41.231.81" : 20, "ipv4:104.18.24.74" : 30, "ipv4:141.3.128.6" : 10 } } } Lachos, et al. Expires 11 January 2024 [Page 4] Internet-Draft ALTO Multi-Domain July 2023 The use case provides an example of a multi-domain setting, because the source and the destinations are located at different networks and the network paths span multiple autonomous networks. For example, the path from CERN (188.184.9.234) to AGLT2 (192.41.231.81) spans CERN, ESNet and U. Michigan to reach the destination. Figure 1 below illustrates a network path from src to dst spanning four networks. AS S AS A AS B AS D +-------------+se1 +---------+ +-----+ +------------+ | src --|-----|ai1 ae1|---| |---|di1 dst | |+--+ --/ | | | | | | +--+ | || | --/ | | | | | | | | | |+--+ \ |se2 | | | | | +--+ | | \__ |_____|ai2 ae2|---| |---|di2 | +-------------+ +---------+ +-----+ +------------+ Figure 1. Multi-domain Network. 2.2. Multi-domain Path -> Link Mapping for FTS The data transport scheduling system of CERN/WLCG is FTS, which schedules given transfers, where a transfer is compromised of a fixed dataset, a source and a destination (for example, chosen by Rucio). One main function of FTS is to satisfy operator resource constraints. As an example, FTS may schedule concurrent transfers from CERN to CalTech and KIT, under the constraint that the total transmission rate for a given link should be within a certain threshold. Consider Figure 1. Assume S is CERN and D is CalTech. The network path from CERN to CalTech uses the top inter-AS link (se1->ai1), and the network path from CERN to KIT uses the lower inter-AS link (se2->ai2). For FTS to know how much demand it is placing on each one of these two links so that it can stay within the constraint of each link, FTS needs to know the network paths, and the paths span multiple autonomous networks. Lachos, et al. Expires 11 January 2024 [Page 5] Internet-Draft ALTO Multi-Domain July 2023 2.3. Multi-domain Resource Discovery for NOTED/SENSE An emerging capacity for LHCONE/WLCG is the ability to create new paths to obtain additional capacity when there is high demand from a given source storage element (SE) to a destination element (SE). In particular, the NOTED project monitors the backlog at Rucio/FTS for each source SE, destination SE pair, and may decide to create a path if one pair has a large backlog. To identify available networking capacity, ALTO Path Vector Service (PVS) is an ideal service. However, as we see from Figure 1, the setting can be a multi-domain setting. 3. Multi-domain Challenges Although implementing ECS/CMS/PVS is relatively straightforward in a single domain, there are challenges implementing these services in multi-domain settings. 3.1. Challenge: Distributed Information Consider Figure 1, to compute the total routing cost of the path from the src to the dst. The routing costs will consist of multiple segments spanning four autonomous networks: (1) from src to the link connecting the egress of AS S (se1) and the ingress of AS A (ia1); (2) from the ingress of A to the ingress of B; (3) from the ingress of B to the ingress of D; ( 4) from the ingress of D to dst. One may think that BGP collects information from multiple autonomous networks through back propagation from the destination, and hence includes information for all four segments. But BGP information is distributed, coarse-grained, and incomplete. Source: The BGP router at AS S knows that the path from src to dst consists of the AS-PATH [S A B D]. Combining BGP and intradomain routing, AS S will also know which one of the two egress routers (se1, se2) that it will use to forward traffic to dst. However, AS S does not know more details downstream: for example, it does not know whether the packet will use ae1 or ae2 as the egress router at AS A to enter AS B; neither does it know the internal routing inside AS A. Hence, an ALTO server provided by AS S cannot provide all of the information for the example ECS query. Lachos, et al. Expires 11 January 2024 [Page 6] Internet-Draft ALTO Multi-Domain July 2023 Non-Source AS: A non-source AS knows the AS-PATH starting from itself to dst. But it may not know the ingress point. For example, AS A does not know whether the packet will come in from ai1 or ai2. Hence, an ALTO server provided by AS A may consider the example ECS query as an ambiguous query (because it gives only source (src) and destination (dst), but it does not in general know the ingress point). 3.2. Challenge: Partial Deployment It is possible to design protocol extensions to collect the aforementioned distributed information to provide complete information (see below), but one challenge is that the deployment may be only incremental and hence is partially deployed during the process. 4. Solution Space Exploration During the process of integrating ALTO to support the Rucio and FTS use cases, multiple potential solutions are implemented. Below we discuss them briefly to motivate discussions. 4.1. Solution Space to Obtain Forwarding Information Bases (FIBs) Obtaining FIBs is a basic building block of implementing ALTO route- related visibility services. At a high-level, FIBs have the following life cycle: FIB-Configure/input: Consider a network as a distributed system. Then the input to the distributed system is the input to the components: the routers. The input to each router is its configuration, and external input such as BGP routes from peers outside the network. FIB-Compute: The distributed system computes the FIBs using a distributed protocol (or configured by a logically centralized control plane). During the computing process, the vendor specific configuration may be turned into standard format such as IGP information model. FIB-Apply: The FIBs are computed and then applied. When FIBs are applied to data packets, the application may be observed by NetFlow/sFlow or similar capturing capabilities; it may also be applied to control mechanisms such as traceoute, which can be observed. Lachos, et al. Expires 11 January 2024 [Page 7] Internet-Draft ALTO Multi-Domain July 2023 4.2. FIB-Compute Based Design This is a type of solution that makes it possible to extends FIB- Compute to compute all needed network information at a single autonomous network, addressing the distributed information challenge. Assume that use an ALTO server at the source network to abstract and expose the information. One natural candidate is to modify the routing control plane itself: BGP extensions, which can be extended to collect needed information and propagate upstream. For example, when a BGP router at AS A (e.g., ai1) propagates BGP info to its peer at AS S (se1), it includes not only the AS-PATH [A, B, D], but also additional information so that the upstream can construct the complete path cost (distance) metrics. The upside of this design is that it integrates with routing systems and hence may even extend routing capabilities. However, routing protocol extensions can be complex in deployment. Further, it provides a different trust model: the original ALTO model is a star trust model, with the application (e.g., Rucio/FTS) at the hub and each AS needs to trust the application. The BGP extension model requires the trust of peers and recursive peers (BGP community may be used to impose policies). 4.3. FIB-Apply Based Design This is a type of solution that allows data path to collect control plane information. For example, a traceroute based system called PerfSonar is widely deployed in our setting. It is also natural to use NetFlow/sFlow to identify ingress points. Such a system can collect other network information such as delay and loss naturally as measurements. However, this solution type can have many issues. For example, traceoute has issues including: anonymous routers, uncertain router IP resulting in node aliasing; load balancing routing resulting in link aliasing. 4.4. Multi-Domain Cascading ALTO This is a model that is presented by Ingmar Poese as Cascading ALTO at IETF 116. For Cascading ALTO by Poese, please see his IETF 116 slides. In the ALTO base model, a network is a container, which we call a big switch, with endpoints attached to the big switch. In the multi- domain model, each network (represented by an ALTO server) has a set of ingress points (in-1 to in-m) and a set of egress points (e-1 to e-n). An endpoint belonging to the network will be attached to an Lachos, et al. Expires 11 January 2024 [Page 8] Internet-Draft ALTO Multi-Domain July 2023 ingress point and an egress point. Hence, a single-domain ALTO query will specify ingress and egress directly attached to an ingress point and an egress point. A source network, to a destination that is not in the same network, however, will only return the egress point; a destination network, when the source is from a different network, will need an ingress point. A general transit network will need an ingress point and return egress point. For consistency, the egress point must be a valid ingress point, represented by a unique address, of the peer. in-1 +-------------+ e-1 ----| |---- | | ----| |---- | | ----| |---- | | ----| |---- in-m +-------------+ e-n ALTO Server Multi-domain Query Model: Each ECS query, if the src is not in the home domain of the ALTO server, should include an ingress point, where the ingress point is returned by the ALTO server of the previous domain. If the domain of the ALTO server is not the home domain of the destination, the ALTO server should return the egress point of the home domain and the ingress of the network domain. As an optional feature, the query should allow indication of iterative or recursive queries. To support incremental deployment, an ALTO server may respond to a query without specifying an ingress point and the source is not in the domain of the ALTO server. In this case, the ALTO server will return the results from each potential ingress point. For each ingress point indicated, the server indicates information of the previous hop (e.g., peer AS number and potential address). 4.5. General-Path Model Supporting Partial Deployment Complementing the cascading model, we introduced a generic-path model at ALTO clients so that they can use the acquired information to gradually refine network information. In particular, it allows the path from a src to a dst to be a directed acyclic graph, with the following components: Lachos, et al. Expires 11 January 2024 [Page 9] Internet-Draft ALTO Multi-Domain July 2023 A set of nodes, where each node has both a type and attributes, where the type can be (1) host: such as src/dst, with attributes such as IP address; (2) AS: which is a group of nodes, i.e., subgraph, with attributes including ASN; (3) router, with subtypes such as BGP- router, with attributes such as IP address. A set of links, where each link has a head and a tail; hence the types of links will be the unique combinations of head-type x tail type. A link can have its attributes as well. Now, some examples of this representation in our deployment use case: For the geo-distance ALTO cost derived from geo-ip: the src is a host and the dst is also a host, and the metric is the geo distance. For CERN looking glass ALTO server, from a src host in CERN to a dst host in another network, say KIT, the src is a host, with two links, one for each of the two looking glass BGP routers from cern; each of these BGP routers links to its BGP peer, and each such BGP peer links to the next AS, in the AS-PATH exposed by CERN. 5. IANA Considerations Some of the solutions will need IANA registrations. 6. Acknowledgments The authors of this document would also like to thank many for the reviews and comments. 7. References 7.1. Normative References [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, . [RFC7285] Alimi, R., Ed., Penno, R., Ed., Yang, Y., Ed., Kiesel, S., Previdi, S., Roome, W., Shalunov, S., and R. Woundy, "Application-Layer Traffic Optimization (ALTO) Protocol", RFC 7285, DOI 10.17487/RFC7285, September 2014, . [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, May 2017, . Lachos, et al. Expires 11 January 2024 [Page 10] Internet-Draft ALTO Multi-Domain July 2023 7.2. Informative References [RFC7971] Stiemerling, M., Kiesel, S., Scharf, M., Seidel, H., and S. Previdi, "Application-Layer Traffic Optimization (ALTO) Deployment Considerations", RFC 7971, DOI 10.17487/RFC7971, October 2016, . [RFC9275] Gao, K., Lee, Y., Randriamasy, S., Yang, Y., and J. Zhang, "An Extension for Application-Layer Traffic Optimization (ALTO): Path Vector", RFC 7971, DOI 10.17487/RFC7971, September 2022, . Authors' Addresses Danny Lachos Benocs Berlin Germany Email: dlachos@benocs.com Ingmar Poese Benocs Berlin Germany Email: ingmar@benocs.com Mario Lassnig CERN CH-1211 Geneva 23 Switzerland Email: mario.lassnig@cern.ch Annie Gu Yale University 51 Prospect St New Haven, CT 06520 United States of America Email: annie.gu@yale.edu Lachos, et al. Expires 11 January 2024 [Page 11] Internet-Draft ALTO Multi-Domain July 2023 Y. Richard Yang Yale University 51 Prospect St New Haven, CT 06520 United States of America Email: yry@cs.yale.edu Jordi Ros Giralt Qualcomm Madrid Spain Email: jros@qti.qualcomm.com Lachos, et al. Expires 11 January 2024 [Page 12]