ATM Forum/96-0354 PROJECT: ATM Forum Technical Committee ***************************************************************************** SOURCE: Ross Callon Bay Networks 3 Federal Street Billerica, MA 01821 Phone: +1-508-436-3936 Email: rcallon@baynetworks.com Jason Jeffords Cabletron Technology Drive Durham, NH 03824 Phone: +1-603-337-7019 Email: jeffords@ctron.com John Drake Fore Systems, Inc Research Park 5800 Corporate Park Pittsburgh, PA 15237 Phone: +1-412-635-3356 Email: drake@fore.com Hal Sandick IBM Corporation 800 Park Office Drive Research Triangle Park, NC Phone: +1-919-254-4614 Fax: +1-919-254-5483 Email: sandick@vnet.ibm.com Joel Halpern Newbridge Networks Corp. 593 Herndon Parkway Herndon, VA 22070-5241 Phone: +1-703-708-5954 Email: jhalpern@Newbridge.COM ***************************************************************************** Title: An Overview of PNNI Augmented Routing ***************************************************************************** DATE: April 15-19, 1996 -- Anchorage ***************************************************************************** DISTRIBUTION: PNNI Subworking Group ***************************************************************************** ABSTRACT: This contribution gives a brief overview of technical issues for PNNI Augmented Routing (PAR). PAR may be thought of as a method for bootstrapping and maintaining an overlay for the purpose of running layered routing for IP (and other internetwork level protocols) over ATM. This requires that ATM-attached routers (routers and route servers which have an interface to an ATM subnet) participate in PNNI, and make use of information gained from PNNI to support maintenance of the overlay. This contribution is intended primarily to generate discussion and an initial consideration of issues. Support for other internetwork level protocols (such as Appletalk, APPN/HPR, CLNP, DECnet, IPv6, and IPX) is for further study. It is felt that progress on PNNI Augmented Routing will be more straightforward if we first define support for one protocol (such as IP) and then consider extensions of PAR to support other internetwork level protocols. ***************************************************************************** NOTICE: This contribution has been prepared to assist the ATM Forum. This document is offered to the Forum as a basis for discussion, and is not a binding proposal on the contributing companies, nor on any other company. The statements are subject to change in form and content after further study. Specifically, we reserve the right to add to, amend, or modify the statements contained herein. *************************************************************************** 1. INTRODUCTION PNNI Augmented Routing (PAR) is a method for routing in a multiprotocol over ATM environment. This consists of running legacy internet level routing protocols (such as OSPF or RIP) over legacy media and also over ATM, but with PNNI used by ATM Border Routers (i.e., routers with interfaces to the ATM network). The ATM border routers participate in PNNI in order to locate each other, bootstrap an overlay network, and simplify maintenance of the legacy routing protocol over ATM. This paper concentrates on methods for routing IP over ATM. It is expected that other internet level protocols (possibly including, but not limited to, Appletalk, APPN/HPR, CLNP, DECnet, IPX, and IPv6) may make use of similar approaches. However, we feel that an understanding of the issues involved will be simplified by initially concentrating on one internet level protocol (IP) over ATM. When PAR is used, all routers run an IP routing protocol (e.g., OSPF, RIP, EGP, or BGP), and use the results of this routing protocol for forwarding of normal IP packets. All ATM switches run normal standard PNNI routing (i.e. PNNI Phase 1). Those routers that have ATM interfaces in addition run PNNI. ATM-attached routers that are using PAR announce a very small amount of extra information into PNNI (beyond that required for PNNI Phase 1) in order to indicate their status as routers. This information allows routers with ATM interfaces to locate other routers on the ATM network. Use of PNNI also provides these routers with detailed internal knowledge of the state of the ATM network. This allows a significant simplification of the initialization and maintenance requirements for the IP over ATM overlay. Specifically, this allows efficient maintenance of SVCs between routers across the ATM network. PAR is a sensible routing method for long term use in an IP over ATM environment. A network administrator may prefer to use PAR rather than I-PNNI when the router network is relatively large compared with the ATM network, and there is no short term need for QoS specific routing at the Internetwork level. PAR is also a sensible migration path from router networks running legacy routing protocols to an I-PNNI network. 1.1 Example Network A simple example network is shown in figure 1. In the figure there are ten routers, numbered R1 through R10, as well as three ATM switches A, B, and C. Routing within the ATM network is independent of routing between routers. Thus one routing protocol is used for routing between ATM switches (e.g., A, B, and C). A separate routing protocol is used for routing between routers (e.g., R1 to R10). Route servers and edge device forwarders are not explicitly illustrated in figure 1. This is in order to simplify the example, and is not meant to minimize the importance of route servers and edge device forwarders. In fact, the routers illustrated in figure 1 may be either conventional routers (in which the routing protocol function and IP forwarding function are done in the same physical device) or virtual routers (in which the routing protocols are done in route servers, and the IP forwarding is done in non-routing edge devices). Similarly, the protocols must support routing to virtual networks that may be distributed amongst multiple routers and amongst hosts attached to a combination of ATM and legacy LAN segments. Figure 1: Example Network with Routers and ATM Switches 1.2 ATM Attached Routers Participate in PNNI Routing PAR requires that the ATM-attached routers (and route servers) participate in the operation of the ATM PNNI routing protocol. In our example, all ten routers participate in an IP-specific routing protocol (for example OSPF may be used). The ATM switches and those routers that are directly attached to the ATM network (routers R3 through R8) participate in the operation of PNNI. Figure 2: A Method for PNNI and OSPF Interaction With this method the routers that have direct ATM interfaces understand the "full topology" as illustrated in figure 2. The full topology includes the routers (as advertised in one routing protocol, such as OSPF) and the ATM network (as advertised in another routing protocol, such as PNNI). 2. LOGICAL ROUTER TOPOLOGY The operation of the two routing protocols (for IP, and for ATM) is kept relatively independent. Clearly the routers can use VCs over the ATM network for exchanging routing information and forwarding packets. Thus it is necessary for the purpose of running a routing protocol between routers to characterize and represent the capabilities of the ATM network. Even if there are no SVCs currently set up across the ATM network, it is still necessary to advertise into the router network (e.g., running OSPF) that it is possible to forward packets across the ATM network. There are two methods that may be used here, as described in sections 2.1 and 2.2. 2.1 Pseudonode The first method is to make use of a logical "ATM pseudonode", as illustrated in figure 3. Here the ATM network is represented via logical ATM node AS ("ATM Subnetwork"), which has links to those routers that have ATM interfaces. The node AS is advertised into the IP routing protocol as if it is a normal router. One real physical system (corresponding to one of the routers with a direct ATM interface) takes on the functions of representing node AS. Selection of which system takes on these functions may be done either by manual configuration (one system is configured to represent node AS) or by running an election algorithm. The election algorithm is for future study. It may be based on the algorithms used in OSPF, IS-IS, or PNNI (each one of which makes use of a different election algorithm that represents a possible choice to be used in this case). Figure 3: Logical Router Topology with Pseudonode With this method, OSPF LSUs are forwarded over the ATM network using the illustrated logical topology. This means that the designated router (the router representing the pseudonode) needs to forward OSPF LSUs between the other routers in the ATM network. When a packet arrives for forwarding across the ATM network it will be necessary (at least initially) to set up SVCs on demand. This will result in some delay in forwarding the packet. However, the SVC should then be left up for a specified period of time, and only closed if no traffic uses that SVC for that length of time. Over time this will result in the set up of the specific SVCs that are actually needed for traffic. When a VC exists (or is created) across the ATM Subnetwork, there is an issue regarding whether to advertise existence of the VC into the IP routing protocol. One choice is to never advertise such links. With this choice the logical topology as advertised into the routing protocol remains as illustrated in figure 3. An alternate choice would be to advertise the existence of such links explicitly. This implies that the topology as illustrated in figure 3 would be enhanced by the addition of links between routers corresponding to the existing VCs. With this second choice, the metrics associated with the additional links would be chosen such that existing VCs would be preferred when available over the route via logical ATM Node. Potentially the number of SVCs could be large, so it is questionable whether it would be desirable to advertise all existing SVCs. One issue with the pseudonode method is that when a packet is to be delivered over the ATM network, the associated SVC might not be ready for use a priori. It would be highly desirable to minimize or eliminate the latency associated with SVC setup. This problem is eliminated by the alternate approach described in section 2.2. 2.2 Default Set of SVCs With this method, the routers open up a partial set of "default overlay SVCs" amongst themselves immediately upon network initialization, before the arrival of any specific traffic. The establishment of SVCs is based on the knowledge gained from running PNNI and PAR, and allows creation of a set of SVCs that span the ATM network. For any two routers X and Y which are participating in PNNI, and which are (for example) in the same OSPF area, there might not necessarily be a direct SVC from X to Y. However there will be some path from X to Y using only other routers in the same OSPF area as intermediate hops. There are a number of ways that this default set of SVCs may be set up. One method that has been proposed is to base this upon the router ID of each router. For example, consider the router ID as a circular number space. Each router may set up a direct SVC to the "k" next higher numbered routers. In the example, if k equals 1, then R3 would set up an SVC to R4. R4 would set up an SVC to R5, etc. Finally R8 (the highest numbered router on the ATM network) would set up an SVC to R3. This would result in the logical router topology illustrated in figure 4. Figure 4: Logical Router Topology with Default Set of SVCs With this method, the SVCs that are set up by default are explicitly advertised into the IP routing protocol (e.g., OSPF). One problem with this approach is that the path between any two routers may require multiple hops across the ATM network. This results in an inaccurate characterization of the capabilities of the ATM network for the purpose of calculating OSPF routes. Other methods are therefore possible, such as: - The above method may be used, but with k larger than one. In our example, with six routers on the ATM network, if k equals three then a full mesh would be set up. - A problem with the above approaches occurs when there is a packet destined to a router which is "across the name space" from the entry router. This problem could be minimized by setting up "major diagonal" SVCs. This could for example be done by having each router which is in the lower half of the name space open an SVC to the router which is exactly half the name space higher in the name space. This problem can also be minimized by setting up additional "on demand" SVCs as discussed below. - If the number of routers on the ATM network is small, then a full mesh may be set up. We expect that during the process of standardization of PNNI Augmented Routing the PNNI subworking group will choose the method to be used for setting up the default set of SVCs. With this basic approach, the default set of SVCs are treated as normal router to router point to point links. For example, IP routing packets (such as OSPF LSUs) and IP packets are forwarded between routers using the set of default SVCs. 2.3 On Demand SVCs For relatively large networks, specifically when there are a large number of routers (or route servers) with interfaces on a single ATM network, it will be infeasible to set up all possible SVCs between every possible pair of routers. Rather, a "spanning mesh" may be set up as discussed above. This implies that in some cases it may be necessary for a single packet to take more than one hop across the ATM network. In some cases, routers may observe that there are a relatively large number of packets that are destined to the same ideal exit router off the ATM subnet, for which the direct SVC is not set up (implying that these packets are taking multiple hops across ATM). In this case, the router may choose to open up an "on demand" SVC to the appropriate next hop router. If the ultimate destination is off the ATM subnet, then the information required to set up the SVC will be available a priori, without the need for query / response. If the ultimate destination is advertised by an ATM-attached router, then there is the possibility that the destination may be on a virtual subnet, and it may be desirable to do a query/ response first in order to optimize the SVC (note: it may be desirable for a router to indicate in its PAR advertisements whether it is currently supporting any virtual subnets, so that the query will be used only if there is at least a possibility that the query is needed). 2.4 Virtual Circuits via Network Management Control It is likely that in many or most cases the network managers in a particular environment may know that there is likely to be frequent traffic long term between particular routers. In this case VCs (either PVCs or SVCs) may be set up via network management control. Such VCs would be in addition to the default SVCs set up as discussed in section 2.2 above. Most likely such links should be announced and used as normal router to router links by the IP routing protocol. 3. WHAT ROUTERS ANNOUNCE INTO PNNI Routers with ATM interfaces that are running PAR participate as regular PNNI nodes, exchange PNNI Hellos with neighboring ATM switches, and broadcast regular PNNI PTSE's throughout the peer group. These routers announce the following information in PNNI packets. Regular PNNI information: - links to neighboring ATM switches - metrics on these links - PNNI node ID - ATM address - "this node is transit-restricted" (for purposes of ATM SVCs) PAR-specific information: - protocol suite supported (IP) - router ID - routing protocol used (OSPF or RIP or EGP or BGP) - if OSPF, associated area 4. NON-ISSUES PAR involves use of PNNI only over ATM interfaces. There is therefore no need to specify an encapsulation of PNNI for operation over legacy media. Similarly, there is no need to determine how to run PNNI over broadcast LANs. 5. OTHER ISSUES 5.1 Choice of IP routing protocol PAR requires that the IP routing protocol treat the default set of SVCs and normal point to point links. This implies that PAR is compatible with any IP routing protocol. For example, RIP, OSPF, EGP, and BGP are all suitable for use with PAR. 5.2 Multiple OSPF Areas in One PNNI Peer Group When there are multiple OSPF areas in the same PNNI Peer Group, each OSPF area operates independently. Each router is able to determine whether other routers are in the same OSPF area by observing the PAR information contained in the PTSE's broadcast by other routers. Routers that are in other areas are ignored. Routers that are in the same area are contacted (for example, by setting up default SVCs as appropriate). Area border routers participate in the operation of multiple OSPF areas. They may therefore be configured to have multiple areas operating over the same ATM interface. In this case they would need to include multiple [router ID, routing protocol, area] tuples in their PTSE's. Where there are multiple OSPF areas in the same ATM subnet in the same PNNI peer group, packets may take multiple hops across the ATM subnet. This may be shortcut by using NHRP. 5.3 Hierarchical Operation We will need to support hierarchical operation of PAR. We suggest that when PAR is used in a hierarchical PNNI network, then each "PAR Group" (i.e., group of routers in same OSPF area or EGP domain or RIP group, within one single peer group) elects a group representative. The representative's announcement is then advertised into the higher level (this may be ensured by using the appropriate attribute tag bits in PNNI). The representative will contact other representatives (e.g., in the same OSPF area) observed in the higher level PNNI peer group. This allows efficient operation of PAR in the case where the OSPF area or BGP confederation boundaries do not align with the PNNI hierarchical boundaries. 5.4 Multicast PNNI Augmented Routing allows ATM-attached routers to create a spanning set of SVCs across the ATM subnetwork (either directly between ATM attached routers as described in section 2.2, or between the ATM attached routers and a "pseudonode" implemented by one of the routers, as described in section 2.1). These may be treated as normal point to point links between routers for the purposes of supporting multiple IP capabilities, including IP multicast. Note that IP multicast operates successfully today over point to point links. This implies that IP multicast will operate correctly over PNNI Augmented Routing. However, there is an issue regarding whether this is the most efficient way to support IP multicast. In an environment where a router has multiple multicast clients on the ATM network, using point to point SVCs would place considerable packet copying demands on the router when point to multipoint SVCs could easily be used. MARS [5], provides a method for determining the ATM addresses of the nodes that are interested in a particular IP multicast group. This information can be used to create a mesh of point to multipoint SVCs or an SVC to a MultiCast Server (MCS), in order to deliver the multicast data. We need to consider methods to use ATM point to multipoint in order to optimize IP multicast over ATM using PAR. PAR provides ATM attached routers with both the topology of the router network (using an IP routing protocol, such as OSPF) as well as the topology of the ATM subnetwork (using PNNI). This provides the topology information needed for setting up ATM point to multipoint links for the purpose of support of IP multicast. The detailed method for support of IP multicast using PAR needs to be studied during the standardization of PAR. 5.5 QoS The combination of PAR and RSVP provides for some support of QoS. RSVP allows reservation of resources in the routers. Also, the flow information contained in RSVP can be used by ATM Border Routers to determine whether to use the default set of SVCs for the associated flow, or whether a custom SVC is needed. It is likely that in some cases it will be desirable to pick a path through the router network based on the requested QoS. In this case there are two requirements necessary in order to accomplish this goal: - It is necessary to announce sufficient metrics to accurately describe the status of the router network. This requires that either (i) OSPF be updated to allow multiple metrics to be advertised; or (ii) I-PNNI be used between routers [4]. - It is necessary to either (i) Standardize the classes of QoS which can be selected and the associated route computation; or (ii) Source route through the router network. The latter approach (source routing) requires some maintenance of state information in the routers. Further details of QoS support are for further study. 5.6 "Poor Man's" PNNI Augmented Routing We may want to consider other ways to distribute functions in order to allow cost reduction in some network devices. For example, one possible option to consider is whether PAR-knowledgeable ATM switches would allow directly attached routers to avoid running PNNI. Consider a switch which understands PAR information, but which is not a router (i.e., the switch does not run an IP level routing protocol, does not calculate IP routes, and does not forward IP packets). The router could register its PAR routing protocol information (IP routing protocol being used, OSPF area if applicable, and router ID) with the switch. The switch could then advertise the router in the switch's PTSEs, and could inform the router of the list of other routers running the same routing protocol (and in the same OSPF area, if applicable). This gives the router sufficient information to manage the SVC overlay to other routers. Whether this option is suitable for standardization may be determined during the standardization of PNNI Augmented Routing. 6. REFERENCES [1] PNNI Subworking group, "PNNI Draft Specification", ed. R.Cherukuri and D.Dykeman, ATM Forum 94-0471R15, January 1996. [2] D.Katz, D.Piscitello, B.Cole, J.Luciani, "NBMA Next Hop Resolution Protocol (NHRP)", Internet Draft, December 1995. [3] "Methods for Routing of Internetwork Level Protocols over ATM", ATM Forum 96-0353, April 1996. [4] "Issues and Approaches for Integrated PNNI", ATM Forum 96-0355, April 1996.