Network Working Group H. Wang Internet-Draft Huawei Intended status: Informational F. Qin Expires: 2 September 2024 China Mobile L. Zhao S. Chen H. Huang Huawei 1 March 2024 Framework of Fast Fault Detection for IP-based Network draft-wang-ffd-framework-02 Abstract The IP-based distributed system and software application layer often use heartbeat to maintain the network topology status. However, the heartbeat setting is long, which prolongs the system fault detection time. IP-based storage network is the typical usage of that scenario. When the IP-based storage network fault occurs, NVMe connections need to be switched over. Currently, no effective method is available for quick detection, switchover is performed only based on keepalive timeout, resulting in low performance. This document defines the basic framework of how network assisted host devices can quickly detect application connection failures caused by network faults. Requirements Language The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [RFC2119]. Status of This Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at https://datatracker.ietf.org/drafts/current/. Wang, et al. Expires 2 September 2024 [Page 1] Internet-Draft Framework of FFD for IP-based Network March 2024 Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." This Internet-Draft will expire on 2 September 2024. Copyright Notice Copyright (c) 2024 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/ license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License. Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 3 3. Reference Models . . . . . . . . . . . . . . . . . . . . . . 3 3.1. Small-scale SAN . . . . . . . . . . . . . . . . . . . . . 4 3.2. Large-scale SAN . . . . . . . . . . . . . . . . . . . . . 5 4. Functional Components . . . . . . . . . . . . . . . . . . . . 6 4.1. Storage Device . . . . . . . . . . . . . . . . . . . . . 6 4.2. Host . . . . . . . . . . . . . . . . . . . . . . . . . . 6 4.3. Network Device . . . . . . . . . . . . . . . . . . . . . 7 5. Procedures . . . . . . . . . . . . . . . . . . . . . . . . . 7 5.1. Network Deployment . . . . . . . . . . . . . . . . . . . 7 5.2. Storage and Host Access . . . . . . . . . . . . . . . . . 7 5.3. Status Infomation Sync And Notification . . . . . . . . . 8 5.3.1. Access Link Failure . . . . . . . . . . . . . . . . . 8 5.3.2. Network Link or Device Failure . . . . . . . . . . . 8 6. Security Considerations . . . . . . . . . . . . . . . . . . . 9 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 9 8. References . . . . . . . . . . . . . . . . . . . . . . . . . 9 8.1. Normative References . . . . . . . . . . . . . . . . . . 9 8.2. Informative References . . . . . . . . . . . . . . . . . 9 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 9 Wang, et al. Expires 2 September 2024 [Page 2] Internet-Draft Framework of FFD for IP-based Network March 2024 1. Introduction Today, distributed systems based on network communication are widely used. In order to ensure that both ends of the distributed system can perceive faults, heartbeat is a common technology. However, relying on the heartbeat to detect whether the peer is faulty also faces challenges: if the heartbeat is set too short, it may be misjudged by network disturbances; if the heartbeat is set too long, when a fault occurs, it will not be found for a long time. Application scenarios such as IP-based NVMe, distributed storage, and cluster computing are typical scenarios for such technologies. The [I-D.guo-ffd-requirement] describes the problems of the current IP-based NVMe solution. On an IP-based storage area network, if the access link of a storage device is faulty, hosts cannot access the storage device. Because the host cannot directly detect the fault, the host has to wait for the KA timeout. To speed up fault detection, hosts and storage devices can implement fast KA or BFD. However, this solution introdueced additional cost on hosts and storage devices and is hard to use in large-scale IP-based storage area network. In fact, the IP network can directly detect these faults, so we can use the IP network to assist these access endpoints to quickly perceive the fault, so as to perform quickly service recovery. 2. Terminology NoF : NVMe of Fabrics FC : Fiber Channel NVMe : Non-Volatile Memory Express SAN: Storage Area Network 3. Reference Models The frame solution here is applicable to the system where the terminals are directly connected to the IP network. +--------+ +-----------+ +-----------+ +--------+ |Terminal|----| IP Network|-----| IP Network|----|Terminal| | Device | | Device | | Device | | Device | +--------+ +-----------+ +-----------+ +--------+ Figure 1 : Basic framework Wang, et al. Expires 2 September 2024 [Page 3] Internet-Draft Framework of FFD for IP-based Network March 2024 Terminals are connected to the IP network, and they establish IP connections through the reachability provided by the IP network. When the connection path fails, they cannot be detected quickly. They can only detect it after the keep-alive timeout, and then can carry out service protection processing. This time may be relatively long. Therefore, it is necessary to notify the terminal device of some failures in the network, such as access port failures and internal network failures that will cause IP connection failures between terminals, so that the terminal device can respond quickly and perform corresponding service processing. As introduced in Introduction, there are scenarios such as IP-based NVMe, distributed storage, and cluster computing. Here we take IP- based NVME as a typical scenario for introduction, and the processing behavior of other scenarios is similar. An IP-based storage area network mainly includes three types of roles: o Initiator, the terminal device, is also called the host. o Switch, which is a network device used to access terminal devices. o Target is also a terminal device, also known as a storage device. The host and storage devices use the Ethernet-based NVMe protocol to transmit data through the IP network to provide high-performance storage services. 3.1. Small-scale SAN +--+ +--+ Host |H1| |H2| (Initiator) +-,+ +_.+ | `', _-` | | _-` | | _-` `', | IP +----+ +----+ Network | SW1| | SW2| +---,+ +_.--+ | `', _-` | | `', | | _-` `', | Storage +-`+ +`'+ (Target) |S1| |S2| +--+ +--+ Figure 1 : Small-scale SAN Wang, et al. Expires 2 September 2024 [Page 4] Internet-Draft Framework of FFD for IP-based Network March 2024 This is the basic model for small-scale storage access networks. Hosts and storage devices are dual-homed to different switches. When the access link of the storage device is faulty, the host needs to quickly detect the fault so that the NVMe connection can be quickly switched to the standby path. 3.2. Large-scale SAN +--+ +--+ +--+ +--+ Host |H1| |H2| |H3| |H4| (Initiator) +/-+ +-,+ +.-+ +/-+ | | '. ,-`| | | | `', | | | | ,-` '. | | +-\--+ +--`-+ +`'--+ +-\--+ | SW1| | SW2| | SW3| | SW4| +--,-+ +---,, +,.--+ +-.--+ `. `'.,` .` `. _,-'` ``'., .` IP +--'`+ +`-`-+ Network | SW5| | SW6| +--,,+ +,.,-+ .` `'., ,.-`` ', .` _,-'` `. +--`-+ +--'`+ `'---+ +-`'-+ | SW7| | SW8| | SW9| |SW10| +-.,-+ +-..-+ +-.,-+ +-_.-+ | '. ,-` | | `., .' | | `', | | '.` | | ,-` '. | | ,-` `', | Storage +-`+ `'\+ +-`+ +`'+ (Target) |S1| |S2| |S3| |S4| +--+ +--+ +--+ +--+ Figure 2 : Large-scale SAN This is a relatively large-scale storage network which applies to a large-scale storage device access network. When the access link of the storage device is faulty, the host needs to quickly detect the fault so that the NVMe connection can be quickly switched to the standby path. Wang, et al. Expires 2 September 2024 [Page 5] Internet-Draft Framework of FFD for IP-based Network March 2024 4. Functional Components The NVMe IP-based SANs consists of storage devices, hosts and switches. Hosts and storage devices need to obtain required fault information from the IP network. Switches need to synchronize locally detected fault information on the IP network so that other switches can obtain the faults and notify hosts or storage devices that require the fault infomation. 4.1. Storage Device As the server side, storage devices provide storage access services for hosts. If a storage device is connected to an IP network and is interested in the status of other devices, the storage device can initiate a subscription request to the connected switch to obtain status notifications of other devices from the access switch. To reduce the complexity of storage devices, it's suggest to extend the LLDP protocol to support subscription from storage devices to switches and use the new L2-based protocol to notify the switch of status to the storage device. +-------+ +------+ |Storage| |Switch| +-------+ +------+ | Subscribe Msg | | ----------------------->| | | | Notification Msg | | <-----------------------| | | | | Figure 3 : Storage Device 4.2. Host The host is the client of the storage device. As the client side, a host needs to quickly obtain the service status of the storage device that provides services. When the host receives a notification message from the switch indicating that the storage device is faulty, the host will quickly disconnect from the storage device and switch to a redundant one. The recommended protocol on the host side is the same as that on the storage device. Wang, et al. Expires 2 September 2024 [Page 6] Internet-Draft Framework of FFD for IP-based Network March 2024 +-------+ +------+ | HOST | |Switch| +-------+ +------+ | Subscribe Msg | | ----------------------->| | | | Notification Msg | | <-----------------------| | | | | Figure 4 : Host Device 4.3. Network Device Switches can quickly detect local faults and synchronize the faults to other switches on the IP network. After detecting a fault, the switch needs to notify the required host or storage device of the fault. +------+ +------+ |Switch| |Switch| +------+ +------+ | Information Sync | | ----------------------->| | | | | | | Figure 5 : Network Device 5. Procedures 5.1. Network Deployment The IP-based SAN uses the standard Ethernet technolog. Network deployments typically use the current IP technologies. For example, OSPF is usually deployed as an underlay protocol. 5.2. Storage and Host Access Hosts and storage devices are connected to the ethernet network. The administrator assigns access IP addresses to the hosts and storage devices. In most scenarios, these routes can be advertised through the underlay protocol. In addition, after hosts and storage devices go online, they needs to send subscription requests to the switch to obtain the status information of the target device. Wang, et al. Expires 2 September 2024 [Page 7] Internet-Draft Framework of FFD for IP-based Network March 2024 To prevent hosts or storage devices from being aware of extra IP address, it is recommended that LLDP be used to implement this message. 5.3. Status Infomation Sync And Notification When hosts and storage devices go online, the switch can calculates an initial state of these devices and synchronizes the state on the IP network. After detecting a local fault, the switch needs to notify other access devices that need the fault information. In addition, the switch needs to synchronize the fault information to other switches on the network. To ensure that synchronization messages can be reliably synchronized to other switches, a reliable transmission protocol, such as TCP or Quic, must be used. For large-scale IP networks, hierarchical synchronization can be used to reduce the number of sessions between switches. The synchronization information about the host and storage devices belongs to the application layer's information. +-------+ +----+ +------+ +----+ +-------+ | HOST |-----------|TOR1|------|Spine1|------|TOR3|---------|Storage| +---/---+ +-/--+ +--/---+ +-/--+ +---/---+ |---------------->| Info Sync | Info Sync |<---------------| | SubscribeMsg |----------->|<-----------| Subscribe Msg | | |<-----------|----------->| | |<----------------| Info Sync | Info Sync | | |Notification Msg | | | | | | | | | Figure 7 : Information Advertisement 5.3.1. Access Link Failure When an access link is faulty, the access switch detects the fault. Based on the faulty link, the access switch can calculate the devices whose IP addresses are affected. The access switch advertises the faulty IP address information on other access links. The switch synchronizes the faulty IP address information on the IP network based on the computation result. After receiving the synchronized fault information, other switches notify the access host or storage device of the fault information. 5.3.2. Network Link or Device Failure ECMP or redundant link protection is usually deployed to prevent this failure. Wang, et al. Expires 2 September 2024 [Page 8] Internet-Draft Framework of FFD for IP-based Network March 2024 6. Security Considerations In order to control the communication range of information and reduce the negative impact of possible information flooding, the Subscribe Msg and Notification Msg considered in this framework are suggested to be implemented through the L2 extension protocol, so that the sending and receiving of this information will only be controlled by the access network device within the domain. At the same time, the network device is not allowed to forward this message, only allowed to receive or send such message as needed. For the communication protocol between network devices, in order to ensure its security, it can be encrypted by commonly used encryption technology, including but not limited to TCP-AO, TLS and other technologies. 7. IANA Considerations This document makes no request of IANA. 8. References 8.1. Normative References [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, . 8.2. Informative References [I-D.guo-ffd-requirement] Guo, L., Feng, Y., Zhao, J., Qin, F., Zhao, L., Wang, H., and W. Quan, "Requirement of Fast Fault Detection for IP- based Network", Work in Progress, Internet-Draft, draft- guo-ffd-requirement-01, 9 March 2023, . Authors' Addresses Haibo Wang Huawei No. 156 Beiqing Road Beijing 100095 P.R. China Email: rainsword.wang@huawei.com Wang, et al. Expires 2 September 2024 [Page 9] Internet-Draft Framework of FFD for IP-based Network March 2024 Fengwei Qin China Mobile Beijing China Email: qinfengwei@chinamobile.com Lily Zhao Huawei No. 3 Shangdi Information Road Beijing 100085 P.R. China Email: Lily.zhao@huawei.com Shuanglong Chen Huawei No. 156 Beiqing Road Beijing 100095 P.R. China Email: chenshuanglong@huawei.com Hongyi Huang Huawei No. 156 Beiqing Road Beijing 100095 P.R. China Email: hongyi.huang@huawei.com Wang, et al. Expires 2 September 2024 [Page 10]