| Internet-Draft | IP in Deep Space | September 2023 |
| Blanchet, et al. | Expires 11 March 2024 | [Page] |
Deep space communications involve long delays (e.g., Earth to Mars is 4-20 minutes) and intermittent communications, because of orbital dynamics. Up to now, communications have been done on a layer-2 point to point basis, with sometimes the use of relays, therefore no layer-3 networking was possible. RFC4838 reports an assessment done around 25 years ago concluding that the IP protocol stack was not suitable for deep space networking. This result lead to the definition of a new protocol stack based on a store-and-forward paradigm implemented in the Bundle Protocol(BP). More recently, space agencies are planning to deploy IP networks on celestial bodies, such as Moon or Mars, ground, and vicinity. This document revisits the initial assessment of rejecting IP and provides solution paths to use the IP protocol stack, from IP forwarding to transport to applications to network management, in deep space communications.¶
This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.¶
Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.¶
Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."¶
This Internet-Draft will expire on 11 March 2024.¶
Copyright (c) 2023 IETF Trust and the persons identified as the document authors. All rights reserved.¶
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License.¶
Deep space communications involve long delays (e.g., Earth to Mars is 4-20 minutes) and intermittent communications, because of orbital dynamics. Up to now, communications have been done on a layer-2 point to point basis, with sometimes the use of relays, therefore no layer-3 networking was possible. [RFC4838] reports an assessment done around 25 years ago concluding that the IP protocol stack was not suitable for deep space networking. This result lead to the definition of a new protocol stack based on a store-and-forward paradigm implemented in the Bundle Protocol(BP) [RFC9171] and its various components, such as convergence-layer adapters([RFC9174], [RFC7122]) and BP Security(BPSEC)[RFC9172].¶
More recently, space agencies are planning to deploy IP networks on celestial bodies, such as Moon[ioag] or Mars[ioag-mars], ground, and vicinity, using layer2 technologies such as WIFI or 5G.¶
This document revisits the initial assessment of rejecting IP and provide solution paths to use IP in deep space communications. IP in deep space means running IP over deep space layer-2 links, a reliable transport over IP, applications protocols over that transport and applying proper routing, security and network management on that IP network. Reusing the whole IP stack in deep space enables the reuse of all protocols, tools and software currently used on Internet. However, as one might already argue, most of the IP stack can not be used as is and therefore requires careful configuration and possibly some protocol changes that are discussed in this document.¶
The examplary network for this document is where deep space links are using IP over CCSDS space links[IPoverCCSDSSpaceLinks] and that on and around a celestial body, a connected network is established with local network infrastructure and services.¶
The keyword Delay-Tolerant Networking (DTN), also expanded to Delay and Disruption-Tolerant Networking, has been used to identify the problem space and given that up to now, the solution was based on the Bundle protocol, DTN was also associated with Bundle protocol. This document tries to solve the DTN problem using the Internet Protocol stack. Therefore, in this document, the DTN keyword is used to name the problem space, not the Bundle protocol solution.¶
This document covers more topics than what may need to be discussed or standardized in IETF, but the intent is to help answer many questions raised while looking at the whole problem space, and, in this context, provides an non-exhaustive list of topics that needs to be addressed.¶
Since Moon is a few light seconds away from Earth, it is possible to somewhat configure and run various IP based protocols and applications to make it "work". Mars with a much longer delay is more difficult. Therefore, this document uses Mars as the base example, knowing that if it works for Mars, a much harder problem, it could be replicated easily for Moon, or for other networks made with relays around a celestial body. This framework shall also work for longer delays, such as reaching Jupiter or the whole Solar System Internet(SSI), but it is not specifically discussed. This document uses "deep space" extensively in order to differentiate with "space" which often includes Earth orbiting communications, which is not discussed in this document.¶
It should also be noted that DTN and BP were also designed for non-space use cases. While this document focuses on the deep space use case, it shall work for the other use cases of BP, but no work or discussion on these other use cases is provided in this document.¶
Space missions are typically planned many years in advance and are long-lived, spanning over many years even decades. Spacecrafts are controlled from Earth and therefore should always be manageable from Earth. Given the remoteness and the difficulty to physically access the spacecraft, software upgrades and configuration changes are avoided whenever possible.¶
As with Bundle protocol, this framework proposes to use IP in deep space with the same store-and-forward paradigm. Therefore, the IP layer has to deal with the fact that a destination may not be currently reachable and that IP packets should be stored for an unusual amount of time, such as minutes or hours or days, in the forwarding device waiting for a new link up opportunity. The transport layer should be able to work with long and variable delays, including intermittent communications. The application protocols and application themselves should be properly set to wait a longer time than on Internet to receive a response to a query. Finally, all network services such as routing, security, naming and network management should also be adapted in this new context. This document is structured around these layers.¶
In a nutshell, this framework is based on the following main pillars: the storage of IP packets in intermediary nodes while the destination is unreachable, the use of the QUIC transport[RFC9000] with proper configuration, the predominance of using HTTP protocol (over QUIC) for applications, and considerations related to time in all levels of the stack.¶
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.¶
The source of this document is located at https://github.com/marcblanchet/draft-deepspace-ip-assessment. Comments or changes are welcomed as a PR or an issue.¶
This subject should be discussed on the deepspace@ietf.org mailing list.¶
In the context of deep space, an IP packet would need to be stored temporarily over some possible longer period than typical Internet when the next hop is currently unreachable or undefined, for example due to orbital dynamics. Therefore, a new queueing discipline might be needed to store packets in this context, that might be implemented as a deep queue with active queue management(AQM)[RFC7567]. Bidirectional Forwarding Detection(BFD)[RFC5880] with large timers should be considered. When the link to the next hop is up, maybe minutes or hours later, forwarding tables would be updated and stored packets would be forwarded on the link for appropriate destinations. This is being discussed in the Time-Variant Routing(TVR) working group [TVRWG] and one part is implemented as a YANG model [I-D.qu-tvr-schedule-yang].¶
The store-and-forward paradigm, either implemented in BP or IP or at higher layers, requires proper sizing of memory and storage at each forwarding node for the target deployment and usage.¶
Store and Forward policies shall be defined and implemented to cover cases such as when storage is full and new packets are received or which priorities should be given to packets when link becomes up. An example is described in [I-D.blanchet-tvr-forwarding].¶
Various IP stack kernel buffers used for example for reassembly queues need to be properly sized for the target usage.¶
Storing IP packets, or BP bundles, may in fact make the buffer bloat issue[buffer-bloat] a much bigger problem if the management of stored packets is not properly implemented with the right queueing discipline.¶
Given the relative static nature of space networks at least for the forseeable future, e.g., new nodes or routers are not often added or deleted in the network, use of static routes configured based on contact plan schedules ([I-D.blanchet-tvr-contactplan],[I-D.qu-tvr-schedule-yang] may be sufficient in the short term.¶
If run over space links, IGPs such as OSPFv3 [RFC5340] have time-related configurable parameters such as RxmtInterval, InfTransDelay, HelloInterval, RouterDeadInterval. There are 16 bits values in seconds, which means a maximum of ~18 hours. This maximum may be too small depending on the contact plan schedule. Time-Variant Routing(TVR) working group [TVRWG] is looking at this problem space.¶
Given that it is likely that multiple network operators will be present on celestial bodies, it is expected that BGP [RFC4271] would be used.¶
[RFC4838]describes various issues that make the IP protocol suite not suitable for space. One of them was TCP handshake and timers that would not work over a 20 minute delay link. If TLS handshake is added on top of TCP, then it is even worse. In fact, this is similar, but not identical because of disruptions, to large bandwidth-delay product use cases[RFC1072], because the delay is large, even if the bandwidth is somewhat much smaller than on Internet. In last 25 years, transport protocols have evolved a lot. QUIC ([RFC9000], [RFC9001], ...) is a potential transport solution to the space communications characteristics, given all its novel features.¶
Current implementations of QUIC typically set the initial RTT estimates in hundreds of milliseconds as it is expected to be a good start for connecting establishment on Internet. However, that value is way too low for long delay communications in deep space. By adjusting initial RTT to a proper value in the QUIC stack for the deep space connection, and given that many timers in QUIC are related to the RTT, a successful connection can be established. An initial proof of concept[picoquic-poc] with a QUIC implementation[picoquic] showed potential use of QUIC in deep space. Additional timers and parameters may need to be adjusted. Further investigation of QUIC use in deep space is needed, especially for congestion control, detection and recovering from loss of data. A document describing how to profile a QUIC stack for this use case is being written[draft-huitema-quic-in-space]. The possible use of careful resume [I-D.ietf-tsvwg-careful-resume] should be considered, as a way to dynamically update QUIC client stacks based on a better known RTT estimate from the server.¶
Establishing a QUIC connections includes discovering the network conditions, which typically requires several RTT. For space communications, we need to minimize the impact of the initial delays either by keeping connections up for a long time, or if that is not possible by using mechanisms like 0RTT or careful resume to accelerate the re-discovery of network conditions by the new connection.¶
The ability to have multiple streams and applications within a single QUIC connection is also very valuable and useful for this use case. A ground station may setup the initial QUIC connection with a spacecraft and then carry all needed applications and streams over that same connection.¶
Given that spacecrafts and ground systems are aware of each other and are typically managed by the same organization, the trust anchors may be preloaded so that each peer already have a trust relationship with the other. This, together with the resume token acquired during a previous connection, would enable the use of the 0-RTT QUIC feature enabling sending application data on the first packet. Even more, the initial QUIC connection could be established while the spacecraft is on the ground, so that while moving in space, the connection only needs to be updated with new RTT estimates, either by configuration or automatically. It should be noted that the 0-RTT feature is encrypted but vulnerable to replay attacks, which should be considered in the missions risk assessment.¶
The mandatory ability to have TLS negotiated at the beginning of the QUIC connection makes the use of IP layer security (aka IPSec) less needed to be deployed, while knowing that IP and transport security are different. Operational complexity in space is very costly and brings brittleness of the reliability and therefore, a single layer of security, managed at the application transport (aka HTTP-QUIC), is very valuable.¶
Session key and certificate lifetime together with certificate validation and trust chain anchors need to be carefully configured and handled. This is further discussed in section Section 7.4.¶
The store-and-forward paradigm, needed for deep space, may also be implemented at the QUIC transport layer instead of the IP layer, by architecting a string of QUIC proxies. Those proxies would need to react to the socket error of no connection by storing the data payload, until the next opportunity arise. However, that would require that the actual topology of the network be known at each proxy and that contact plans also be known at the QUIC proxy layer so that the proxy does not try to contact when in fact, no connectivity is happening. There would be no routing updates, so any link change from the contact plan would not be used. However, some messaging layer may be put on a network of message relays, connected by QUIC connections.¶
QUIC in deep space has very good promise but requires extensive investigation and testing to ensure proper usage and deployment.¶
HTTP by itself has no notion of time. An HTTP request and response may take minutes or hours to be completed, theoretically. However, current infrastructure and software on Internet have various time-related configurations that will not work as is in the deep space context.¶
HTTP headers containing time, such as Cache-Control and Expires [RFC9111] need to be set large enough to cover the longest delay so that expiration does not happen before the actual data arrives at the destination. As with any HTTP application and content on Internet, these headers should be set properly based on the deployment use case, which is ever more important for deep space. Similarly, when continuous content transfer is used, as with 100-Continue [RFC9110], proper values for headers should be set.¶
HTTP clients and servers typically have default timeouts that shall be modified. For example, curl [curl] has the "-m" option for this use case. Similarly, HTTP server implementations have various timeouts configuration variables to be set properly. Testing with HTTP client Curl and HTTP server nginx and an introduced network delay of 20 minutes showed that HTTP communications work just fine with very basic configuration changes.¶
HTTP applications themselves must be developed using an asynchronous pattern and if they have timeouts, they should be adjusted /appropriately.¶
Internet Web sites are designed with the assumption of hundred of milliseconds delay and relatively always connected, where pages contain multiple queries to further get resources, media, queries to web services and downloading additional code and frameworks. This could work in theory in this context of space, but it will not be optimal, as multiple queries will be generated and therefore taking multiple RTT before the whole page is received complete. This issue can be mitigated by using various techniques such as Web Assembly [wasm] or pre-caching. Moreover, tt could be possible to have very basic HTML pages with zero or very few href and no media content unless locally cached to be used. An example would be a rover on Mars presenting an HTTP server with a base and bare HTML page to offer basic info on its status (maybe all in text) and some additional detailed pages, most likely also in base html text. However, it is foreseen that most applications based on QUIC-HTTP transport in deep space would be using REST or similar asynchronous patterns and not typical web browsing.¶
Caching should be used extensively on celestial bodies networks to maximize local fetching. Preemptive caching by pre-populating caches with data that shall be used locally on the celestial body network shall be done as much as possible to provide better response time on the local celestial body network.¶
QPACK [RFC9204] should be considered for higher bandwidth efficiency.¶
It should be noted that COAP [RFC7252] is worth considering for application transport in deep space.¶
There are a large number of IETF-defined IP-based application protocols, as well as non-standard ones. Some may work as is in a deep space environment, some others may require changes in timers or else in protocol or in the implementation, and some may not work at all. It would be appropriate to pick the most likely used application protocols and assess their usability for this use case. It may also be useful to test implementations. Obviously, the needed characteristic of these application protocols is the asynchronous paradigm, given long delays and intermittent communications. One outcome of this assessment could be a best practice document on how to write applications in such use case.¶
It should be noted that if the application is using HTTP as a transport, and that guidance on using HTTP as described in Section 5 is followed, then the HTTP application should work.¶
Domain name requests and response over long delays generate timeouts and when there is no reachability to the DNS server, requests will not be answered. Therefore, on celestial bodies IP networks, a local DNS infrastructure with all the names and values stored locally is needed. Moreover, to keep the same DNS root and the current DNSSEC trust chain, all keys necessary for validation should also be stored locally. The DNSSEC RR TTL values would need to be longer than the mission lifetime. [dns-isolated-networks] describes the various ways to achieve naming in isolated networks use case, which applies to deep space.¶
NETCONF[RFC6241] and RESTCONF[RFC8040] can be used with proper configuration values in implementations to avoid timeouts. RESTCONF with appropriate HTTP config would enable long delayed queries to be working. NETCONF uses TCP which won't work on delayed and intermittent communications. If NETCONF is defined over QUIC, then it could be used with proper QUIC profile as discussed in Section 4. On the other hand, RESTCONF with proper HTTP profile would just work fine.¶
RESTCONF uses various timestamps and HTTP time related headers to compare transactions time, so for example to avoid any race in configuration changes. However, there is no notion of timeout in the protocol itself, so it should work as is. Implementations, at the RESTCONF level or underneath (HTTP) may have implementation-specific timeouts that should be configured properly to handle long delays.¶
A network manager should obviously be aware that a RESTCONF notification sent by a server that travels over some delayed links or networks will arrive later than typical, and such notification may be less useful at the time it arrives. However, this is the reality of a delayed and intermittent communications network.¶
SSH in its normal interactive mode sends each character in a separate packet, which over long delay networks, will not be optimal. SSH can be configured in line mode where packets are sent only when a full line is entered. That mode shall be preferred. However, SSH like NETCONF runs over TCP which will not work. However, SSH over QUIC was proposed[I-D.bider-ssh-quic] which proper QUIC configuration might work.¶
As a summary, it seems possible that all IP nodes (and even dual-stack (BP and IP) nodes) in space can be remotely managed using RESTCONF, as well as locally managed.¶
On Earth, as it is planned today, the space network shall be isolated from the current Internet by "air gap", to disable any direct communications from Internet to deep space. Moreover, destination IP prefixes filtering shall be used to restrict the traffic to only the relevant one for each link. Note that this shall also be implemented in the routing control plane, but additional security might be appropriate to further protect the deep space links.¶
Each celestial network edge device shall have firewall rules to disable non-useful trafic to go through deep space links. If communications from Mars may only occur to Earth, but not Moon, then appropriate filtering based on destination IPv6 prefixes shall be used.¶
Given the air gap on Earth for Internet, there shall be no default route advertised in space that could for example point to Earth Internet.¶
Caching should be used aggressively in all levels of the IP stack in this architecture. For example, DNS servers on remote celestial bodies should have all useful names already configured. HTTP Caches should be deployed and preemptively filled with all necessary objects.¶
IPsec may be used to provide secure tunnelling or VPN set of services. However, QUIC-TLS may be instead investigated to provide the needed security, given that at the transport level, the security parameters may be dictated by the applications, therefore ensuring security policies from end to end applications, instead of at the network level.¶
There will likely be multiple "operators" on celestial IP networks. Therefore it is likely needed to provide some kind of exchange point similar to the Internet Exchange Point(IXP) as we know today on Internet. This would enable local exchange of trafic on the celestial body without going through deep space links. BGP should then be considered.¶
Protocols or infrastructure using crypto keys and certificates should be carefully managed. Certificates and session keys should have a lifetime large enough so that they will still be valid even when longer than expected disruptions happen. For missions, certificate lifetimes should be considered based on a mission extended lifetime. The whole trust chain for certificate validation, including updates, should also be transferred in advance to the appropriate location on the celestial body network to avoid invalid verifications because of the lack of an intermediate certificate, as one do not want to query any parent certificate over space links. Similarly, Certificate Revocation Lists(CRL) real-time fetching [RFC5280] or Online Certificate Status Protocol[RFC2560] should be investigated before considering their use in this environment.¶
Since this framework reuses the IP protocol stack, it inherently assumes time coherence between the celestial bodies networks and no changes in how protocols are specifying time. Therefore, it practically assumes UTC-based time being available on all celestial bodies, so that time related comparisons can be achieved. The way to accomplish this on the celestial bodies networks and in space is out of scope for this document.¶
With proper profiling of protocols, software and operations, and with possibly little to no changes in protocols, the use of the IP protocol stack in deep space with long delays and intermittent communications seems possible and provides an alternative to a Bundle protocol based network. This is now possible to envision because of the various advances in the IP stack, specially the QUIC transport, compared to the initial assessment done 25 years ago.¶
This memo includes no request to IANA.¶
This section lists additional considerations that are important for the deployment of IP in deep space, but may not require changes to protocols or implementations, as they are more related to network operations.¶
As discussed in the previous section, space missions are long-lived and require full reachability to every spacecraft. Since IPv4 address space is consumed, the use of IPv4 address space would rely on using Network Address Translation(NAT) in space. The significant consequence is the one-way reachability that NAT creates: as soon as there is a NAT in the path from the source to the destination, the destination is not be directly reachable. For example, if a NAT is deployed in space, the spacecrafts behind the NAT will not be directly reachable from Earth missions operations and network management consoles. If the NAT is on the other side of the connections, then spacecrafts will not be able to communicate or send notifications to mission operations or network management consoles. This is a significant issue if using IPV4 in deep space.¶
The Internet has been transitioning from IPv4 to IPv6 to continue its expansion and since the missions are long-lived, IPv6 is the only IP version that has the appropriate lifetime for the deep space network.¶
IPv6 also brings specific features such as IPv6 DHCP prefix delegation[RFC3633] that could be used for spacecrafts as mobile networks docking into a temporary or more permanent network and getting a prefix from the attaching network.¶
Finally, running both IPv4 and IPv6 simultaneously, while doable as we do on Internet today, brings additional operational challenges both in security, such as handling multiple versions of access control lists, and network management that one should avoid as much as possible in space.¶
Therefore, IPv6 is recommended to be the only IP protocol version used in space. Moreover, if there is any required protocol change, IPv6 should be the base IP protocol used underneath.¶
Most protocols and topics discussed in this document are independent of the IP version, but if there are differences, only the IPv6 version is discussed.¶
Space communications infrastructure must avoid at all costs any address space collision, since it would prevent any reachability between the colliding networks. For example, if one organization creates a network on a celestial body using an address space and another organization, or even worse the same organization, creates another network anywhere in space using the same address space, those two networks will not be able to reach each other, undermining any communications. NATs can be used between the two but this will just further complexify or disable remote network management.¶
IPv6 addressing in space should only use non-overlapping address space, based on duly allocated IPv6 space assigned to each organization, from the public IPv6 address space[RFC4291] managed by registries, providing IPv6 network services in space. IPv6 unique-local addressing [RFC4193] can be used within a single domain but such traffic should not cross domains using the unique-local addressing, and should instead used the global addressing as managed by the IPv6 preferred address algorithm [RFC6724].¶
Current Regional Internet Registries(RIR) may not have in place the appropriate policies for the deep space use case, since these policies are aimed towards terrestrial Internet usage, such as broadband usage, ISP peering and other considerations. These policies may need to be revised to include deep space usage or another organization with proper membership to be put in place to allocate and assign IPv6 address space, and possibly Autonomous System Numbers, for that specific community and usage.¶
The Consultative Committee for Space Data Systems (CCSDS, [CCSDSWEB] is a standard organization for space communications, which membership is space agencies and related commercial organizations. IP packet encapsulation into space links is defined in CCSDS 702 [IPoverCCSDSSpaceLinks] and the codepoint for IPv6 packet encapsulation is 87 for space links, as specified in the Space Assigned Numbers Registry(SANA) [SANAIPEHeaderRegistry]. However, it is unknown if IPv6 has been implemented and tested, given long delays and Neighbor Discovery [RFC4861] timers. An IPv6 over CCSDS Space Links specification may need to be defined.¶
As discussed in Section 1, the Bundle Protocol [RFC9171] has been designed to create a store-and-forward networking capability for space and other use cases. This document framework specifies an alternate way to accomplish the networking for the same use case by reusing as much as possible the IP protocol stack, therefore inheriting all the engineering and implementations implemented and running daily on Internet. This re-use also means that implementations have been exercised on real traffic orders of magnitude more than what has been achieved with Bundle protocol stacks.¶
It is possible to mix BP and IP on the same links or networks. Therefore, this framework and the BP stack can create independent and simultaneous networks or can be mixed.¶
This work started by reassessing the use of the whole IP stack in the context of deep space. Soon, QUIC was identified as the key technology for this endeavour. Christian Huitema was very helpful in not only confirming the ability to use QUIC but also took the time and effort to test and modify its picoquic stack[picoquic] to confirm the initial hypothesis[picoquic-poc]. Its involvement and confirmation are the key for the launch of this work. Then, Martin Thompson has been also kind to take time to answer initial questions on QUIC, further confirming the possibility of using QUIC for deep space. Since then, many individuals have provided significant comments and perspectives on this subject.¶
This document and its underlying work has been reviewed and discussed by many, who have provided valuable feedback and comments, including disagreements, and made an overall more solid document. These people are, in no specific order: Geoff Huston, Martin Thompson, Peter Ashwood-Smith, Tony Li, George Neville-Neil, Jim Gettys, Nicolas Kuhn, Eric Vyncke, Russ Housley, Emile Stephan, Dave Taht, Jean-Philippe Dionne.¶