Main logging schema for qlog

1.1.1. Schema definition

To define events and data structures, all qlog documents use the Concise Data Definition Language [CDDL]. This document uses the basic syntax, the specific text, uint, float32, float64, bool, and any types, as well as the .default, .size, and .regexp control operators, the ~ unwrapping operator, and the $ extension point syntax from [CDDL].¶

Additionally, this document defines the following custom types for clarity:¶

; CDDL's uint is defined as being 64-bit in size
; but for many protocol fields it is better to be restrictive
; and explicit
uint8 = uint .size 1
uint16 = uint .size 2
uint32 = uint .size 4
uint64 = uint .size 8

; an even-length lowercase string of hexadecimally encoded bytes
; examples: 82dc, 027339, 4cdbfd9bf0
; this is needed because the default CDDL binary string (bytes/bstr)
; is only CBOR and not JSON compatible
hexstring = text .regexp "([0-9a-f]{2})*"

All timestamps and time-related values (e.g., offsets) in qlog are logged as float64 in the millisecond resolution.¶

Other qlog documents can define their own CDDL-compatible (struct) types (e.g., separately for each Packet type that a protocol supports).¶

• Note to RFC editor: Please remove the following text in this section before publication.¶

The main general CDDL syntax conventions in this document a reader should be aware of for easy reading comprehension are:¶

• ? obj : this object is optional¶
• TypeName1 / TypeName2 : a union of these two types (object can be either type 1 OR type 2)¶
• obj: TypeName : this object has this concrete type¶
• obj: [* TypeName] : this object is an array of this type with minimum size of 0 elements¶
• obj: [+ TypeName] : this object is an array of this type with minimum size of 1 element¶
• TypeName = ... : defines a new type¶
• EnumName = "entry1" / "entry2" / entry3 / ...: defines an enum¶
• StructName = { ... } : defines a new struct type¶
• ; : single-line comment¶
• * text => any : special syntax to indicate 0 or more fields that have a string key that maps to any value. Used to indicate a generic JSON object.¶

All timestamps and time-related values (e.g., offsets) in qlog are logged as float64 in the millisecond resolution.¶

Other qlog documents can define their own CDDL-compatible (struct) types (e.g., separately for each Packet type that a protocol supports).¶

1.1.2. Serialization examples

Serialization examples in this document use JSON ([JSON]) unless otherwise indicated.¶

3.2.1. time_offset

The time_offset field indicates by how many milliseconds the starting time of the current trace should be offset. This is useful when comparing logs taken from various systems, where clocks might not be perfectly synchronous. Users could use manual tools or automated logic to align traces in time and the found optimal offsets can be stored in this field for future usage. The default value is 0.¶

3.2.2. vantage_point

The vantage_point field describes the vantage point from which the trace originates, see Figure 8. Each trace MUST have a single vantage_point. By virtue of this, a single qlog file can only include events from multiple vantage_points if it includes multiple traces.¶

Definitions:¶

VantagePoint = {
    ? name: text
    type: VantagePointType
    ? flow: VantagePointType
}

; client = endpoint which initiates the connection
; server = endpoint which accepts the connection
; network = observer in between client and server
VantagePointType = "client" /
                   "server" /
                   "network" /
                   "unknown"

JSON serialization examples:¶

{
    "name": "aioquic client",
    "type": "client"
}

{
    "name": "wireshark trace",
    "type": "network",
    "flow": "client"
}

The flow field is only required if the type is "network" (for example, the trace is generated from a packet capture). It is used to disambiguate events like "packet sent" and "packet received". This is indicated explicitly because for multiple reasons (e.g., privacy) data from which the flow direction can be otherwise inferred (e.g., IP addresses) might not be present in the logs.¶

Meaning of the different values for the flow field: * "client" indicates that this vantage point follows client data flow semantics (a "packet sent" event goes in the direction of the server). * "server" indicates that this vantage point follow server data flow semantics (a "packet sent" event goes in the direction of the client). * "unknown" indicates that the flow's direction is unknown.¶

Depending on the context, tools confronted with "unknown" values in the vantage_point can either try to heuristically infer the semantics from protocol-level domain knowledge (e.g., in QUIC, the client always sends the first packet) or give the user the option to switch between client and server perspectives manually.¶

3.3.1. Timestamps

The "time" field indicates the timestamp at which the event occurred. Its value is typically the Unix timestamp since the 1970 epoch (number of milliseconds since midnight UTC, January 1, 1970, ignoring leap seconds). However, qlog supports two more succinct timestamps formats to allow reducing file size. The employed format is indicated in the "time_format" field, which allows one of three values: "absolute", "delta" or "relative".¶

Definition:¶

TimeFormat = "absolute" /
             "delta" /
             "relative"

• Absolute: Include the full absolute timestamp with each event. This approach uses the largest amount of characters. This is also the default value of the "time_format" field.¶
• Delta: Delta-encode each time value on the previously logged value. The first event in a trace typically logs the full absolute timestamp. This approach uses the least amount of characters.¶
• Relative: Specify a full "reference_time" timestamp (typically this is done up-front in "common_fields", see Section 3.3.8) and include only relatively-encoded values based on this reference_time with each event. The "reference_time" value is typically the first absolute timestamp. This approach uses a medium amount of characters.¶

The first option is good for stateless loggers, the second and third for stateful loggers. The third option is generally preferred, since it produces smaller files while being easier to reason about. An example for each option can be seen in Figure 13.¶

The absolute approach will use:
1500, 1505, 1522, 1588

The delta approach will use:
1500, 5, 17, 66

The relative approach will:
- set the reference_time to 1500 in "common_fields"
- use: 0, 5, 22, 88

One of these options is typically chosen for the entire trace (put differently: each event has the same value for the "time_format" field). Each event MUST include a timestamp in the "time" field.¶

Events in each individual trace SHOULD be logged in strictly ascending timestamp order (though not necessarily absolute value, for the "delta" format). Tools MAY sort all events on the timestamp before processing them, though are not required to (as this could impose a significant processing overhead). This can be a problem especially for multi-threaded and/or streaming loggers, who could consider using a separate post-processor to order qlog events in time if a tool do not provide this feature.¶

Timestamps do not have to use the UNIX epoch timestamp as their reference. For example for privacy considerations, any initial reference timestamps (for example "endpoint uptime in ms" or "time since connection start in ms") can be chosen. Tools SHOULD NOT assume the ability to derive the absolute Unix timestamp from qlog traces, nor allow on them to relatively order events across two or more separate traces (in this case, clock drift should also be taken into account).¶

3.3.2. Event Names

Events differ mainly in the type of metadata associated with them. The "name" field is an identifier that parsers can use to decide how to interpret the event metadata contained in the "data" field (see Section 3.3.3).¶

Event names indicate a category and type. The "name" field MUST contain a non-empty character sequence representing a category, followed by a colon (':'), followed by a non-empty character sequence representing a type.¶

Category allows a higher-level grouping of events per specific event type. For example for QUIC and HTTP/3, the different categories could be "quic", "http", "qpack", and "recovery". Within these categories, the event type provides additional granularity. For example for QUIC and HTTP/3, within the "quic" category, there would be "packet_sent" and "packet_received" events.¶

JSON serialization example:¶

{
    "name": "quic:packet_sent"
}

3.3.3. Data

The data field is a generic object. It contains the per-event metadata and its form and semantics are defined per specific sort of event. For example, data field value definitions for QUIC and HTTP/3 can be found in [QLOG-QUIC] and [QLOG-H3].¶

This field is defined here as a CDDL extension point (a "socket" or "plug") named $ProtocolEventBody. Other documents MUST properly extend this extension point when defining new data field content options to enable automated validation of aggregated qlog schemas.¶

The only common field defined for the data field is the trigger field, which is discussed in Section 3.3.5.¶

Definition:¶

; The ProtocolEventBody is any key-value map (e.g., JSON object)
; only the optional trigger field is defined in this document
$ProtocolEventBody /= {
    ? trigger: text
    * text => any
}
; event documents are intended to extend this socket by using:
; NewProtocolEvents = EventType1 /
;                     EventType2 /
;                     ... /
;                     EventTypeN
; $ProtocolEventBody /= NewProtocolEvents

One purely illustrative example for a QUIC "packet_sent" event is shown in Figure 16:¶

TransportPacketSent = {
    ? packet_size: uint16
    header: PacketHeader
    ? frames:[* QuicFrame]
    ? trigger: "pto_probe" /
               "retransmit_timeout" /
               "bandwidth_probe"
}

could be serialized as

{
    "packet_size": 1280,
    "header": {
        "packet_type": "1RTT",
        "packet_number": 123
    },
    "frames": [
        {
            "frame_type": "stream",
            "length": 1000,
            "offset": 456
        },
        {
            "frame_type": "padding"
        }
    ]
}

3.3.4. protocol_type

The "protocol_type" array field indicates to which protocols (or protocol "stacks") this event belongs. This allows a single qlog file to aggregate traces of different protocols (e.g., a web server offering both TCP+HTTP/2 and QUIC+HTTP/3 connections).¶

Definition:¶

ProtocolType = [+ text]

For example, QUIC and HTTP/3 events have the "QUIC" and "HTTP3" protocol_type entry values, see [QLOG-QUIC] and [QLOG-H3].¶

Typically however, all events in a single trace are of the same few protocols, and this array field is logged once in "common_fields", see Section 3.3.8.¶

3.3.5. Triggers

Sometimes, additional information is needed in the case where a single event can be caused by a variety of other events. In the normal case, the context of the surrounding log messages gives a hint as to which of these other events was the cause. However, in highly-parallel and optimized implementations, corresponding log messages might separated in time. Another option is to explicitly indicate these "triggers" in a high-level way per-event to get more fine-grained information without much additional overhead.¶

In qlog, the optional "trigger" field contains a string value describing the reason (if any) for this event instance occurring, see Section 3.3.3. While this "trigger" field could be a property of the qlog Event itself, it is instead a property of the "data" field instead. This choice was made because many event types do not include a trigger value, and having the field at the Event-level would cause overhead in some serializations. Additional information on the trigger can be added in the form of additional member fields of the "data" field value, yet this is highly implementation-specific, as are the trigger field's string values.¶

One purely illustrative example of some potential triggers for QUIC's "packet_dropped" event is shown in Figure 18:¶

TransportPacketDropped = {
    ? packet_type: PacketType
    ? raw_length: uint16
    ? trigger: "key_unavailable" /
               "unknown_connection_id" /
               "decrypt_error" /
               "unsupported_version"
}

3.3.6. group_id

As discussed in Section 3.2, a single qlog file can contain several traces taken from different vantage points. However, a single trace from one endpoint can also contain events from a variety of sources. For example, a server implementation might choose to log events for all incoming connections in a single large (streamed) qlog file. As such, a method for splitting up events belonging to separate logical entities is required.¶

The simplest way to perform this splitting is by associating a "group identifier" to each event that indicates to which conceptual "group" each event belongs. A post-processing step can then extract events per group. However, this group identifier can be highly protocol and context-specific. In the example above, the QUIC "Original Destination Connection ID" could be used to uniquely identify a connection. As such, they might add a "ODCID" field to each event. However, a middlebox logging IP or TCP traffic might rather use four-tuples to identify connections, and add a "four_tuple" field.¶

As such, to provide consistency and ease of tooling in cross-protocol and cross-context setups, qlog instead defines the common "group_id" field, which contains a string value. Implementations are free to use their preferred string serialization for this field, so long as it contains a unique value per logical group. Some examples can be seen in Figure 20.¶

Definition:¶

GroupID = text

JSON serialization example for events grouped by four tuples and QUIC connection IDs:¶

"events": [
    {
        "time": 1553986553579,
        "protocol_type": ["TCP", "TLS", "HTTP2"],
        "group_id": "ip1=2001:67c:1232:144:9498:6df6:f450:110b,
                   ip2=2001:67c:2b0:1c1::198,port1=59105,port2=80",
        "name": "quic:packet_received",
        "data": { ... }
    },
    {
        "time": 1553986553581,
        "protocol_type": ["QUIC","HTTP3"],
        "group_id": "127ecc830d98f9d54a42c4f0842aa87e181a",
        "name": "quic:packet_sent",
        "data": { ... }
    }
]

Note that in some contexts (for example a Multipath transport protocol) it might make sense to add additional contextual per-event fields (for example "path_id"), rather than use the group_id field for that purpose.¶

Note also that, typically, a single trace only contains events belonging to a single logical group (for example, an individual QUIC connection). As such, instead of logging the "group_id" field with an identical value for each event instance, this field is typically logged once in "common_fields", see Section 3.3.8.¶

3.3.7. system_info

The "system_info" field can be used to record system-specific details related to an event. This is useful, for instance, where an application splits work across CPUs, processes, or threads and events for a single trace occur on potentially different combinations thereof. Each field is optional to support deployment diversity.¶

Definition:¶

SystemInformation = {
  ? processor_id: uint32
  ? process_id: uint32
  ? thread_id: uint32
}

3.3.8. common_fields

As discussed in the previous sections, information for a typical qlog event varies in three main fields: "time", "name" and associated data. Additionally, there are also several more advanced fields that allow mixing events from different protocols and contexts inside of the same trace (for example "protocol_type" and "group_id"). In most "normal" use cases however, the values of these advanced fields are consistent for each event instance (for example, a single trace contains events for a single QUIC connection).¶

To reduce file size and making logging easier, qlog uses the "common_fields" list to indicate those fields and their values that are shared by all events in this component trace. This prevents these fields from being logged for each individual event. An example of this is shown in Figure 21.¶

JSON serialization with repeated field values
per-event instance:

{
    "events": [{
            "group_id": "127ecc830d98f9d54a42c4f0842aa87e181a",
            "protocol_type": ["QUIC","HTTP3"],
            "time_format": "relative",
            "reference_time": 1553986553572,

            "time": 2,
            "name": "quic:packet_received",
            "data": { ... }
        },{
            "group_id": "127ecc830d98f9d54a42c4f0842aa87e181a",
            "protocol_type": ["QUIC","HTTP3"],
            "time_format": "relative",
            "reference_time": 1553986553572,

            "time": 7,
            "name": "http:frame_parsed",
            "data": { ... }
        }
    ]
}

JSON serialization with repeated field values instead
extracted to common_fields:

{
    "common_fields": {
        "group_id": "127ecc830d98f9d54a42c4f0842aa87e181a",
        "protocol_type": ["QUIC","HTTP3"],
        "time_format": "relative",
        "reference_time": 1553986553572
    },
    "events": [
        {
            "time": 2,
            "name": "quic:packet_received",
            "data": { ... }
        },{
            "time": 7,
            "name": "http:frame_parsed",
            "data": { ... }
        }
    ]
}

The "common_fields" field is a generic dictionary of key-value pairs, where the key is always a string and the value can be of any type, but is typically also a string or number. As such, unknown entries in this dictionary MUST be disregarded by the user and tools (i.e., the presence of an unknown field is explicitly NOT an error).¶

The list of default qlog fields that are typically logged in common_fields (as opposed to as individual fields per event instance) are shown in the listing below:¶

Definition:¶

CommonFields = {
    ? time_format: TimeFormat
    ? reference_time: float64
    ? protocol_type: ProtocolType
    ? group_id: GroupID
    * text => any
}

Tools MUST be able to deal with these fields being defined either on each event individually or combined in common_fields. Note that if at least one event in a trace has a different value for a given field, this field MUST NOT be added to common_fields but instead defined on each event individually. Good example of such fields are "time" and "data", who are divergent by nature.¶

5.2.1. error

Importance: Core¶

Used to log details of an internal error that might not get reflected on the wire.¶

Definition:¶

GenericError = {
    ? code: uint64
    ? message: text
}

5.2.2. warning

Importance: Base¶

Used to log details of an internal warning that might not get reflected on the wire.¶

Definition:¶

GenericWarning = {
    ? code: uint64
    ? message: text
}

5.2.3. info

Importance: Extra¶

Used mainly for implementations that want to use qlog as their one and only logging format but still want to support unstructured string messages.¶

Definition:¶

GenericInfo = {
    message: text
}

5.2.4. debug

Importance: Extra¶

Used mainly for implementations that want to use qlog as their one and only logging format but still want to support unstructured string messages.¶

Definition:¶

GenericDebug = {
    message: text
}

5.2.5. verbose

Importance: Extra¶

Used mainly for implementations that want to use qlog as their one and only logging format but still want to support unstructured string messages.¶

Definition:¶

GenericVerbose = {
    message: text
}

5.3.1. scenario

Importance: Extra¶

Used to specify which specific scenario is being tested at this particular instance. This supports, for example, aggregation of several simulations into one trace (e.g., split by group_id).¶

Definition:¶

SimulationScenario = {
    ? name: text
    ? details: {* text => any }
}

5.3.2. marker

Importance: Extra¶

Used to indicate when specific emulation conditions are triggered at set times (e.g., at 3 seconds in 2% packet loss is introduced, at 10s a NAT rebind is triggered).¶

Definition:¶

SimulationMarker = {
    ? type: text
    ? message: text
}

6.1.1. I-JSON

For some use cases, it should be taken into account that not all popular JSON parsers support the full JSON format. Especially for parsers integrated with the JavaScript programming language (e.g., Web browsers, NodeJS), users are recommended to stick to a JSON subset dubbed [I-JSON] (or Internet-JSON).¶

One of the key limitations of JavaScript and thus I-JSON is that it cannot represent full 64-bit integers in standard operating mode (i.e., without using BigInt extensions), instead being limited to the range of [-(2**53)+1, (2**53)-1]. In these circumstances, Appendix E of [CDDL] recommends defining new CDDL types for int64 and uint64 that limit their values to this range.¶

While this can be sensible and workable for most use cases, some protocols targeting qlog serialization (e.g., QUIC, HTTP/3), might require full uint64 variables in some (rare) circumstances. In these situations, it should be allowed to also use the string-based representation of uint64 values alongside the numerical representation. Concretely, the following definition of uint64 should override the original and (web-based) tools should take into account that a uint64 field can be either a number or string.¶

uint64 = text /
         uint .size 8

6.1.2. Truncated values

For some use cases (e.g., limiting file size, privacy), it can be necessary not to log a full raw blob (using the hexstring type) but instead a truncated value (for example, only the first 100 bytes of an HTTP response body to be able to discern which file it actually contained). In these cases, the original byte-size length cannot be obtained from the serialized value directly.¶

As such, all qlog schema definitions SHOULD include a separate, length-indicating field for all fields of type hexstring they specify, see for example Section 5.1. This not only ensures the original length can always be retrieved, but also allows the omission of any raw value bytes of the field completely (e.g., out of privacy or security considerations).¶

To reduce overhead however and in the case the full raw value is logged, the extra length-indicating field can be left out. As such, tools MUST be able to deal with this situation and derive the length of the field from the raw value if no separate length-indicating field is present. The main possible permutations are shown by example in Figure 32.¶

// both the full raw value and its length are present
// (length is redundant)
{
    "raw_length": 5,
    "raw": "051428abff"
}

// only the raw value is present, indicating it
// represents the fields full value the byte
// length is obtained by calculating raw.length / 2
{
    "raw": "051428abff"
}

// only the length field is present, meaning the
// value was omitted
{
    "raw_length": 5,
}

// both fields are present and the lengths do not match:
// the value was truncated to the first three bytes.
{
    "raw_length": 5,
    "raw": "051428"
}

6.2.1. Supporting JSON Text Sequences in tooling

Note that JSON Text Sequences are not supported in most default programming environments (unlike normal JSON). However, several custom JSON-SEQ parsing libraries exist in most programming languages that can be used and the format is easy enough to parse with existing implementations (i.e., by splitting the file into its component records and feeding them to a normal JSON parser individually, as each record by itself is a valid JSON object).¶

6.3.1. Data structure optimizations

The first general category of optimizations is to alter the representation of data within an JSON(-SEQ) qlog file to reduce file size.¶

The first option is to employ a scheme similar to the CSV (comma separated value [RFC4180]) format, which utilizes the concept of column "headers" to prevent repeating field names for each datapoint instance. Concretely for JSON qlog, several field names are repeated with each event (i.e., time, name, data). These names could be extracted into a separate list, after which qlog events could be serialized as an array of values, as opposed to a full object. This approach was a key part of the original qlog format (prior to draft-02) using the "event_fields" field. However, tests showed that this optimization only provided a mean file size reduction of 5% (100MB to 95MB) while significantly increasing the implementation complexity, and this approach was abandoned in favor of the default JSON setup. Implementations using this format should not employ a separate file extension (as it still uses JSON), but rather employ a new value of "JSON.namedheaders" (or "JSON-SEQ.namedheaders") for the "qlog_format" field (see Section 3).¶

The second option is to replace field values and/or names with indices into a (dynamic) lookup table. This is a common compression technique and can provide significant file size reductions (up to 50% in tests, 100MB to 50MB). However, this approach is even more difficult to implement efficiently and requires either including the (dynamic) table in the resulting file (an approach taken by for example Chromium's NetLog format) or defining a (static) table up-front and sharing this between implementations. Implementations using this approach should not employ a separate file extension (as it still uses JSON), but rather employ a new value of "JSON.dictionary" (or "JSON-SEQ.dictionary") for the "qlog_format" field (see Section 3).¶

As both options either proved difficult to implement, reduced qlog file readability, and provided too little improvement compared to other more straightforward options (for example Section 6.3.2), these schemes are not inherently part of qlog.¶

6.3.2. Compression

The second general category of optimizations is to utilize a (generic) compression scheme for textual data. As qlog in the JSON(-SEQ) format typically contains a large amount of repetition, off-the-shelf (text) compression techniques typically succeed very well in bringing down file sizes (regularly with up to two orders of magnitude in tests, even for "fast" compression levels). As such, utilizing compression is recommended before attempting other optimization options, even though this might (somewhat) increase processing costs due to the additional compression step.¶

The first option is to use GZIP compression ([RFC1952]). This generic compression scheme provides multiple compression levels (providing a trade-off between compression speed and size reduction). Utilized at level 6 (a medium setting thought to be applicable for streaming compression of a qlog stream in commodity devices), gzip compresses qlog JSON files to 7% of their initial size on average (100MB to 7MB). For this option, the file extension .(s)qlog.gz SHOULD BE used. The "qlog_format" field should still reflect the original JSON formatting of the qlog data (e.g., "JSON" or "JSON-SEQ").¶

The second option is to use Brotli compression ([RFC7932]). While similar to gzip, this more recent compression scheme provides a better efficiency. It also allows multiple compression levels. Utilized at level 4 (a medium setting thought to be applicable for streaming compression of a qlog stream in commodity devices), brotli compresses qlog JSON files to 7% of their initial size on average (100MB to 7MB). For this option, the file extension .(s)qlog.br SHOULD BE used. The "qlog_format" field should still reflect the original JSON formatting of the qlog data (e.g., "JSON" or "JSON-SEQ").¶

Other compression algorithms of course exist (for example xz, zstd, and lz4). The gzip and brotli are recommended because of their tweakable behaviour and wide support in web-based environments, which is envisioned as the main tooling ecosystem (see also Section 8).¶

6.3.3. Binary formats

The third general category of optimizations is to use a more optimized (often binary) format instead of the textual JSON format. This approach inherently produces smaller files and often has better (de)serialization performance. However, the resultant files are no longer human readable and some formats require hard tradeoffs between flexibility for performance.¶

The first option is to use the CBOR (Concise Binary Object Representation [RFC7049]) format. For the purposes of qlog, CBOR can be viewed as a straightforward binary variant of JSON. As such, existing JSON qlog files can be trivially converted to and from CBOR (though slightly more work is needed for JSON-SEQ qlogs to convert them to CBOR-SEQ, see [RFC8742]). While CBOR thus does retain the full qlog flexibility, it only provides a 25% file size reduction (100MB to 75MB) compared to textual JSON(-SEQ). As CBOR support in programming environments is not as widespread as that of textual JSON and the format lacks human readability, CBOR was not chosen as the default qlog format. For this option, the file extension .(s)qlog.cbor SHOULD BE used. The "qlog_format" field should still reflect the original JSON formatting of the qlog data (e.g., "JSON" or "JSON-SEQ"). The media type should indicate both whether JSON or JSON Text Sequences are used, as well as whether CBOR or CBOR Sequences are used (see the table below).¶

A second option is to use a more specialized binary format, such as Protocol Buffers (protobuf). This format is battle-tested, has support for optional fields and has libraries in most programming languages. Still, it is significantly less flexible than textual JSON or CBOR, as it relies on a separate, pre-defined schema (a .proto file). As such, it it not possible to (easily) log new event types in protobuf files without adjusting this schema as well, which has its own practical challenges. As qlog is intended to be a flexible, general purpose format, this type of format was not chosen as its basic serialization. The lower flexibility does lead to significantly reduced file sizes. A straightforward mapping of the qlog main schema and QUIC/HTTP3 event types to protobuf created qlog files 24% as large as the raw JSON equivalents (100MB to 24MB). For this option, the file extension .(s)qlog.protobuf SHOULD BE used. The "qlog_format" field should reflect the different internal format, for example: "qlog_format": "protobuf".¶

Note that binary formats can (and should) also be used in conjunction with compression (see Section 6.3.2). For example, CBOR compresses well (to about 6% of the original textual JSON size (100MB to 6MB) for both gzip and brotli) and so does protobuf (5% (gzip) to 3% (brotli)). However, these gains are similar to the ones achieved by simply compression the textual JSON equivalents directly (7%, see Section 6.3.2). As such, since compression is still needed to achieve optimal file size reductions event with binary formats, the more flexible compressed textual JSON options are likely a better default for the qlog format in general.¶

6.3.4. Overview and summary

In summary, textual JSON was chosen as the main qlog format due to its high flexibility and because its inefficiencies can be largely solved by the utilization of compression techniques (which are needed to achieve optimal results with other formats as well).¶

Still, qlog implementers are free to define other qlog formats depending on their needs and context of use. These formats should be described in their own documents, the discussion in this document mainly acting as inspiration and high-level guidance. Implementers are encouraged to add concrete qlog formats and definitions to the designated public repository.¶

The following table provides an overview of all the discussed qlog formatting options with examples:¶