| Internet-Draft | Document Review and AD Workload | March 2023 |
| Nottingham | Expires 1 October 2023 | [Page] |
Arguably, IETF Area Directors are overloaded with document review duties. This document surveys the relevant background, discusses the implications, and makes a proposal for improvements.¶
This note is to be removed before publishing as an RFC.¶
Status information for this document may be found at https://datatracker.ietf.org/doc/draft-nottingham-iesg-review-workload/.¶
information can be found at https://mnot.github.io/I-D/.¶
Source for this draft and an issue tracker can be found at https://github.com/mnot/I-D/labels/ad-workload.¶
This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.¶
Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.¶
Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."¶
This Internet-Draft will expire on 1 October 2023.¶
Copyright (c) 2023 IETF Trust and the persons identified as the document authors. All rights reserved.¶
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License.¶
The job of an IETF Area Director is notoriously difficult. Beyond the impact on the individual, this reputation is widely thought to reduce the pool of potential candidates for the position, resulting in a lack of diversity, vulnerability to failure to find a candidate, or the possibility of having to accept a poor candidate. See Section 2.6.2 of [RFC3774] for further discussion.¶
One of the key responsibilities of an Area Director is reviewing each document that comes to the IESG for publication. Although Area Directors do much more than simply review documents, the sheer volume of pages that the IETF publishes makes this a significant component of the job, in terms of both their time and attention.¶
Section 2 surveys relevant background materials. Section 3 discusses the requirements for document review in the IETF process, and Section 4 makes a proposal to modify the Area Director's role in it, in order to make it possible for more people to consider putting their hands up to fill the position.¶
The responsibility of an Area Director to review all documents being considered for publication originates in Section 6.1.2 of [RFC2026], which describes the responsibilities of the IESG when reviewing a specification for approval:¶
The IESG shall determine whether or not a specification submitted to it according to section 6.1.1 satisfies the applicable criteria for the recommended action (see sections 4.1 and 4.2), and shall in addition determine whether or not the technical quality and clarity of the specification is consistent with that expected for the maturity level to which the specification is recommended.¶
The criteria referred to regard the maturity of the document as indicated by its stability, its resolution of design choices, level of community review, implementation and operational experience.¶
Section 5 of [RFC3710] states that:¶
The IESG is expected to ensure that the documents are of a sufficient quality for release as RFCs, that they describe their subject matter well, and that there are no outstanding engineering issues that should be addressed before publication. The degree of review will vary with the intended status and perceived importance of the documents.¶
However, that document notes that "[i]t does not claim to represent consensus of the IETF" but rather "was written as a 'documentation of existing practice'".¶
The IESG has created internal policies to ensure that this goal of sufficient review is achieved. The IESG Ballot Procedures document describes the system used. Notably, the description of the "No Objection" position includes this statement:¶
This ballot position may be interpreted as "This is outside my area of expertise or have no cycles", in that you exercise the ability to move a document forward on the basis of trust towards the other ADs.¶
This language implies that Area Directors need not read every page of every document; it is an "escape clause" for an overloaded AD.¶
Expectations for document review are also set by the job descriptions used by the NOMCOM. The General IESG Member Expertise desired by the 2023 NOMCOM includes:¶
An AD should be able to personally review every Internet-Draft that they sponsor. For other Internet-Drafts an AD needs to be satisfied that adequate review has taken place, though many ADs personally review these documents as well.¶
However, it later sets a more onerous expectation:¶
Basic IESG activities can consume significant time during a typical non-meeting week. Enough time must be allocated to [...] read on the order of 500 pages of internet-drafts every two weeks [...]¶
... which implies that they should be reading every draft.¶
IESG document review undeniably serves an important function: maintaining the output quality of IETF specifications, by making Area Directors both responsible for document review and accountable to the community (through transparency mechanisms like the Open Mic session at plenaries, and ultimately through the risk of appeal and recall). This accountability and quality enhances the legitimacy of the IETF's work, and ultimately, its success.¶
However, the expectations for review are not clearly stated: while there are clear affordances in the IESG-internal policies and in the NOMCOM job description for an AD to skip a detailed review of the document, many IETF participants and Area Directors alike seem to believe that ADs have a fundamental responsibility to review every document published for any concerns they may have -- to "read on the order of 500 pages [...] every two weeks."¶
These conflicting and ill-stated expectations arguably have the effect of discouraging many qualified candidates from applying for Area Director positions.¶
A solution should:¶
These requirements rule out many potential solutions. For example, allowing Area Directors to delegate their reviews to Directorates would be seen to harm the accountability relationship, since the person reviewing the document would no longer be directly accountable to the community.¶
On the other hand, it is clear that Area Directors are not expected to be expert in every technology that crosses their doorstep; per the 2023 NOMCOM guidelines:¶
An AD need not be the ultimate expert and authority in any technical area. The abilities to manage, to guide and judge consensus, to know who the subject-matter experts are and to seek their advice, and to mentor other IETF participants to take the technical lead is at least as important as their own technical abilities.¶
This implies that an Area Director might rely on others in their review responsibilities, but cannot avoid responsibility for them.¶
But what is that responsibility, exactly? Clearly, they are not exercising only their own judgement; an Area Director who refused documents based only upon their personal beliefs would effectively be a tyrant.¶
This is supported by the materials. At their core, the desired criteria listed in Sections 4.1 and 4.2 of [RFC2026] are all fairly objective; for example:¶
A Proposed Standard specification is generally stable, has resolved known design choices, is believed to be well-understood, has received significant community review, and appears to enjoy enough community interest to be considered valuable.¶
Any proposal, then, should also be focused on assuring these properties.¶
This document makes one proposal below, as a starting point for discussion; others might also meet these requirements.¶
To clarify the nature and role of Area Director reviews and thereby partially address workload issues as described in [RFC3774], this document recommends that the policy described below be adopted, either as an IESG Statement or through IETF Consensus. Once that takes place, supporting material (such as the NOMCOM job descriptions) should be clarified and aligned with it.¶
This policy sets the expectation that Area Directors are responsible for assuring that, from the perspective of their Area, documents being considered for publication on the IETF stream meet the requirements in Section 4 of [RFC2026]. This implies that an Area Director can ballot "No Objection" for a document that they judge to have no implications for their Area without further review. Furthermore, they need only review those portions of other documents that do have implications for their Area.¶
When reviewing a document, Area Directors may rely on expertise of others to judge the desired properties; they need not be expert in every technology in their Area. However, in doing so, they do not avoid responsibility for meeting the requirements stated in [RFC2026], and they may be held accountable if those requirements are not met, from the perspective of their Area.¶
This policy does not prevent an Area Director from exceeding these expectations. However, Area Director reviews should be based in the requirements of [RFC2026], as elaborated upon by the DISCUSS Criteria.¶
This policy is not a very large change from current practice of at least some ADs, based upon discussions I've had. As such, its most important function is to level-set expectations between the community and the IESG.¶
Practically, this allows an AD to delegate their review (or partially do so) to someone that they judge as appropriate, based upon their expertise. However, they cannot avoid responsibility -- which means that delegating to a review directorate of volunteers may be unwise.¶
Overall, the expectation is that ADs can (and perhaps should) rely on other experts in their area more than they do the reviewing themselves. They are responsible for understanding and managing any objections that come from the experts they rely on, and are expected to decide the relative importance of actually requiring that any issue raised by such an expert be addressed.¶
This document has no actions for IANA.¶
Undoubtedly changing how the IESG reviews documents has potential security implications. Caveat emptor.¶
Thanks to Pete Resnick for his thoughts and a snippet of text.¶