First SSAC Organizational Review

The First SSAC Organizational Review was initiated in June 2008 and completed in June 2010, with the implementation of improvements continuing through 2011.

Background

Article 4.4 of the ICANN Bylaws requires periodic review of all supporting organizations and advisory committees, as well as the Nominating Committee. The bylaws state three objectives for the review:

1. to determine whether that organization, council or committee has a continuing purpose in the ICANN structure;
2. if so, whether any change in structure or operations is desirable to improve its effectiveness; and
3. whether that organization, council or committee is accountable to its constituencies, stakeholder groups, organizations and other stakeholders.

Organizational reviews are conducted by independent examiners, selected through a competitive bidding process. The independent examiner works in consultation with a working group assembled by the board, who will act as implementation shepherds once the final report of the independent examiner is submitted. The review parameters are set by the ICANN Board, and those parameters as well as other avenues of inquiry are typically included in the request for proposals (RFP) for independent examiners. Reviews can take anywhere from three to five years to complete. The full review process includes seven phases, including the implementation of recommendations from the review. Reviews must be conducted at least every five years, measuring from the date that the final report of the previous review was accepted by the ICANN Board. The Security and Stability Advisory Committee is one of the organizations subject to the review requirements of Article 4.4.

Initiation

The ICANN Board formed the SSAC1 Review Working Group (RWG) in June 2008, naming the following members to the RWG: Dennis Jennings (Chair), Robert Blokzijl, Reinhard Scholl and Suzanne Woolf. In September 2008, the board posted an RFP for an independent reviewer to conduct the review of the SSAC. The Terms of Reference (ToR) for the RFP included 30 questions related to the two principal questions in the Article 4.4 of the ICANN Bylaws at the time: does the SSAC have a continuing purpose in the ICANN structure; and is there any change in structure or operations that could improve the SSAC's effectiveness?

At a special meeting in October 2008, the board approved the charter for the RWG.

Independent Examiner

In the lead-up to ICANN 33 in Cairo, JAS Communications was selected as the independent examiner for SSAC1. This permitted JAS the opportunity to begin work at the meeting.

Methodology

JAS's standard approach to reviews of technical organizations was put to the test, as the IE's draft final report makes clear:

Data collection began in Cairo and it quickly became apparent that organizational issues would dominate our study. With so much uncertainty around scope and mission, we found it difficult to apply typical group performance metrics because there was no baseline in terms of a clear charter, specific tasking, or even agreed upon strategic direction. Put another way, it is difficult to evaluate the performance of a group when it is not clear what they are supposed to be doing. It was very clear that SSAC was working diligently, but JAS was asked whether the work it was doing was the right work.

JAS determined that the assessment would largely be based on qualitative findings. In order to ensure that there was a basis for community input on quantifiable performance metrics, JAS worked with an organizational behavioral expert to create a "custom survey instrument to produce quantitative data around specific recurring organizational questions unearthed during the interviews." The survey was deployed with an eye toward verifying qualitative observations of the review team, and to provide a dataset against which demographic and cultural correlations could be tested.

After a series of rounds of hypothesis testing and data collection, JAS submitted a draft version of its findings, analysis, and recommendations to selected individuals within the ICANN community for feedback.

IE Draft Final Report

Following receipt of feedback from the limited-distribution draft, JAS published its Draft Final Report for public comment on February 16, 2009. The IE's draft final report identified three core themes within their findings:

• Lack of organizational clarity and charter;
• Lack of formality leading to concerns about transparency; and
• Perceived and actual conflicts of interest.

In particular, the longstanding relationships of SSAC members, most of which predated ICANN involvement, led to a collegial and informal culture that affected both the outside perception of the SSAC and internal and external understanding of the SSAC's role within the ICANN ecosystem. As an example, JAS noted that policies and procedures around conflicts of interest were nearly nonexistent:

An unreleased internal draft SSAC policies and procedures document provided to JAS contains the following language regarding perceived and actual conflicts of interest:

The committee does not ordinarily concern itself with conflicts of interest. All members are always permitted to participate in all activities. However, the committee may elect to state potential conflicts of interest as an integral part of any publication if it deems this to enhance the final result. An individual may elect not to participate in any activity of the committee at his or her own discretion.

We observe that this is an accurate characterization of SSAC's current views on perceived or actual conflicts of interest; in practice, we find little to no formal attention to the issue.

Despite the perceptual issues, the importance and quality of the SSAC's work was universally acknowledged. The executive summary of the draft final report stated that the SSAC was "functioning, functioning well, and filling a relevant purpose." In addition, the report called out the community's appreciation and respect for the SSAC's handling of the Verisign Site Finder situation. The draft report offered thirty-four recommendations to improve organizational and communication clarity inside the SSAC and in cross-community work, address gaps in formalized policy or procedures within the SSAC, and generally alleviate the "growing pains" experienced by the SSAC in the first years of its existence.

Public Comment on Draft Final Report

JAS presented its findings and recommendations in a workshop session at ICANN 34 in Mexico City. The comments receieved at the session were largely from current or former members of the SSAC, and expressed varying degrees of skepticism about the value and clarity of the findings and recommendations.

In addition to the session in Mexico City, two written comments were received on the draft report during the public comment period. Hiro Hotta expressed admiration for the SSAC and suggested that diverse viewpoints might even further improve the SSAC's abilities. ISOC submitted a statement in agreement with many of the recommendations, particularly the implementation of a conflicts of interest policy, and emphasizing focus on SSAC's core mission as an advisory committee.

IE Final Report

JAS submitted its final report on May 15, 2009. The final report acknowledged the comments received, as well as feedback from the RWG and other sources in the time between the draft final report and the final report.SSAC1 Review - IE Final Report, May 15, 2009 (PDF)</ref> JAS noted that the feedback and input had resulted in several changes and refinements, including a stronger focus on the big picture issues presented by the data, and the simplification of some of the more heavily detailed recommendations in favor of addressing those big picture issues. The findings and recommendations of the final report were largely the same.

Public Comment on the IE's Final Report

The final report received only one substantive written comment, which encouraged the SSAC to work more closely with other constituencies and "improve its overall transparency."

Review Working Group Reports

After receipt of the IE's final report, the SSAC membership engaged in a self-assessment exercise, resulting in a report to the RWG in June 2009 During ICANN 35 in Sydney, the RWG discussed the final report from JAS as well as the SSAC self-assessment report, and began to formulate proposals for the implementation of improvements recommended in each report.

The RWG collated the recommendations, the SSAC's response, and the working group's consensus opinion on each recommendation from the IE's final report, as well as the additional recommendations from the SSAC Self-Assessment: