Results of the Web Archiving API Survey of IIPC Members

If you attended the recent GA, or read some of the many blog posts about it, you probably heard about the potential benefits of standardized web archiving APIs. This was a common theme that came up in multiple presentations and informal discussions. During a conversation over lunch mid-week one person suggested that the IIPC form a new working group to focus on web archiving APIs. Clearly some institutions were interested in this, but how many? And are they interested enough to participate in a new working group? A group of us at Harvard decided to find out. We developed a short survey and advertised it on the IIPC mailing list.

The survey was open from May 14 through June 1 and was filled out 18 times, by 17 different institutions from 8 different countries.

Country Institutions
Czech Republic National Library of the Czech Republic
France Bibliothèque nationale de France
Iceland National and University Library of Iceland
New Zealand National Library of New Zealand
Spain National Library of Spain
United Kingdom The British Library, The National Archives
United States Stanford University Libraries, Old Dominion University, Internet Archive, LANL, California Digital Library, Harvard Library, Library of Congress, UCLA, University of North Texas

Table 1: The institutions that responded to the survey

The survey asked “Is the topic of web archiving APIs of interest to your institution?” The answer was overwhelmingly “Yes”. All 17 institutions are interested in web archiving APIs. Personally this was the first unanimous survey question I have ever seen.

api1Figure 1: A rare unanimous response

When asked “Why are web archiving APIs of interest to your institution?” the responses (see Figure 2) were varied but had common themes. Many of the reasons were from the perspective of an institution providing or maintaining web archiving programs or infrastructure, for example:

  • “The sustainability of our program depends on the web archiving community as a whole better aligning itself to collaboratively maintain and augment a core set of interoperable systems…”
  • “…appreciate how this would reduce our technical spend in the long term…”
  • “… APIs should ease the maintenance and evolution of the complex set of tools we are using to complete the document cycle: selection, collect, access and preservation.”

Another common response was from the perspective of providing a better service for researchers, for example:

  • “…a common/standard API would make it easier for researchers to work with multiple web archives with standard methodologies.”
  • “To help researchers explore our collection, including within our catalogue system, to link with other web archive collections and potentially to interface with different components of our infrastructure.”
  • “We often do aggregation and want to have a way to archive resources of interest with the help of scripts, in both of these cases an API would be ideal.”

api2Figure 2: A word cloud generated from the free text responses to why the institution is interested in web archiving APIs [Used Word Cloud Generator by Jason Davies]

The respondents were asked “If we organized a new working group within the IIPC to work on web archiving APIs would your institution be willing to participate?” All but one institution said “Yes”. The institution that said “No” said that they were interested but did not have enough staff resources currently to actively participate.

api3Figure 3: Most of the institutions are willing to participate in the new working group.

We asked “In what specific ways could your institution participate? Please select all that apply.” The results are shown in Table 2. Most of the respondents would like to help define the functional requirements, but a good amount would also like to contribute use cases and help design the technical details. Importantly, there are institutions willing to help run the meetings.

Specific Way % of Respondents Count of Respondents
Help define the functional requirements for a web archiving API 94% 15
Contribute curatorial, researcher or management requirements and use cases 81% 13
Help design the technical details of a web archiving API 69% 11
Help schedule and run the working group meetings 19% 3
Other* 6% 1

Table 2: The specific ways institutions would participate in the working group
* One institution said that they would be willing to implement and test web archiving APIs where appropriate and aligned with local needs

So the answer to our original question is a clear YES! There are enough IIPC institutions that are interested and willing to participate in meaningful ways in this new working group. Stay tuned while we work through the logistics of how to start. One of the first steps will be to identify co-chairs for the group. If you are interested in this please let me know! And thanks everyone for taking the time to fill out this survey.

By Andrea Goethals, Manager of Digital Preservation and Repository Services, Harvard Library

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s