Is my web archive usable?

by Fernando Melo and Daniel Gomes, Arquivo.pt – the Portuguese Web Archive

Arquivo.pt has preserved pages from the Web archived since 1996 and provided a public search service to access such information. Fulfilling Arquivo.pt goals relies on the continuous quality assurance of the provided service.

Figure 1: Desktop version of the ReplayBar User Interface of Arquivo.pt

A new version of Arquivo.pt was released on May 2016. This release was based on PyWb wayback software and included major design changes in the user interfaces for the replay of archived Web pages (ReplayBar), namely:

• a drill-down navigation bar that enables fast browsing across several archived versions of a given web page;
• a top banner that enables the users to take full-page screenshots, print and share archived Web pages.

We conducted usability testing sessions for quality assurance. The obtained results showed an effectiveness average rate of 91% and satisfaction of 80%. Usability testing showed that the new user interfaces were well-accepted by participants of the test which enabled us to identify and prioritise issues in order to plan future developments.

Methodology

The methodology applied for usability testing was based on the observation and registration of user behaviours while performing designated tasks. A pre-test was first conducted to identify issues with task descriptions.

Sessions were conducted by a facilitator and notes registered by an observer. Each session began with an explanation of its purpose by the facilitator who also asked the user to perform the tasks in the most natural way possible while thinking-aloud.

The users were asked to perform the tasks using their own favourite device (e.g. tablet, smartphone, laptop). The objective was to maximise the detection of unforeseen compatibility issues and to prevent difficulties stemming from the unfamiliarity with the test device.

Figure 2: Tablet version of the ReplayBar User Interface of Arquivo.pt

Figure 3: Smartphone version of the ReplayBar User Interface of Arquivo.pt

The observer took notes of the difficulties and comments from the users. Rates were assigned to the effectiveness while performing each task:

• 0: The user could not complete the task;
• 1: The user completed the task with difficulties or help;
• 2: The user completed the task without any difficulty.

At the end of each session, the users were asked to fill an anonymous satisfaction questionnaire.

Devices and Tasks

The users chose to use the following combinations of device, operating system and web browser:

• Laptop w/ Windows 8 + Chrome Web Browser
• IPhone 6 plus w/ iOS + Safari Web Browser
• Tablet w/Android + Chrome Web Browser
• Laptop w/ Windows 10 + Chrome Web Browser
• Laptop w/ MacOS 10 + Chrome Web Browser

The users were asked to execute the following tasks:

1. Find in Arquivo.pt the website of the school/university where you studied.
2. Check if there is an archived version of the site when you attended the school/university. Write the date of this version.
3. Write the date of the earliest archived version of the Website of your school/university.
4. Share this version with your former colleagues.
5. Print an archived Web page of your choice to show at home.
6. You were asked to write an article about your old school/ university. Save the home page of that site as image to be included in your article.
7. Use Arquivo.pt freely and search for something you find interesting.

Effectiveness and Satisfaction results

Table 1. Tasks and average effectiveness rate for their completion.

Our results show that the users were able to browse and perform the defined tasks with an average effectiveness rate of 91% (Table 1). Each usability session ended with the completion of a satisfaction questionnaire, where the users were asked to assess the degree of agreement with the following statements:

Table 2. Results of the Satisfaction Questionnaire.

The overall average satisfaction was 80%. The statements to which users most agreed were “The information provided for the system was easy to understand” (86%), and “This system has all the functions and capabilities I expect it to have” (86%). The statement “The organization of information on the system screens was clear” was the one with a lower agreement percentage (69%).

Lessons learned

The obtained results helped us identify the issues to be addressed in the future developments. For instance, regarding the new ReplayBar user interface:

• the meaning of the magnifying glass icon seems confusing to the users;
• the size and contrast of some texts needs to be increased;
• a progress indicator after user actions becomes crucial in low-bandwidth conditions;
• users should be able to select the file location of the generated screenshots;
• the function “Search in other archives”, that links to the Memento Time Travel portal when the users reach a content that was not preserved by Arquivo.pt, needs to be emphasised.

Figure 4. Arquivo.pt links to the Memento Time Travel portal when users browse an unavailable content so that they can find it in other web archives.

It was very important to allow users to perform the usability tasks with their own devices because it enabled us to test a wider variety of Web Browsers and Operating Systems and detect unforeseen compatibility issues. We identified, for instance, that the new ReplayBar used a CSS element that prevented it from working on any iOS device, which was a major drawback that had remained undetected until usability testing. The definition of broad but realistic tasks for the usability sessions, even if we could not have anticipated the archived pages that the users would access, was fundamental to avoid influencing user answers, observe unexpected behaviours, as well as, understanding user’s  expectations and difficulties.

Once again, the obtained results showed that usability testing is a mandatory step in the quality assurance of a web archive. Not performing usability testing is an unnecessary risk that jeopardises the success of the significant investments required to build a web archive.

Find out more

• Usability Report for the ReplayBar version 1.0 of Arquivo.pt, Fernando Melo and Daniel Gomes, 2016
• Arquivo.pt: search pages from the past, FCT-FCCN, 2016
• IBM Computer Usability Satisfaction Questionnaires: Psychometric Evaluation and Instructions for Use, James R. Lewis, 1995
• Usability.gov: Improving the User Experience, U.S. Department of Health & Human Services, 2016

Visit us at the Web Archiving Conference 2017 in Lisbon!

Update:

The new version of Arquivo.pt named Hercules presents design improvements and addresses usability issues detected outlined above. For more information, see: http://sobre.arquivo.pt/news-1/news/arquivo.pt-new-version