By Youssef Eldakar of Bibliotheca Alexandrina
We are pleased to invite the web archiving community to visit LinkGate at linkgate.bibalex.org.
LinkGate is scalable web archive graph visualization. The project was launched with funding by the IIPC in January 2020. During the term of this round of funding, Bibliotheca Alexandrina (BA) and the national Library of New Zealand (NLNZ) partnered together to develop core functionality for a scalable graph visualization solution geared towards web archiving and to compile an inventory of research use cases to guide future development of LinkGate.
What does LinkGate do?
LinkGate seeks to address the need to visualize data stored in a web archive. Fundamentally, the web is a graph, where nodes are webpages and other web resources, and edges are the hyperlinks that connect web resources together. A web archive introduces the time dimension to this pool of data and makes the graph a temporal graph, where each node has multiple versions according to the time of capture. Because the web is big, web archive graph data is big data, and scalability of a visualization solution is a key concern.
APIs and use cases
We developed a scalable graph data service that exposes temporal graph data via an API, a data collection tool for feeding interlinking data extracted from web archive data files into the data service, and a web-based frontend for visualizing web archive graph data streamed by the data service. Because this project was first conceived to fulfill a research need, we reached out to the web archive community and interviewed researchers to identify use cases to guide development beyond core functionality. Source code for the three software components, link-serv, link-indexer, and link-viz, respectively, as well as the use cases, are openly available on GitHub.
Using LinkGate
An instance of LinkGate is deployed on Bibliotheca Alexandrina’s infrastructure and accessible at linkgate.bibalex.org. Insertion of data into the backend data service is ongoing. The following are a few screenshots of the frontend:
Please see the project’s IIPC Discretionary Funding Program (DFP) 2020 final report for additional details.
We will presenting about the project at the upcoming IIPC Web Archiving Conference on Tuesday, 15 June 2021 and also share the results of our work at an Research Speakers Series webinars on 28 July. If you have any questions or feedback, please contact the LinkGate Team at linkgate[at]iipc.simplelists.com.
Next steps
This development phase of Project LinkGate has been for the core functionality of a scalable, modular graph visualization environment for web archive data. Our team shares a common passion for this work and we remain committed to continuing to build up the components, including:
- Improved scalability
- Design and development of the plugin API to support the implementation of add-on finders and vizors (graph exploration tools)
- Enriched metadata
- Integration of alternative data stores (e.g., the Solr index in SolrWayback, so that data may be served by link-serv to visualize in link-viz or Gephi)
- Improved implementation of the software in general.
BA intends to maintain and expand the deploymentat linkgate.bibalex.org on a long-term basis.
Acknowledgements
The LinkGate team is grateful to the IIPC for providing the funding to get the project started and develop the core functionality. The team is passionate about this work and is eager to carry on with development.
LinkGate Team
- Lana Alsabbagh, NLNZ, Research Use Cases
- Youssef Eldakar, BA, Project Coordination
- Mohammed Elfarargy, BA, Link Visualizer (link-viz) & Development Coordination
- Mohamed Elsayed, BA, Link Indexer (link-indexer)
- Andrea Goethals, NLNZ, Project Coordination
- Amr Morad, BA, Link Service (link-serv)
- Ben O’Brien, NLNZ, Research Use Cases
- Amr Rizq, BA, Link Visualizer (link-viz)
Additional Thanks
- Tasneem Allam, BA, link-viz development
- Suzan Attia, BA, UI design
- Dalia Elbadry, BA, UI design
- Nada Eliba, BA, link-serv development
- Mirona Gamil, BA, link-serv development
- Olga Holownia, IIPC, project support
- Andy Jackson, British Library, technical advice
- Amged Magdey, BA, logo design
- Liquaa Mahmoud, BA, logo design
- Alex Osborne, National Library of Australia, technical advice
We would also like to thank the researchers who agreed to be interviewed for our Inventory of Use Cases.