Awesome Web Archiving

Easy-to-use Web archiver

Java library for reading and writing WARC files with a typed API

A dockerized, queued high fidelity web archiver based on Squidwarc

:gear: A Rust library for reading and writing WARC files

Converts WARC files to static HTML

Chrome extension that uses Memento to indicate that a page a user is viewing on the live web has an archived copy and to give the user access to the copy

Convert HTTP Archive (HAR) -> Web Archive (WARC) format

NPM package and CLI tool for saving web page as single HTML file

A Rails engine supporting the discovery of web archives.

A whirlwind tour of Common Crawl's data using Python

Tools for bulk indexing of WARC/ARC files on Hadoop, EMR or local file system.

Web archive index server based on RocksDB

Web archiving using Google Chrome

Prototype SOLR-powered web archive exploration UI.

Download and attach provenance to public datasets

Converts HTTrack crawls to WARC files

Command-line tool and Rust library for handling Web ARChive (WARC) files

No description.

Various examples of notebooks for working with web archives with the Archives Unleashed Toolkit, and derivatives generated by the Archives Unleashed Toolkit.

A tool for exploring, analyzing, transforming, recombining, and extracting data from WARC (Web ARChive) files.

An easy-to-use and highly customizable crawler that enables you to create your own little Web archives (WARC/CDX)

Wget with Lua extension

simple script to convert web resources to a single warc file

Web Archiving Course

DuckDB extension to fetch pages from Wayback Machine & Common Crawl

Create Robust Links from within Zotero

Web application for distributed compute analysis of Archive-It web archive collections.

golang readers for ARC and WARC webarchive formats

A tool for detecting viruses and NSFW material in WARC files

No description.

Internet Archive's Sparkling Data Processing Library

A data retrieval & exploration protocol designed to investigate science and policy processes

A client for the Archive-It And Webrecorder WASAPI Data Transfer API

Object Resource Stream and CDXJ Drafts

No description.

Playback webpages from Wayback Machine

A Tool to Summarize Web Archive Holdings

Tika based link (URL) extractor for httpreserve

Web archive deduplication tools

An open-source toolkit for analyzing line-oriented JSON Twitter archives with Apache Spark.

CLI implementation of httpreserve that can test links and retrieve internet archive replacements

The UKWA Heritrix3 custom modules and Docker builder.

No description.

A Splitable Hadoop InputFormat for Concatenated GZIP Files and *.(w)arc.gz

DuckDB extension for parsing WARC files

Java application to download WARCs from WASAPI

JWAT Tools

A jupyter notebook illistrating the basics of Common Crawl's datasets.

Java Web Archive Toolkit

A whirlwind tour of Common Crawl's data using Java