Archival

tags

Storage , Scraping , peer-to-peer

• 38% of webpages that existed in 2013 are no - longer accessible a decade later | Hacker News 
  • 27.6% of the Top 10 Million Sites are Dead | Lobsters 
• Ask HN: What would you preserve if the internet were to go down tomorrow? | Hacker News 
• DIY Web Archiving Zine | Hacker News 

Archiving formats

• See web_archiving_formats - Google Sheets 

Web Archiving Workflows and Best Practices

Institutional Archiving

Large institutions typically employ a workflow that involves:

1. Selection - Identifying content to preserve
2. Acquisition - Using crawlers like Heritrix to collect content
3. Storage - Preserving WARC files with redundancy
4. Access - Providing replay through Wayback Machine-like interfaces
5. Preservation - Ensuring long-term accessibility through format migration

Personal Archiving

Individual users have different needs:

1. On-the-fly capture - Browser extensions like ArchiveWeb.page or SingleFile
2. Local storage - Managing personal collections with tools like ReplayWeb.page
3. Format considerations - Balancing completeness vs. convenience
4. Sharing capabilities - Using portable formats like WACZ

Quality Assurance in Web Archiving

Critical considerations for effective archiving:

• Completeness - Capturing all required resources
• Fidelity - How closely the archive resembles the original
• Replayability - Whether interactive elements function
• Longevity - Format sustainability and migration paths

Usecases

Tools

Enterprise/Traditional Tools

Wayback Machine Tools

Miscellaneous Legacy Tools

Other Archiving Solutions

YouTube Archiving Tools

Digital Archiving Organizations and Tools

Major Digital Archives

Regional Archives

Smaller Archives & Tools

Wikipedia & Related Tools

Physical Archival

• How do archivists package things? The battle of the boxes | Hacker News 
• Archival Storage | Hacker News 

Other notes

• Use the Webrecorder tool suite https://webrecorder.net! It uses a new package file format for web archivss called WACZ (Web Archive Zipped) which produces a single file which you can store anywhere and playback offline. It automatically indexes different file formats such as PDFs or media files contained on the website and is versioned. You can record WACZ using the Chrome extension ArchiveWeb.page https://archiveweb.page/ or use the Internet Archive’s Save Page Now button to preserve a website and have the WACZ file sent to you via email: https://inkdroid.org/2023/04/03/spn-wacz/. There are also more sophisticated tools like the in-browser crawler ArchiveWeb.page Express https://express.archiveweb.page or the command-line crawler BrowserTrix https://webrecorder.net/tools#browsertrix-crawler. But manually recording using the Chrome extension is definitely the easiest and most reliable way. To play back the WACZ file just open it in the offline web-app ReplayWeb.page https://replayweb.page.
• Slightly biased (I work with Webrecorder haha) but yeah, our tools are really good at preserving complete webpages. u/CollapsedWave Give the ArchiveWebpage browser extension a shot! If you’re looking to save single pages as you come across them, it’s a good tool! Every page you capture gets its text extracted for text search. I’ll also add (because they mentioned file format standardization and longevity) that WACZ files are actually ZIP files which contain some indexing metadata that enables fast playback within a single portable file. The actual archived data is stored as a WARC wthin the WACZ and it doesn’t get much more standardized than that! Regardless of what you end up using, I’d really recommend capturing as WARCs or WACZ for cross-compatibility with other software.