Metadata Scraper

Description of Software	The Creative Commons Metadata Scraper is a simple crawler used by the license engine to detect metadata stored in pages.
Bug Tracker	Link to Bug Tracker
Code Repository	Link to Code repository
Mailing List	Link to Mailing list

Decommissioned

The Creative Commons Metadata Scraper was decommissioned on Monday 8th January 2018. Due to the changing nature of the web, in particular the widespread adoption of HTTPS, the Scraper had seen greatly reduced use. For potential future directions in metadata presentation, see Creative Commons' Open Ledger project.

What it Did

The metadata scraper was used by the license deeds to extract embedded metadata from pages. It scanned the page for RDFa using rdfadict and pyRdfa.

When it Happened

When a creator or copyright holder selects a license, they have the opportunity to provide additional metadata about their work. This includes information such as creator and medium, as well as how the creator would like to be attributed. If this information is provided, it is encoded in the HTML generated using RDFa.

In order to provide more context relevant information for visitors, the license deeds loaded a script which looks at the referring page for metadata. If found, the metadata was used to update the deed and display additional attribution or CC+ details.

Developer Information

Metadata Scraper was implemented in Python using the CherryPy web framework. The source code is available on GitHub at https://github.com/cc-archive/metadata_scraper .

Contact

Contact Rob Myers with questions about the scraper.