Creative Commons - User contributions [en]

Extend Metadata

2008-08-18T10:56:38Z

Hdworak: /* Conclusions */ Improved the accessibility of the code

[[Category:Developer]]
[[Category:Technology]]
[[Category:metadata]]
[[Category:guide]]

== Extending Creative Commons Metadata ==

Familiar with RDF and want to know how this fits in with the larger picture?

* View our [http://creativecommons/schema.rdf RDF schema] for information about [http://creativecommons.org/ns our model] in a machine-processable form.
* Read our report on how to use [[#Defining dc:rights|dc:rights]].
* Take a look at [[#Embedding RDF in HTML|our conclusions on using RDF in HTML]].

If you've got comments or suggestions on any of this, don't hesitate to [[Contact|send us a note]].

== Defining <code>dc:rights</code> ==

Use of the Dublin Core dc:rights element has long been fraught with confusion and misunderstanding. A study of its use in practice shows a variety of different purposes and formats with little commonality. For the Creative Commons metadata specification, we wanted to define a profile of the rights element that was more useful and structured.

Most copyright statements consist of a statement of the date and copyright holder (i.e. "(C) 2002 Gnomovision Records. All rights reserved.") and the actual text of the legal license. The date can be described using dc:date, the copyright holder using dc:rights and the license can be linked to with cc:license.

We've decided to use dc:rights to hold a more structured description of the copyright holder. Here's an example:

<source lang="XML">
<dc:rights>
<Agent rdf:about="http://me.yoyo.dyne.name/">
<dc:title>Yo-Yo Dyne</dc:title>
<dc:date>1001-10-01</dc:date>
</Agent>

</dc:rights>
</source>

As you can see the copyright holder is a person with a URL, a name and a date (of birth).

In this way we cover all the parts of the copyright notice in a way that RDF tools can deal with. As a full example, this plain text statement:

<pre>
(C) 2002 Gnomovision Records. All rights reserved.

This document has NO WARRANTY. You are permitted to
copy it, modify it, and place up to three (3) copies
of it on the White House lawn.
</pre>

becomes this chunk of RDF:

<source lang="XML">
<rdf:Description>
<dc:date>2002</dc:date>
<cc:license rdf:resource="http://flf.org/licenses/whiteHouseLawn" />
<dc:rights><Agent>
<dc:title>Gnomovision Records</dc:title>
</Agent></dc:rights>
</rdf:Description>
</source>

'''Note:''' The actual text and description of the license ("...place up to three (3) copies...") is not included in this RDF, but would presumably be available at the URL given for the license.

== Embedding RDF in HTML ==

There may be lots of RDF out there, but it's no good if no one can find it. Since almost everything on the Web is HTML, we need a good way to hook RDF into the HTML Web. Here are some ways to do so:

=== LINKing to an external file ===

RDF can be kept in a separate file (<code>filename.rdf</code>) and linked to using <code><link rel="meta" type="application/rdf+xml" ... /></code> or an equivalent a href tag.

* '''Pro:''' Doesn't break anything.
* '''Con:''' Requires users to make files. Not supported by all readers.

=== LINKing to a data: URL ===

A data: URL (see [http://www.ietf.org/rfc/rfc2397.txt RFC 2397]) encapsulates data into the URL itself. If used as the <code>href</code> attribute of a <code><link></code> element, it can include an RDF document verbatim.

<source lang="XML"><link rel="meta" type="application/rdf+xml" href="data:application/rdf+xml,%3Crdf%3ARDF%20xmlns%3D%22http%3A%2F%2Fweb.resource.org%2Fcc%2F%22%0D%0A%20%20%20%20xmlns%3Adc%3D%22http%3A%2F%2Fpurl.org%2Fdc%2Felements%2F1.1%2F%22%0D%0A%20%20%20%20xmlns%3Ardf%3D%22http%3A%2F%2Fwww.w3.org%2F1999%2F02%2F22-rdf-syntax-ns%23%22%3E%0D%0A%3CWork%20rdf%3Aabout%3D%22http%3A%2F%2Fexample.org%2Fgnomophone.mp3%22%3E%0D%0A%20%20%3Cdc%3Atitle%3ECompilers%20in%20the%20Key%20of%20C%3C%2Fdc%3Atitle%3E%0D%0A%20%20%3Cdc%3Adescription%3EA%20lovely%20classical%20work%20on%20compiling%20code.%3C%2Fdc%3Adescription%3E%0D%0A%20%20%3Cdc%3Acreator%3E%3CAgent%3E%0D%0A%20%20%20%20%3Cdc%3Atitle%3EYo-Yo%20Dyne%3C%2Fdc%3Atitle%3E%0D%0A%20%20%3C%2FAgent%3E%3C%2Fdc%3Acreator%3E%0D%0A%20%20%3Cdc%3Arights%3E%3CAgent%3E%0D%0A%20%20%20%20%3Cdc%3Atitle%3EGnomophone%3C%2Fdc%3Atitle%3E%0D%0A%20%20%3C%2FAgent%3E%3C%2Fdc%3Arights%3E%0D%0A%20%20%3Cdc%3Adate%3E1842%3C%2Fdc%3Adate%3E%0D%0A%20%20%3Cdc%3Aformat%3Eaudio%2Fmpeg%3C%2Fdc%3Aformat%3E%0D%0A%20%20%3Cdc%3Atype%20rdf%3Aresource%3D%22http%3A%2F%2Fpurl.org%2Fdc%2Fdcmitype%2FSound%22%20%2F%3E%0D%0A%20%20%3Cdc%3Asource%20rdf%3Aresource%3D%22http%3A%2F%2Fexample.net%2Fgnomovision.mov%22%20%2F%3E%0D%0A%20%20%3Clicense%20rdf%3Aresource%3D%22http%3A%2F%2Fcreativecommons.org%2Flicenses%2Fby-nc-nd%2F2.0%2F%22%20%2F%3E%0D%0A%20%20%3Clicense%20rdf%3Aresource%3D%22http%3A%2F%2Fwww.eff.org%2FIP%2FOpen_licenses%2Feff_oal.html%22%20%2F%3E%0D%0A%0D%0A%3C%2FWork%3E%0D%0A%0D%0A%3CLicense%20rdf%3Aabout%3D%22http%3A%2F%2Fcreativecommons.org%2Flicenses%2Fby-nc-nd%2F2.0%2F%22%3E%0D%0A%20%20%3Cpermits%20rdf%3Aresource%3D%22http%3A%2F%2Fweb.resource.org%2Fcc%2FReproduction%22%20%2F%3E%0D%0A%20%20%3Cpermits%20rdf%3Aresource%3D%22http%3A%2F%2Fweb.resource.org%2Fcc%2FDistribution%22%20%2F%3E%0D%0A%20%20%3Crequires%20rdf%3Aresource%3D%22http%3A%2F%2Fweb.resource.org%2Fcc%2FNotice%22%20%2F%3E%0D%0A%20%20%3Crequires%20rdf%3Aresource%3D%22http%3A%2F%2Fweb.resource.org%2Fcc%2FAttribution%22%20%2F%3E%0D%0A%20%20%3Cprohibits%20rdf%3Aresource%3D%22http%3A%2F%2Fweb.resource.org%2Fcc%2FCommercialUse%22%20%2F%3E%0D%0A%0D%0A%3C%2FLicense%3E%0D%0A%3C%2Frdf%3ARDF%3E" /></source>

* '''Pro:''' It includes the data in the HTML document; it doesn't break old browsers; it's standard-compliant; it uses the proper element (link) for associating metadata
* '''Con:''' It requires the RDF to be URL-encoded; it's hard to read; not all URL libraries can recognize or decode a data: URL; Creative Commons tools ([[MozCC]], [[CcLookup]], etc) don't support it, likely browsers don't/won't support it; not a preferred method.

See also: [http://software.hixie.ch/utilities/cgi/data/data data: URI kitchen] (tool for making <code>data:</code> URLs), [http://www.mozilla.org/quality/networking/docs/aboutdata.html about data: URLs] (from Mozilla)

=== In the HEAD ===

RDF can be included inline in the <code><head></code> tag of the document, along with the title and any <code><link></code>s.

* '''Pro:''' Should be ignored by browsers.
* '''Con:''' Not all users may be able to insert content into the HEAD of their documents. Doesn't validate. Not a preferred method.

=== In the BODY ===

RDF can be included inline in the <code><body></code> of the document, like any other HTML.

* '''Pro:''' Super simple.
* '''Con:''' RDF needs to be formatted as attributes. May cause problems with some browsers. Doesn't validate. Not a preferred method.

=== In META ===

A reference to the license a web page is offered under may be included in a <code><meta></code> tag in the document's <code><head></code>.

Example:

<source lang="XML">
<meta name="DC.rights"
scheme="DCTERMS.URI"
content="http://creativecommons.org/licenses/by-sa/2.0/" />
<meta name="DC.rights"
content="(c) 2002 Gnomovision Records" />
</source>

* '''Pro:''' Won't break anything.
* '''Con:''' Not all users may be able to. '''Not RDF''' (included in this list due to obviousness). Limited to statements about the current page. Not a preferred method.

Further reading: [http://dublincore.org/documents/dcq-html/ Expressing Dublin Core in HTML/XHTML meta and link elements]

=== In a comment ===

The RDF can be included in a <code></code> in the document. The TrackBack system embeds its RDF in this way.

Note that if a literal "--" is included in a comment, it must be encoded as "--".

* '''Pro:''' Won't break anything. Super simple.
* '''Con:''' Not supported by all readers. Makes people queasy.

=== Using element attributes ([[RDFa]]) ===

[[RDFa]] allows RDF to be expressed using a small set of attributes. The simple case of asserting a license only needs <code>rel="license"</code> added to the license link. Example:

<source lang="XML">
This work is licensed under the <a rel="license"
href="http://creativecommons.org/licenses/by/2.0/">Creative
Commons Attribution License</a>.
</source>

* '''Pro:''' Super simple, extensible, [http://rdfa.info/wiki/How-to-validate validates].
* '''Con:''' Validation requires modifying the DOCTYPE which some users may be unable to do.

See also: [[ccREL|Creative Commons Rights Expression Language]]

=== Conclusions ===

RDF readers should support all of these methods.

We use and recommend [[RDFa]] since it's easy, straight forward, powerful and doesn't break existing web clients (browsers). Our previous recommendation, including the RDF in an HTML comment, is no longer recommended. For more information on our use of RDFa and metadata strategy generally, see information on the [[ccREL|CC Rights Expression Language]].

For the RDFa equivalent of the RDF/XML embedded using the data: URI scheme above see below:
<source lang="XML"><div xmlns:cc="http://web.resource.org/cc/" xmlns:dc="http://purl.org/dc/elements/1.1/">
<div about="http://example.org/gnomophone.mp3" typeof="cc:Work">
<span property="dc:title" content="Compilers in the Key of C">foo</span>
<span property="dc:description" content="A lovely classical work on compiling code.">foo</span>
<span rel="dc:creator"><span typeof="cc:Agent" property="dc:title" content="Yo-Yo Dyne">foo</span></span>
<span rel="dc:rights"><span typeof="cc:Agent" property="dc:title" content="Gnomophone">foo</span></span>
<span property="dc:date" content="1842">foo</span>
<span property="dc:format" content="audio/mpeg">foo</span>
<a rel="dc:type" href="http://purl.org/dc/dcmitype/Sound">foo</a>,
<a rel="dc:source" href="http://example.net/gnomovision.mov">foo</a>,
<a rel="cc:license" href="http://creativecommons.org/licenses/by-nc-nd/2.0/">foo</a>,
<a rel="cc:license" href="http://artlibre.org/licence/lal/en/">foo</a>,
</div>
<div about="http://creativecommons.org/licenses/by-nc-nd/2.0/" typeof="cc:License">
<a rel="cc:permits" href="http://web.resource.org/cc/Reproduction">foo</a>,
<a rel="cc:permits" href="http://web.resource.org/cc/Distribution">foo</a>,
<a rel="cc:requires" href="http://web.resource.org/cc/Notice">foo</a>,
<a rel="cc:requires" href="http://web.resource.org/cc/Attribution">foo</a>,
<a rel="cc:prohibits" href="http://web.resource.org/cc/CommercialUse">foo</a>,
</div>
</div></source>

Further reading: [http://esw.w3.org/topic/EmbeddingRDFinHTML Embedding RDF in HTML]

Rewrite Metadata Validator/SoC 2008/PyDev integration with the Darklooks theme

2008-08-12T12:31:35Z

Hdworak:

If one uses Darklooks, a dark-coloured theme for [http://www.ubuntu.com/ Ubuntu], it might be desirable to make [http://pydev.sourceforge.net/ PyDev], a plug-in for [http://www.eclipse.org/ Eclipse SDK] to facilitate programming in [http://www.python.org/ Python], look similar. In order to do this, one has to save the public‑domain file listed below under the filename “dark.epf”, run Eclipse SDK with PyDev, and choose ''File'' » ''Import'' » ''General'' » ''Preferences'' (click “Next”), then select the file from the disk while preserving the option “Import all” enabled. After clicking “Finish”, Eclipse SDK should be restarted.
<pre>#Wed Aug 06 12:00:00 CEST 2008
/instance/org.python.pydev/STRING_COLOR=255,159,236
/instance/org.eclipse.pde.ui/editor.color.tag_bold=true
/instance/org.python.pydev/BACKQUOTES_COLOR=131,220,232
/instance/org.eclipse.pde.ui/editor.color.header_value=217,227,208
/instance/org.python.pydev/COMMENT_COLOR=186,233,147
/instance/org.python.pydev/CODE_COLOR=217,227,208
/instance/org.eclipse.core.resources/version=1
/instance/org.eclipse.pde.ui/editor.color.instr_bold=true
/instance/org.eclipse.ui/showIntro=false
/instance/org.python.pydev/NUMBER_COLOR=255,164,7
/instance/org.eclipse.ui.editors/AbstractTextEditor.Color.FindScope=122,73,95
/instance/org.eclipse.jdt.ui/org.eclipse.jface.textfont=1|DejaVu Sans Mono|10|0|GTK|1|;
/instance/org.eclipse.pde.ui/editor.color.tag=131,220,232
/instance/org.eclipse.ui/SHOW_TRADITIONAL_STYLE_TABS=true
/instance/org.eclipse.ui/ENABLE_ANIMATIONS=false
/instance/org.eclipse.pde.ui/editor.color.header_key=255,142,142
/instance/org.eclipse.ui.workbench/org.eclipse.ui.ide.systemDefault.org.eclipse.jface.textfont=1|DejaVu Sans Mono|10|0|GTK|1|;
/instance/org.python.pydev/CLASS_NAME_COLOR=246,152,42
/instance/org.eclipse.ui.editors/AbstractTextEditor.Color.SelectionBackground=58,73,76
/instance/org.eclipse.ui.editors/hyperlinkColor=173,173,255
/instance/org.eclipse.pde.ui/editor.color.xml_comment=186,233,147
/instance/org.eclipse.ui/CURRENT_THEME_ID=org.eclipse.ui.ide.systemDefault
/instance/org.python.pydev/KEYWORD_COLOR=131,220,232
/instance/org.python.pydev/SELF_COLOR=200,255,85
/instance/org.eclipse.ui.editors/lineNumberRuler=true
/instance/org.eclipse.ui.editors/lineNumberColor=255,255,255
/instance/org.python.pydev/FUNC_NAME_COLOR=246,236,42
/instance/org.python.pydev/EDITOR_MATCHING_BRACKETS_COLOR=207,0,0
/instance/org.eclipse.pde.ui/editor.color.header_assignment=217,227,208
/instance/org.eclipse.ui.editors/currentLineColor=70,70,70
/instance/org.eclipse.jdt.ui/fontPropagated=true
/instance/org.eclipse.pde.ui/editor.color.string=255,254,179
/instance/org.eclipse.pde.ui/editor.color.instr=131,220,232
/instance/org.eclipse.pde.ui/editor.color.default=217,227,208
/instance/org.eclipse.pde.ui/editor.color.header_attributes=131,220,232</pre>
See also:
* [http://www.koders.com/noncode/fid8221DE2EE9B51242CB891283859739473FA9CD23.aspx An alternative dark theme for PyDev]
* [http://drnicwilliams.com/2006/08/08/textmate-theme-for-radrails/ TextMate theme for RadRails]

Rewrite Metadata Validator/SoC 2008

2008-08-12T07:28:10Z

Hdworak: PyDev integration with the Darklooks theme

[[Category:Challenge]]
[[Category:Developer]]
[[Category:Metadata]]
[[Category:Opensource]]
[[Category:Project]]
== Introduction ==
Creative Commons [http://code.google.com/soc/2008/cc/about.html participates] in [http://code.google.com/soc/2008/ Google Summer of Code™] and has accepted a proposal (see the [http://code.google.com/soc/2008/cc/appinfo.html?csaid=3C284456287C63B9 abstract]) of [http://hugo.dworak.info Hugo Dworak] based on its description of a task to [[Rewrite Metadata Validator|rewrite]] its now-defunct [http://validator.creativecommons.org metadata validator]. [http://creativecommons.org/about/people/#83 Asheesh Laroia] has been assigned as a mentor of the project. The work began on May 26th, 2008 as per the [http://code.google.com/soc/2008/faqs.html#0.1_timeline project timeline].
== Synopsis ==
A licensor or a licensee wants to check whether a digitally embedded Creative Commons license associated with a file (a Web site in particular) is valid, does not use deprecated means to express it, and matches what the user has been expecting. Therefore, one opens the Web site and either pastes the source code as a direct input, uploads a file, or provides a link to retrieve it. The software displays the results of the analysis in a human-readable manner.
== Proposed timeline ==
*Week 1 — Preparing the SVN/Git or alike repository. Installing the required framework and libraries. Setting up temporary input (source code) and output (result) facilities. Testing the Python Web environment.
*Week 2 — Parsing cc-related RDFa information from well-formed XHTML files.
*Week 3 — Parsing cc-related RDF comments embedded in the XHTML code and those put directly in "head" and "body" elements.
*Week 4 — Parsing cc-related RDF files linked externally or embedded in the "link" element in the header section of the XHTML.
*Week 5 — Parsing cc-related dc-style XHTML-conforming information (embedded in "meta" elements or anchors).
*Week 6 — Ability to clean up the invalid XHTML code whenever possible.
*'''Midterm milestone''' — Developing a raw metadata validator capable of parsing Web sites and outputting valid cc-related information about them in a human-readable fashion.
*Week 7 — Parsing cc-related information contained within syndication feeds (RSS 1.0, RSS 2.0, Atom 1.0).
*Week 8 — Ability to submit an URI to the Web site to be parsed and upload a file using a form. Auto-detection of the MIME content type of such submissions based on HTTP headers and file extension. Ability for the user to override the automatic choice.
*Week 9 — Traversal of embedded objects and special links to obtain more information about licensing terms.
*Week 10 — Generating and storing statistics about the effects of validation such as: content type, errors, means of input, types of licences.
*Week 11 — Extensive testing and providing automatic test suites covering all of the aforementioned capabilities.
*Week 12 — Writing the documentation summarising the architecture. Making the application more user-friendly. Cloning the layout of the Creative Commons Web site.
*'''Final milestone''' — A full-fledged Web application capable of parsing licensing information from a variety of sources.
== Accessing the source code ==
The project has been split into two Git repositories called [http://code.creativecommons.org/viewgit?r=libvalidator.git libvalidator] and [http://code.creativecommons.org/viewgit?r=validator.git validator]. The first one is responsible for parsing the input with regard to licensing information. The latter is a Web application that is going to utilise the former to provide an interface for end users.
To browse the source code, one only needs to click on the two hyperlinks in the previous paragraph. To download the source code to a local machine, one needs to install Git first (which is available under the [http://packages.ubuntu.com/search?keywords=git-core git-core package] in [http://www.ubuntu.com/ Ubuntu]) and then issue the following commands in the console:
<source lang="bash">
git clone git://code.creativecommons.org/libvalidator.git
git clone git://code.creativecommons.org/validator.git
</source>
== Related Web applications ==
* [http://validator.w3.org/ The W3C Markup Validation Service]
* [http://jigsaw.w3.org/css-validator/ The W3C CSS Validation Service]
* [http://www.w3.org/RDF/Validator/ The W3C RDF Validation Service]
* [http://www.w3.org/2007/08/pyRdfa/ RDFa Distiller]
* [http://hcs.science.uva.nl/projects/SWI-Prolog/packages/sgml/online.html Online SWI-Prolog RDF parser demo]
== Web framework and libraries ==
Once [http://www.python.org/ Python] becomes the programming language of the application, one has to decide about the software to be used in building a [http://en.wikipedia.org/wiki/Web_application Web application], so that not everything is written from scratch. First of all, the [http://en.wikipedia.org/wiki/Web_application_framework framework] which will handle the tasks that are typically found in Web applications such as the implementation of the [http://en.wikipedia.org/wiki/Model-view-controller MVC pattern], [http://en.wikipedia.org/wiki/Web_application_framework#URL_mapping URL mapping], and so on. There are several [http://wiki.python.org/moin/WebFrameworks Web application frameworks in Python], for instance [http://www.djangoproject.com/ Django] and [http://www.turbogears.org/ TurboGears]. As Creative Commons uses the BSD-like licensed [http://www.pylonshq.com/ Pylons], this is the preferred choice for the project. Pylons supports many [http://en.wikipedia.org/wiki/Web_template_system template systems], like [http://www.makotemplates.org/ Mako] and [http://jinja.pocoo.org/ Jinja], and it is our design choice to use the BSD-like licensed [http://genshi.edgewall.org/ Genshi] for this purpose.

Other than the above, it comes to choosing Python packages to facilitate parsing and extracting information from the documents. To detect the character encoding of files provided by the users of the validator, the dual-licensed (cc-by and LGPL 3 or later) [http://cthedot.de/encutils/ encutils] might be used. It is powered by the [http://chardet.feedparser.org/ Universal Encoding Detector] (if present) and is designed to handle XML (including XHTML and RSS) and HTML. Next, one cannot expect that the users will provide only well-formed documents, therefore [http://utidylib.berlios.de/ µTidylib] (available under a MIT-style license) and the BSD-like licensed [http://www.crummy.com/software/BeautifulSoup/ Beautiful Soup] may be used to clean up the ill-formed mark-up, so that it can still be parsed. Since Python packages based on tidy have poor Unicode support (as of July 2008), only the Beautiful Soup will be used to provide the fallback.

Once the document is well-formed and decoded properly, one can proceed to extracting its embedded information relevant to the license terms. It can be represented in [[Extend Metadata|numerous different ways]]. For instance, RDF data can be provided inside comments or as elements inside head and body elements. Such data can also be encoded using a [http://en.wikipedia.org/wiki/Data:_URI_scheme data: URI scheme] or linked externally using the link element. To parse RDF data one can use the BSD-like licensed [http://rdflib.net/ RDFLib], the dual-licensed (W3C Software License and GNU GPL 2 or newer) [http://infomesh.net/2003/rdfparser/ rdfxml.py], and many others. To do the same with RDFa one can employ [http://rdfa.digitalbazaar.com/librdfa/ librdfa] (which has Python bindings and is licensed under GNU LGPL 2.1 or newer) or the MIT-style licensed [http://pypi.python.org/pypi/rdfadict rdfadict]. However, we have decided to take advantage of [http://www.w3.org/2007/08/pyRdfa/ pyRdfa] (available under the W3C Software License) as it supports not only RDFa, but also [http://dublincore.org/documents/dcq-html/ expressing Dublin Core in HTML/XHTML meta and link elements].

We shall use [http://somethingaboutorange.com/mrl/projects/nose/ nose] for [http://en.wikipedia.org/wiki/Unit_testing unit testing] and both [http://peak.telecommunity.com/DevCenter/setuptools setuptools] and [http://pypi.python.org/pypi/zc.buildout zc.buildout] to handle dependencies and build Python packages. [http://git.or.cz/ Git] is the version control system of choice.
== Helper scripts ==
The following tools have been developed in order to facilitate the accomplishment of the goal, though they are not directly related to the project:
* [[Rewrite_Metadata_Validator/SoC_2008/Wine_Browsers_Setup|Wine browsers setup]] — a Bash script to install [http://www.apple.com/safari/ Safari] and [http://www.microsoft.com/windows/products/winfamily/ie/default.mspx Microsoft Internet Explorer] with the [http://www.microsoft.com/downloads/details.aspx?familyid=E59C3964-672D-4511-BB3E-2D5E1DB91038&displaylang=en Internet Explorer Developer Toolbar] under [http://www.winehq.org/ Wine],
* [[Rewrite_Metadata_Validator/SoC_2008/IRC_Scanner|IRC Scanner]] — a PHP 5 script that searches for the occurrences of a particular nickname in the logs of [[IRC|the Creative Commons' IRC channel]],
* [[Rewrite_Metadata_Validator/SoC_2008/PyDev_integration_with_the_Darklooks_theme|PyDev integration with the Darklooks theme]] — syntax highlighting on a dark background for [http://pydev.sourceforge.net/ PyDev].

Rewrite Metadata Validator/SoC 2008/PyDev integration with the Darklooks theme

2008-08-12T07:24:03Z

Hdworak: Initial import

If one uses Darklooks, a dark-coloured theme for [http://www.ubuntu.com/ Ubuntu], it might be desirable to make [http://pydev.sourceforge.net/ PyDev], a plug-in for [http://www.eclipse.org/ Eclipse SDK] to facilate programming in [http://www.python.org/ Python], look similar. In order to do this, one has to save the public‑domain file listed below under the filename “dark.epf”, run Eclipse SDK with PyDev, and choose ''File'' » ''Import'' » ''General'' » ''Preferences'' (click “Next”), then select the file from the disk while preserving the option “Import all” enabled. After clicking “Finish”, Eclipse SDK should be restarted.
<pre>#Wed Aug 06 12:00:00 CEST 2008
/instance/org.python.pydev/STRING_COLOR=255,159,236
/instance/org.eclipse.pde.ui/editor.color.tag_bold=true
/instance/org.python.pydev/BACKQUOTES_COLOR=131,220,232
/instance/org.eclipse.pde.ui/editor.color.header_value=217,227,208
/instance/org.python.pydev/COMMENT_COLOR=186,233,147
/instance/org.python.pydev/CODE_COLOR=217,227,208
/instance/org.eclipse.core.resources/version=1
/instance/org.eclipse.pde.ui/editor.color.instr_bold=true
/instance/org.eclipse.ui/showIntro=false
/instance/org.python.pydev/NUMBER_COLOR=255,164,7
/instance/org.eclipse.ui.editors/AbstractTextEditor.Color.FindScope=122,73,95
/instance/org.eclipse.jdt.ui/org.eclipse.jface.textfont=1|DejaVu Sans Mono|10|0|GTK|1|;
/instance/org.eclipse.pde.ui/editor.color.tag=131,220,232
/instance/org.eclipse.ui/SHOW_TRADITIONAL_STYLE_TABS=true
/instance/org.eclipse.ui/ENABLE_ANIMATIONS=false
/instance/org.eclipse.pde.ui/editor.color.header_key=255,142,142
/instance/org.eclipse.ui.workbench/org.eclipse.ui.ide.systemDefault.org.eclipse.jface.textfont=1|DejaVu Sans Mono|10|0|GTK|1|;
/instance/org.python.pydev/CLASS_NAME_COLOR=246,152,42
/instance/org.eclipse.ui.editors/AbstractTextEditor.Color.SelectionBackground=58,73,76
/instance/org.eclipse.ui.editors/hyperlinkColor=173,173,255
/instance/org.eclipse.pde.ui/editor.color.xml_comment=186,233,147
/instance/org.eclipse.ui/CURRENT_THEME_ID=org.eclipse.ui.ide.systemDefault
/instance/org.python.pydev/KEYWORD_COLOR=131,220,232
/instance/org.python.pydev/SELF_COLOR=200,255,85
/instance/org.eclipse.ui.editors/lineNumberRuler=true
/instance/org.eclipse.ui.editors/lineNumberColor=255,255,255
/instance/org.python.pydev/FUNC_NAME_COLOR=246,236,42
/instance/org.python.pydev/EDITOR_MATCHING_BRACKETS_COLOR=207,0,0
/instance/org.eclipse.pde.ui/editor.color.header_assignment=217,227,208
/instance/org.eclipse.ui.editors/currentLineColor=70,70,70
/instance/org.eclipse.jdt.ui/fontPropagated=true
/instance/org.eclipse.pde.ui/editor.color.string=255,254,179
/instance/org.eclipse.pde.ui/editor.color.instr=131,220,232
/instance/org.eclipse.pde.ui/editor.color.default=217,227,208
/instance/org.eclipse.pde.ui/editor.color.header_attributes=131,220,232</pre>
See also:
* [http://www.koders.com/noncode/fid8221DE2EE9B51242CB891283859739473FA9CD23.aspx An alternative dark theme for PyDev]
* [http://drnicwilliams.com/2006/08/08/textmate-theme-for-radrails/ TextMate theme for RadRails]

Rewrite Metadata Validator/SoC 2008

2008-07-13T08:07:48Z

Hdworak: "Accessing the source code"

Rewrite Metadata Validator/SoC 2008

2008-07-13T07:50:57Z

Hdworak: The information on what was actually chosen

[[Category:Challenge]]
[[Category:Developer]]
[[Category:Metadata]]
[[Category:Opensource]]
[[Category:Project]]
== Introduction ==
Creative Commons [http://code.google.com/soc/2008/cc/about.html participates] in [http://code.google.com/soc/2008/ Google Summer of Code™] and has accepted a proposal (see the [http://code.google.com/soc/2008/cc/appinfo.html?csaid=3C284456287C63B9 abstract]) of [http://hugo.dworak.info Hugo Dworak] based on its description of a task to [[Rewrite Metadata Validator|rewrite]] its now-defunct [http://validator.creativecommons.org metadata validator]. [http://creativecommons.org/about/people/#83 Asheesh Laroia] has been assigned as a mentor of the project. The work began on May 26th, 2008 as per the [http://code.google.com/soc/2008/faqs.html#0.1_timeline project timeline].
== Synopsis ==
A licensor or a licensee wants to check whether a digitally embedded Creative Commons license associated with a file (a Web site in particular) is valid, does not use deprecated means to express it, and matches what the user has been expecting. Therefore, one opens the Web site and either pastes the source code as a direct input, uploads a file, or provides a link to retrieve it. The software displays the results of the analysis in a human-readable manner.
== Proposed timeline ==
*Week 1 — Preparing the SVN/Git or alike repository. Installing the required framework and libraries. Setting up temporary input (source code) and output (result) facilities. Testing the Python Web environment.
*Week 2 — Parsing cc-related RDFa information from well-formed XHTML files.
*Week 3 — Parsing cc-related RDF comments embedded in the XHTML code and those put directly in "head" and "body" elements.
*Week 4 — Parsing cc-related RDF files linked externally or embedded in the "link" element in the header section of the XHTML.
*Week 5 — Parsing cc-related dc-style XHTML-conforming information (embedded in "meta" elements or anchors).
*Week 6 — Ability to clean up the invalid XHTML code whenever possible.
*'''Midterm milestone''' — Developing a raw metadata validator capable of parsing Web sites and outputting valid cc-related information about them in a human-readable fashion.
*Week 7 — Parsing cc-related information contained within syndication feeds (RSS 1.0, RSS 2.0, Atom 1.0).
*Week 8 — Ability to submit an URI to the Web site to be parsed and upload a file using a form. Auto-detection of the MIME content type of such submissions based on HTTP headers and file extension. Ability for the user to override the automatic choice.
*Week 9 — Traversal of embedded objects and special links to obtain more information about licensing terms.
*Week 10 — Generating and storing statistics about the effects of validation such as: content type, errors, means of input, types of licences.
*Week 11 — Extensive testing and providing automatic test suites covering all of the aforementioned capabilities.
*Week 12 — Writing the documentation summarising the architecture. Making the application more user-friendly. Cloning the layout of the Creative Commons Web site.
*'''Final milestone''' — A full-fledged Web application capable of parsing licensing information from a variety of sources.
== Related Web applications ==
* [http://validator.w3.org/ The W3C Markup Validation Service]
* [http://jigsaw.w3.org/css-validator/ The W3C CSS Validation Service]
* [http://www.w3.org/RDF/Validator/ The W3C RDF Validation Service]
* [http://www.w3.org/2007/08/pyRdfa/ RDFa Distiller]
* [http://hcs.science.uva.nl/projects/SWI-Prolog/packages/sgml/online.html Online SWI-Prolog RDF parser demo]
== Web framework and libraries ==
Once [http://www.python.org/ Python] becomes the programming language of the application, one has to decide about the software to be used in building a [http://en.wikipedia.org/wiki/Web_application Web application], so that not everything is written from scratch. First of all, the [http://en.wikipedia.org/wiki/Web_application_framework framework] which will handle the tasks that are typically found in Web applications such as the implementation of the [http://en.wikipedia.org/wiki/Model-view-controller MVC pattern], [http://en.wikipedia.org/wiki/Web_application_framework#URL_mapping URL mapping], and so on. There are several [http://wiki.python.org/moin/WebFrameworks Web application frameworks in Python], for instance [http://www.djangoproject.com/ Django] and [http://www.turbogears.org/ TurboGears]. As Creative Commons uses the BSD-like licensed [http://www.pylonshq.com/ Pylons], this is the preferred choice for the project. Pylons supports many [http://en.wikipedia.org/wiki/Web_template_system template systems], like [http://www.makotemplates.org/ Mako] and [http://jinja.pocoo.org/ Jinja], and it is our design choice to use the BSD-like licensed [http://genshi.edgewall.org/ Genshi] for this purpose.

Other than the above, it comes to choosing Python packages to facilitate parsing and extracting information from the documents. To detect the character encoding of files provided by the users of the validator, the dual-licensed (cc-by and LGPL 3 or later) [http://cthedot.de/encutils/ encutils] shall be used. It is powered by the [http://chardet.feedparser.org/ Universal Encoding Detector] (if present) and is designed to handle XML (including XHTML and RSS) and HTML. Next, one cannot expect that the users will provide only well-formed documents, therefore [http://utidylib.berlios.de/ µTidylib] (available under a MIT-style license) and the BSD-like licensed [http://www.crummy.com/software/BeautifulSoup/ Beautiful Soup] may be used to clean up the ill-formed mark-up, so that it can still be parsed. Since Python packages based on tidy have poor Unicode support (as of July 2008), only the Beautiful Soup will be used to provide the fallback.

Once the document is well-formed and decoded properly, one can proceed to extracting its embedded information relevant to the license terms. It can be represented in [[Extend Metadata|numerous different ways]]. For instance, RDF data can be provided inside comments or as elements inside head and body elements. Such data can also be encoded using a [http://en.wikipedia.org/wiki/Data:_URI_scheme data: URI scheme] or linked externally using the link element. To parse RDF data one can use the BSD-like licensed [http://rdflib.net/ RDFLib], the dual-licensed (W3C Software License and GNU GPL 2 or newer) [http://infomesh.net/2003/rdfparser/ rdfxml.py], and many others. To do the same with RDFa one can employ [http://rdfa.digitalbazaar.com/librdfa/ librdfa] (which has Python bindings and is licensed under GNU LGPL 2.1 or newer) or the MIT-style licensed [http://pypi.python.org/pypi/rdfadict rdfadict]. However, we have decided to take advantage of [http://www.w3.org/2007/08/pyRdfa/ pyRdfa] (available under the W3C Software License) as it supports not only RDFa, but also [http://dublincore.org/documents/dcq-html/ expressing Dublin Core in HTML/XHTML meta and link elements].

We shall use [http://somethingaboutorange.com/mrl/projects/nose/ nose] for [http://en.wikipedia.org/wiki/Unit_testing unit testing] and both [http://peak.telecommunity.com/DevCenter/setuptools setuptools] and [http://pypi.python.org/pypi/zc.buildout zc.buildout] to handle dependencies and build Python packages.

== Helper scripts ==
The following tools have been developed in order to facilitate the accomplishment of the goal, though they are not directly related to the project:
* [[Rewrite_Metadata_Validator/SoC_2008/Wine_Browsers_Setup|Wine browsers setup]] — a Bash script to install [http://www.apple.com/safari/ Safari] and [http://www.microsoft.com/windows/products/winfamily/ie/default.mspx Microsoft Internet Explorer] with the [http://www.microsoft.com/downloads/details.aspx?familyid=E59C3964-672D-4511-BB3E-2D5E1DB91038&displaylang=en Internet Explorer Developer Toolbar] under [http://www.winehq.org/ Wine],
* [[Rewrite_Metadata_Validator/SoC_2008/IRC_Scanner|IRC Scanner]] — a PHP 5 script that searches for the occurrences of a particular nickname in the logs of [[IRC|the Creative Commons' IRC channel]].

Extend Metadata

2008-06-28T19:41:19Z

Hdworak: An RDFa example (available under public domain)

[[Category:Developer]]
[[Category:Technology]]
[[Category:metadata]]
[[Category:guide]]

== Extending Creative Commons Metadata ==

Familiar with RDF and want to know how this fits in with the larger picture?

* View our [http://creativecommons/schema.rdf RDF schema] for information about [http://creativecommons.org/ns our model] in a machine-processable form.
* Read our report on how to use [[#Defining dc:rights|dc:rights]].
* Take a look at [[#Embedding RDF in HTML|our conclusions on using RDF in HTML]].

If you've got comments or suggestions on any of this, don't hesitate to [[Contact|send us a note]].

== Defining <code>dc:rights</code> ==

Use of the Dublin Core dc:rights element has long been fraught with confusion and misunderstanding. A study of its use in practice shows a variety of different purposes and formats with little commonality. For the Creative Commons metadata specification, we wanted to define a profile of the rights element that was more useful and structured.

Most copyright statements consist of a statement of the date and copyright holder (i.e. "(C) 2002 Gnomovision Records. All rights reserved.") and the actual text of the legal license. The date can be described using dc:date, the copyright holder using dc:rights and the license can be linked to with cc:license.

We've decided to use dc:rights to hold a more structured description of the copyright holder. Here's an example:

<source lang="XML">
<dc:rights>
<Agent rdf:about="http://me.yoyo.dyne.name/">
<dc:title>Yo-Yo Dyne</dc:title>
<dc:date>1001-10-01</dc:date>
</Agent>

</dc:rights>
</source>

As you can see the copyright holder is a person with a URL, a name and a date (of birth).

In this way we cover all the parts of the copyright notice in a way that RDF tools can deal with. As a full example, this plain text statement:

<pre>
(C) 2002 Gnomovision Records. All rights reserved.

This document has NO WARRANTY. You are permitted to
copy it, modify it, and place up to three (3) copies
of it on the White House lawn.
</pre>

becomes this chunk of RDF:

<source lang="XML">
<rdf:Description>
<dc:date>2002</dc:date>
<cc:license rdf:resource="http://flf.org/licenses/whiteHouseLawn" />
<dc:rights><Agent>
<dc:title>Gnomovision Records</dc:title>
</Agent></dc:rights>
</rdf:Description>
</source>

'''Note:''' The actual text and description of the license ("...place up to three (3) copies...") is not included in this RDF, but would presumably be available at the URL given for the license.

== Embedding RDF in HTML ==

There may be lots of RDF out there, but it's no good if no one can find it. Since almost everything on the Web is HTML, we need a good way to hook RDF into the HTML Web. Here are some ways to do so:

=== LINKing to an external file ===

RDF can be kept in a separate file (<code>filename.rdf</code>) and linked to using <code><link rel="meta" type="application/rdf+xml" ... /></code> or an equivalent a href tag.

* '''Pro:''' Doesn't break anything.
* '''Con:''' Requires users to make files. Not supported by all readers.

=== LINKing to a data: URL ===

A data: URL (see [http://www.ietf.org/rfc/rfc2397.txt RFC 2397]) encapsulates data into the URL itself. If used as the <code>href</code> attribute of a <code><link></code> element, it can include an RDF document verbatim.

<source lang="XML"><link rel="meta" type="application/rdf+xml" href="data:application/rdf+xml,%3Crdf%3ARDF%20xmlns%3D%22http%3A%2F%2Fweb.resource.org%2Fcc%2F%22%0D%0A%20%20%20%20xmlns%3Adc%3D%22http%3A%2F%2Fpurl.org%2Fdc%2Felements%2F1.1%2F%22%0D%0A%20%20%20%20xmlns%3Ardf%3D%22http%3A%2F%2Fwww.w3.org%2F1999%2F02%2F22-rdf-syntax-ns%23%22%3E%0D%0A%3CWork%20rdf%3Aabout%3D%22http%3A%2F%2Fexample.org%2Fgnomophone.mp3%22%3E%0D%0A%20%20%3Cdc%3Atitle%3ECompilers%20in%20the%20Key%20of%20C%3C%2Fdc%3Atitle%3E%0D%0A%20%20%3Cdc%3Adescription%3EA%20lovely%20classical%20work%20on%20compiling%20code.%3C%2Fdc%3Adescription%3E%0D%0A%20%20%3Cdc%3Acreator%3E%3CAgent%3E%0D%0A%20%20%20%20%3Cdc%3Atitle%3EYo-Yo%20Dyne%3C%2Fdc%3Atitle%3E%0D%0A%20%20%3C%2FAgent%3E%3C%2Fdc%3Acreator%3E%0D%0A%20%20%3Cdc%3Arights%3E%3CAgent%3E%0D%0A%20%20%20%20%3Cdc%3Atitle%3EGnomophone%3C%2Fdc%3Atitle%3E%0D%0A%20%20%3C%2FAgent%3E%3C%2Fdc%3Arights%3E%0D%0A%20%20%3Cdc%3Adate%3E1842%3C%2Fdc%3Adate%3E%0D%0A%20%20%3Cdc%3Aformat%3Eaudio%2Fmpeg%3C%2Fdc%3Aformat%3E%0D%0A%20%20%3Cdc%3Atype%20rdf%3Aresource%3D%22http%3A%2F%2Fpurl.org%2Fdc%2Fdcmitype%2FSound%22%20%2F%3E%0D%0A%20%20%3Cdc%3Asource%20rdf%3Aresource%3D%22http%3A%2F%2Fexample.net%2Fgnomovision.mov%22%20%2F%3E%0D%0A%20%20%3Clicense%20rdf%3Aresource%3D%22http%3A%2F%2Fcreativecommons.org%2Flicenses%2Fby-nc-nd%2F2.0%2F%22%20%2F%3E%0D%0A%20%20%3Clicense%20rdf%3Aresource%3D%22http%3A%2F%2Fwww.eff.org%2FIP%2FOpen_licenses%2Feff_oal.html%22%20%2F%3E%0D%0A%0D%0A%3C%2FWork%3E%0D%0A%0D%0A%3CLicense%20rdf%3Aabout%3D%22http%3A%2F%2Fcreativecommons.org%2Flicenses%2Fby-nc-nd%2F2.0%2F%22%3E%0D%0A%20%20%3Cpermits%20rdf%3Aresource%3D%22http%3A%2F%2Fweb.resource.org%2Fcc%2FReproduction%22%20%2F%3E%0D%0A%20%20%3Cpermits%20rdf%3Aresource%3D%22http%3A%2F%2Fweb.resource.org%2Fcc%2FDistribution%22%20%2F%3E%0D%0A%20%20%3Crequires%20rdf%3Aresource%3D%22http%3A%2F%2Fweb.resource.org%2Fcc%2FNotice%22%20%2F%3E%0D%0A%20%20%3Crequires%20rdf%3Aresource%3D%22http%3A%2F%2Fweb.resource.org%2Fcc%2FAttribution%22%20%2F%3E%0D%0A%20%20%3Cprohibits%20rdf%3Aresource%3D%22http%3A%2F%2Fweb.resource.org%2Fcc%2FCommercialUse%22%20%2F%3E%0D%0A%0D%0A%3C%2FLicense%3E%0D%0A%3C%2Frdf%3ARDF%3E" /></source>

* '''Pro:''' It includes the data in the HTML document; it doesn't break old browsers; it's standard-compliant; it uses the proper element (link) for associating metadata
* '''Con:''' It requires the RDF to be URL-encoded; it's hard to read; not all URL libraries can recognize or decode a data: URL; Creative Commons tools ([[MozCC]], [[CcLookup]], etc) don't support it, likely browsers don't/won't support it; not a preferred method.

See also: [http://software.hixie.ch/utilities/cgi/data/data data: URI kitchen] (tool for making <code>data:</code> URLs), [http://www.mozilla.org/quality/networking/docs/aboutdata.html about data: URLs] (from Mozilla)

=== In the HEAD ===

RDF can be included inline in the <code><head></code> tag of the document, along with the title and any <code><link></code>s.

* '''Pro:''' Should be ignored by browsers.
* '''Con:''' Not all users may be able to insert content into the HEAD of their documents. Doesn't validate. Not a preferred method.

=== In the BODY ===

RDF can be included inline in the <code><body></code> of the document, like any other HTML.

* '''Pro:''' Super simple.
* '''Con:''' RDF needs to be formatted as attributes. May cause problems with some browsers. Doesn't validate. Not a preferred method.

=== In META ===

A reference to the license a web page is offered under may be included in a <code><meta></code> tag in the document's <code><head></code>.

Example:

<source lang="XML">
<meta name="DC.rights"
scheme="DCTERMS.URI"
content="http://creativecommons.org/licenses/by-sa/2.0/" />
<meta name="DC.rights"
content="(c) 2002 Gnomovision Records" />
</source>

* '''Pro:''' Won't break anything.
* '''Con:''' Not all users may be able to. '''Not RDF''' (included in this list due to obviousness). Limited to statements about the current page. Not a preferred method.

Further reading: [http://dublincore.org/documents/dcq-html/ Expressing Dublin Core in HTML/XHTML meta and link elements]

=== In a comment ===

The RDF can be included in a <code></code> in the document. The TrackBack system embeds its RDF in this way.

Note that if a literal "--" is included in a comment, it must be encoded as "--".

* '''Pro:''' Won't break anything. Super simple.
* '''Con:''' Not supported by all readers. Makes people queasy.

=== Using element attributes ([[RDFa]]) ===

[[RDFa]] allows RDF to be expressed using a small set of attributes. The simple case of asserting a license only needs <code>rel="license"</code> added to the license link. Example:

<source lang="XML">
This work is licensed under the <a rel="license"
href="http://creativecommons.org/licenses/by/2.0/">Creative
Commons Attribution License</a>.
</source>

* '''Pro:''' Super simple, extensible, [http://rdfa.info/wiki/How-to-validate validates].
* '''Con:''' Validation requires modifying the DOCTYPE which some users may be unable to do.

See also: [[ccREL|Creative Commons Rights Expression Language]]

=== Conclusions ===

RDF readers should support all of these methods.

We use and recommend [[RDFa]] since it's easy, straight forward, powerful and doesn't break existing web clients (browsers). Our previous recommendation, including the RDF in an HTML comment, is no longer recommended. For more information on our use of RDFa and metadata strategy generally, see information on the [[ccREL|CC Rights Expression Language]].

For the RDFa equivalent of the RDF/XML embedded using the data: URI scheme above see below:
<source lang="XML"><div xmlns:cc="http://web.resource.org/cc/" xmlns:dc="http://purl.org/dc/elements/1.1/">
<div about="http://example.org/gnomophone.mp3" typeof="cc:Work">
<span property="dc:title" content="Compilers in the Key of C">foo</span>
<span property="dc:description" content="A lovely classical work on compiling code.">foo</span>
<span rel="dc:creator"><span typeof="cc:Agent" property="dc:title" content="Yo-Yo Dyne">foo</span></span>
<span rel="dc:rights"><span typeof="cc:Agent" property="dc:title" content="Gnomophone">foo</span></span>
<span property="dc:date" content="1842">foo</span>
<span property="dc:format" content="audio/mpeg">foo</span>
<a rel="dc:type" href="http://purl.org/dc/dcmitype/Sound">foo</a>
<a rel="dc:source" href="http://example.net/gnomovision.mov">foo</a>
<a rel="cc:license" href="http://creativecommons.org/licenses/by-nc-nd/2.0/">foo</a>
<a rel="cc:license" href="http://artlibre.org/licence/lal/en/">foo</a>
</div>
<div about="http://creativecommons.org/licenses/by-nc-nd/2.0/" typeof="cc:License">
<a rel="cc:permits" href="http://web.resource.org/cc/Reproduction">foo</a>
<a rel="cc:permits" href="http://web.resource.org/cc/Distribution">foo</a>
<a rel="cc:requires" href="http://web.resource.org/cc/Notice">foo</a>
<a rel="cc:requires" href="http://web.resource.org/cc/Attribution">foo</a>
<a rel="cc:prohibits" href="http://web.resource.org/cc/CommercialUse">foo</a>
</div>
</div></source>

Further reading: [http://esw.w3.org/topic/EmbeddingRDFinHTML Embedding RDF in HTML]

Rewrite Metadata Validator/SoC 2008/Wine Browsers Setup

2008-06-16T08:04:05Z

Hdworak: Syntax highlighting

If one wants to use two popular browsers (namely [http://www.apple.com/safari/ Safari] and [http://www.microsoft.com/windows/products/winfamily/ie/default.mspx Microsoft Internet Explorer] with the [http://www.microsoft.com/downloads/details.aspx?familyid=E59C3964-672D-4511-BB3E-2D5E1DB91038&displaylang=en Internet Explorer Developer Toolbar]) under [http://www.ubuntu.com/ Ubuntu] or [http://www.ubuntu.com/products/whatisubuntu/derivatives one of its flavours] to test the compatibility of his or her Web application, he or she must use [http://www.winehq.org/ Wine] for this purpose. It is important to note that in order to run Microsoft Internet Explorer under Ubuntu, one has to obtain a legal copy of [http://www.microsoft.com/windows/ Microsoft Windows]. One can also experience problems when installing Internet Explorer Developer Toolbar, because [http://www.tatanka.com.br/ies4linux/page/Main_Page IEs 4 Linux] installs Microsoft Internet Explorer 6 in such a fashion, that it runs under a simulated Microsoft Windows 98 environment, whereas the aforementioned toolbar requires Microsoft Windows 2000 or newer. If [http://www.wine-doors.org/ Wine-Doors] is used to accomplish this goal, it may suffer from [http://www.wine-doors.org/trac/ticket/723 a bug that has been closed just recently] (as of June 2008). The following public domain Bash script can be helpful to install both browsers:
<source lang="bash">#!/bin/bash
sudo apt-get install sed wget gedit wine binfmt-support msttcorefonts
cd ~
if [ ! -f ies4linux-latest.tar.gz ]
then
wget http://www.tatanka.com.br/ies4linux/downloads/ies4linux-latest.tar.gz
fi
if [ ! -f IEDevToolBarSetup.msi ]
then
wget http://download.microsoft.com/download/f/3/c/f3c93e70-ccdc-46c9-bbd4-70d94bdd0cc9/IEDevToolBarSetup.msi
fi
if [ ! -f SafariSetup.exe ]
then
wget http://appldnld.apple.com.edgesuite.net/content.info.apple.com/Safari3/061-4602.20080416.t5rGb/SafariSetup.exe
fi
if [ ! -f winetricks ]
then
wget http://www.kegel.com/wine/winetricks
fi
chmod u+x winetricks
cat > theme.reg <<'EOF'
Windows Registry Editor Version 5.00

[HKEY_CURRENT_USER\Control Panel\Colors]
"ActiveBorder"="212 208 200"
"ActiveTitle"="10 36 106"
"AppWorkSpace"="128 128 128"
"Background"="58 110 165"
"ButtonAlternateFace"="181 181 181"
"ButtonDkShadow"="64 64 64"
"ButtonFace"="212 208 200"
"ButtonHilight"="255 255 255"
"ButtonLight"="212 208 200"
"ButtonShadow"="128 128 128"
"ButtonText"="0 0 0"
"GradientActiveTitle"="166 202 240"
"GradientInactiveTitle"="192 192 192"
"GrayText"="128 128 128"
"Hilight"="10 36 106"
"HilightText"="255 255 255"
"HotTrackingColor"="0 0 128"
"InactiveBorder"="212 208 200"
"InactiveTitle"="128 128 128"
"InactiveTitleText"="212 208 200"
"InfoText"="0 0 0"
"InfoWindow"="255 255 225"
"Menu"="212 208 200"
"MenuBar"="212 208 200"
"MenuHilight"="0 0 0"
"MenuText"="0 0 0"
"Scrollbar"="212 208 200"
"TitleText"="255 255 255"
"Window"="255 255 255"
"WindowFrame"="0 0 0"
"WindowText"="0 0 0"
EOF
mkdir ~/.browsers
WINEPREFIX=~/.browsers/safari wineprefixcreate
WINEPREFIX=~/.browsers/safari ./winetricks corefonts vcrun2005
WINEPREFIX=~/.browsers/safari wine SafariSetup.exe # uncheck additional commands
WINEPREFIX=~/.browsers/safari wineserver -k
WINEPREFIX=~/.browsers/safari regedit theme.reg
tar zxvf ies4linux-latest.tar.gz
cd ies4linux-*
./ies4linux --no-gui --locale PL --basedir ~/.browsers
cd ~
rm -f ~/.browsers/ie6/.firstrun
sed -i 's/DllOverrides/DllOverridez/g;s/win98/win2k/g' ~/.browsers/ie6/user.reg
WINEPREFIX=~/.browsers/ie6 msiexec /i IEDevToolBarSetup.msi</source>
The following commands have to be executed afterwards:
<source lang="bash">sed -i 's/DllOverridez/DllOverrides/g;s/win2k/win98/g' ~/.browsers/ie6/user.reg
WINEPREFIX=~/.browsers/ie6 regedit theme.reg
rm -rf ies4linux-* theme.reg IEDevToolBarSetup.msi ~/.browsers/ie6/.firstrun SafariSetup.exe</source>

Rewrite Metadata Validator/SoC 2008/IRC Scanner

2008-06-16T08:02:29Z

Hdworak: Syntax highlighting

If one wants to search for his or her activity at [[IRC|the IRC channel of Creative Commons]], he or she can use [http://www.google.com/search?&q=john+site%3Ahttp%3A%2F%2Fmirrors.creativecommons.org%2Firc%2Fcc%2F Google] or the following public domain script written in PHP 5:
<source lang="php"><?php

$nick = 'john';
$path = 'http://mirrors.creativecommons.org/irc/cc/';
preg_match_all('/%23cc\.'.date('Y').'\-\d\d\-\d\d\.log\.html/', file_get_contents($path), $matches);
$irrelevant = array();
if (file_exists('irrelevant.txt')) {
$irrelevant = unserialize(file_get_contents('irrelevant.txt'));
}
foreach ($matches[0] as $url) {
if (file_exists('relevant/'.($filename = str_replace('%23', '', $url)))
|| in_array($filename, $irrelevant)) {
echo 'Skipped ', $filename, PHP_EOL;
continue;
}
$contents = file_get_contents($path.$url);
if (!strstr($contents, $nick.'</th><td class="text"')) {
echo 'Irrelevant ', $filename, PHP_EOL;
$irrelevant[] = $filename;
continue;
}
file_put_contents('relevant/'.$filename, $contents);
echo 'Downloaded ', $filename, PHP_EOL;
}
file_put_contents('irrelevant.txt', serialize($irrelevant));
echo 'Saved irrelevant.txt', PHP_EOL;</source>
Please note that the herewith enclosed script searches the logs saved the current year. All relevant logs are downloaded to the local machine.

Rewrite Metadata Validator/SoC 2008/Wine Browsers Setup

2008-06-15T16:14:51Z

Hdworak:

If one wants to use two popular browsers (namely [http://www.apple.com/safari/ Safari] and [http://www.microsoft.com/windows/products/winfamily/ie/default.mspx Microsoft Internet Explorer] with the [http://www.microsoft.com/downloads/details.aspx?familyid=E59C3964-672D-4511-BB3E-2D5E1DB91038&displaylang=en Internet Explorer Developer Toolbar]) under [http://www.ubuntu.com/ Ubuntu] or [http://www.ubuntu.com/products/whatisubuntu/derivatives one of its flavours] to test the compatibility of his or her Web application, he or she must use [http://www.winehq.org/ Wine] for this purpose. It is important to note that in order to run Microsoft Internet Explorer under Ubuntu, one has to obtain a legal copy of [http://www.microsoft.com/windows/ Microsoft Windows]. One can also experience problems when installing Internet Explorer Developer Toolbar, because [http://www.tatanka.com.br/ies4linux/page/Main_Page IEs 4 Linux] installs Microsoft Internet Explorer 6 in such a fashion, that it runs under a simulated Microsoft Windows 98 environment, whereas the aforementioned toolbar requires Microsoft Windows 2000 or newer. If [http://www.wine-doors.org/ Wine-Doors] is used to accomplish this goal, it may suffer from [http://www.wine-doors.org/trac/ticket/723 a bug that has been closed just recently] (as of June 2008). The following public domain Bash script can be helpful to install both browsers:
<pre>#!/bin/bash
sudo apt-get install sed wget gedit wine binfmt-support msttcorefonts
cd ~
if [ ! -f ies4linux-latest.tar.gz ]
then
wget http://www.tatanka.com.br/ies4linux/downloads/ies4linux-latest.tar.gz
fi
if [ ! -f IEDevToolBarSetup.msi ]
then
wget http://download.microsoft.com/download/f/3/c/f3c93e70-ccdc-46c9-bbd4-70d94bdd0cc9/IEDevToolBarSetup.msi
fi
if [ ! -f SafariSetup.exe ]
then
wget http://appldnld.apple.com.edgesuite.net/content.info.apple.com/Safari3/061-4602.20080416.t5rGb/SafariSetup.exe
fi
if [ ! -f winetricks ]
then
wget http://www.kegel.com/wine/winetricks
fi
chmod u+x winetricks
cat > theme.reg <<'EOF'
Windows Registry Editor Version 5.00

[HKEY_CURRENT_USER\Control Panel\Colors]
"ActiveBorder"="212 208 200"
"ActiveTitle"="10 36 106"
"AppWorkSpace"="128 128 128"
"Background"="58 110 165"
"ButtonAlternateFace"="181 181 181"
"ButtonDkShadow"="64 64 64"
"ButtonFace"="212 208 200"
"ButtonHilight"="255 255 255"
"ButtonLight"="212 208 200"
"ButtonShadow"="128 128 128"
"ButtonText"="0 0 0"
"GradientActiveTitle"="166 202 240"
"GradientInactiveTitle"="192 192 192"
"GrayText"="128 128 128"
"Hilight"="10 36 106"
"HilightText"="255 255 255"
"HotTrackingColor"="0 0 128"
"InactiveBorder"="212 208 200"
"InactiveTitle"="128 128 128"
"InactiveTitleText"="212 208 200"
"InfoText"="0 0 0"
"InfoWindow"="255 255 225"
"Menu"="212 208 200"
"MenuBar"="212 208 200"
"MenuHilight"="0 0 0"
"MenuText"="0 0 0"
"Scrollbar"="212 208 200"
"TitleText"="255 255 255"
"Window"="255 255 255"
"WindowFrame"="0 0 0"
"WindowText"="0 0 0"
EOF
mkdir ~/.browsers
WINEPREFIX=~/.browsers/safari wineprefixcreate
WINEPREFIX=~/.browsers/safari ./winetricks corefonts vcrun2005
WINEPREFIX=~/.browsers/safari wine SafariSetup.exe # uncheck additional commands
WINEPREFIX=~/.browsers/safari wineserver -k
WINEPREFIX=~/.browsers/safari regedit theme.reg
tar zxvf ies4linux-latest.tar.gz
cd ies4linux-*
./ies4linux --no-gui --locale PL --basedir ~/.browsers
cd ~
rm -f ~/.browsers/ie6/.firstrun
sed -i 's/DllOverrides/DllOverridez/g;s/win98/win2k/g' ~/.browsers/ie6/user.reg
WINEPREFIX=~/.browsers/ie6 msiexec /i IEDevToolBarSetup.msi</pre>
The following commands have to be executed afterwards:
<pre>sed -i 's/DllOverridez/DllOverrides/g;s/win2k/win98/g' ~/.browsers/ie6/user.reg
WINEPREFIX=~/.browsers/ie6 regedit theme.reg
rm -rf ies4linux-* theme.reg IEDevToolBarSetup.msi ~/.browsers/ie6/.firstrun SafariSetup.exe</pre>

Rewrite Metadata Validator/SoC 2008

2008-06-15T16:13:45Z

Hdworak: /* Helper scripts */

[[Category:Challenge]]
[[Category:Developer]]
[[Category:Metadata]]
[[Category:Opensource]]
[[Category:Project]]
== Introduction ==
Creative Commons [http://code.google.com/soc/2008/cc/about.html participates] in [http://code.google.com/soc/2008/ Google Summer of Code™] and has accepted a proposal (see the [http://code.google.com/soc/2008/cc/appinfo.html?csaid=3C284456287C63B9 abstract]) of [http://hugo.dworak.info Hugo Dworak] based on its description of a task to [[Rewrite Metadata Validator|rewrite]] its now-defunct [http://validator.creativecommons.org metadata validator]. [http://creativecommons.org/about/people/#83 Asheesh Laroia] has been assigned as a mentor of the project. The work began on May 26th, 2008 as per the [http://code.google.com/soc/2008/faqs.html#0.1_timeline project timeline].
== Synopsis ==
A licensor or a licensee wants to check whether a digitally embedded Creative Commons license associated with a file (a Web site in particular) is valid, does not use deprecated means to express it, and matches what the user has been expecting. Therefore, one opens the Web site and either pastes the source code as a direct input, uploads a file, or provides a link to retrieve it. The software displays the results of the analysis in a human-readable manner.
== Proposed timeline ==
*Week 1 — Preparing the SVN/Git or alike repository. Installing the required framework and libraries. Setting up temporary input (source code) and output (result) facilities. Testing the Python Web environment.
*Week 2 — Parsing cc-related RDFa information from well-formed XHTML files.
*Week 3 — Parsing cc-related RDF comments embedded in the XHTML code and those put directly in "head" and "body" elements.
*Week 4 — Parsing cc-related RDF files linked externally or embedded in the "link" element in the header section of the XHTML.
*Week 5 — Parsing cc-related dc-style XHTML-conforming information (embedded in "meta" elements or anchors).
*Week 6 — Ability to clean up the invalid XHTML code whenever possible.
*'''Midterm milestone''' — Developing a raw metadata validator capable of parsing Web sites and outputting valid cc-related information about them in a human-readable fashion.
*Week 7 — Parsing cc-related information contained within syndication feeds (RSS 1.0, RSS 2.0, Atom 1.0).
*Week 8 — Ability to submit an URI to the Web site to be parsed and upload a file using a form. Auto-detection of the MIME content type of such submissions based on HTTP headers and file extension. Ability for the user to override the automatic choice.
*Week 9 — Traversal of embedded objects and special links to obtain more information about licensing terms.
*Week 10 — Generating and storing statistics about the effects of validation such as: content type, errors, means of input, types of licences.
*Week 11 — Extensive testing and providing automatic test suites covering all of the aforementioned capabilities.
*Week 12 — Writing the documentation summarising the architecture. Making the application more user-friendly. Cloning the layout of the Creative Commons Web site.
*'''Final milestone''' — A full-fledged Web application capable of parsing licensing information from a variety of sources.
== Related Web applications ==
* [http://validator.w3.org/ The W3C Markup Validation Service]
* [http://jigsaw.w3.org/css-validator/ The W3C CSS Validation Service]
* [http://www.w3.org/RDF/Validator/ The W3C RDF Validation Service]
* [http://www.w3.org/2007/08/pyRdfa/ RDFa Distiller]
* [http://hcs.science.uva.nl/projects/SWI-Prolog/packages/sgml/online.html Online SWI-Prolog RDF parser demo]
== Web framework and libraries ==
Once [http://www.python.org/ Python] becomes the programming language of the application, one has to decide about the software to be used in building a [http://en.wikipedia.org/wiki/Web_application Web application], so that not everything is written from scratch. First of all, the [http://en.wikipedia.org/wiki/Web_application_framework framework] which will handle the tasks that are typically found in Web applications such as the implementation of the [http://en.wikipedia.org/wiki/Model-view-controller MVC pattern], [http://en.wikipedia.org/wiki/Web_application_framework#URL_mapping URL mapping], and so on. There are several [http://wiki.python.org/moin/WebFrameworks Web application frameworks in Python], for instance [http://www.djangoproject.com/ Django] and [http://www.turbogears.org/ TurboGears]. As Creative Commons uses the BSD-like licensed [http://www.pylonshq.com/ Pylons], this is the preferred choice for the project. Pylons supports many [http://en.wikipedia.org/wiki/Web_template_system template systems], like [http://www.makotemplates.org/ Mako] and [http://jinja.pocoo.org/ Jinja], and it is our design choice to use the BSD-like licensed [http://genshi.edgewall.org/ Genshi] for this purpose.

Other than the above, it comes to choosing Python packages to facilitate parsing and extracting information from the documents. To detect the character encoding of files provided by the users of the validator, the dual-licensed (cc-by and LGPL 3 or later) [http://cthedot.de/encutils/ encutils] shall be used. It is powered by the [http://chardet.feedparser.org/ Universal Encoding Detector] (if present) and is designed to handle XML (including XHTML and RSS) and HTML. Next, one cannot expect that the users will provide only well-formed documents, therefore [http://utidylib.berlios.de/ µTidylib] (available under a MIT-style license) and the BSD-like licensed [http://www.crummy.com/software/BeautifulSoup/ Beautiful Soup] may be used to clean up the ill-formed mark-up, so that it can still be parsed.

Once the document is well-formed and decoded properly, one can proceed to extracting its embedded information relevant to the license terms. It can be represented in [[Extend Metadata|numerous different ways]]. For instance, RDF data can be provided inside comments or as elements inside head and body elements. Such data can also be encoded using a [http://en.wikipedia.org/wiki/Data:_URI_scheme data: URI scheme] or linked externally using the link element. To parse RDF data one can use the BSD-like licensed [http://rdflib.net/ RDFLib], the dual-licensed (W3C Software License and GNU GPL 2 or newer) [http://infomesh.net/2003/rdfparser/ rdfxml.py], and many others. To do the same with RDFa one can employ [http://rdfa.digitalbazaar.com/librdfa/ librdfa] (which has Python bindings and is licensed under GNU LGPL 2.1 or newer) or the MIT-style licensed [http://pypi.python.org/pypi/rdfadict rdfadict].
== Helper scripts ==
The following tools have been developed in order to facilitate the accomplishment of the goal, though they are not directly related to the project:
* [[Rewrite_Metadata_Validator/SoC_2008/Wine_Browsers_Setup|Wine browsers setup]] — a Bash script to install [http://www.apple.com/safari/ Safari] and [http://www.microsoft.com/windows/products/winfamily/ie/default.mspx Microsoft Internet Explorer] with the [http://www.microsoft.com/downloads/details.aspx?familyid=E59C3964-672D-4511-BB3E-2D5E1DB91038&displaylang=en Internet Explorer Developer Toolbar] under [http://www.winehq.org/ Wine],
* [[Rewrite_Metadata_Validator/SoC_2008/IRC_Scanner|IRC Scanner]] — a PHP 5 script that searches for the occurrences of a particular nickname in the logs of [[IRC|the Creative Commons' IRC channel]].

Rewrite Metadata Validator/SoC 2008

2008-06-15T16:10:36Z

Hdworak: Helper scripts

[[Category:Challenge]]
[[Category:Developer]]
[[Category:Metadata]]
[[Category:Opensource]]
[[Category:Project]]
== Introduction ==
Creative Commons [http://code.google.com/soc/2008/cc/about.html participates] in [http://code.google.com/soc/2008/ Google Summer of Code™] and has accepted a proposal (see the [http://code.google.com/soc/2008/cc/appinfo.html?csaid=3C284456287C63B9 abstract]) of [http://hugo.dworak.info Hugo Dworak] based on its description of a task to [[Rewrite Metadata Validator|rewrite]] its now-defunct [http://validator.creativecommons.org metadata validator]. [http://creativecommons.org/about/people/#83 Asheesh Laroia] has been assigned as a mentor of the project. The work began on May 26th, 2008 as per the [http://code.google.com/soc/2008/faqs.html#0.1_timeline project timeline].
== Synopsis ==
A licensor or a licensee wants to check whether a digitally embedded Creative Commons license associated with a file (a Web site in particular) is valid, does not use deprecated means to express it, and matches what the user has been expecting. Therefore, one opens the Web site and either pastes the source code as a direct input, uploads a file, or provides a link to retrieve it. The software displays the results of the analysis in a human-readable manner.
== Proposed timeline ==
*Week 1 — Preparing the SVN/Git or alike repository. Installing the required framework and libraries. Setting up temporary input (source code) and output (result) facilities. Testing the Python Web environment.
*Week 2 — Parsing cc-related RDFa information from well-formed XHTML files.
*Week 3 — Parsing cc-related RDF comments embedded in the XHTML code and those put directly in "head" and "body" elements.
*Week 4 — Parsing cc-related RDF files linked externally or embedded in the "link" element in the header section of the XHTML.
*Week 5 — Parsing cc-related dc-style XHTML-conforming information (embedded in "meta" elements or anchors).
*Week 6 — Ability to clean up the invalid XHTML code whenever possible.
*'''Midterm milestone''' — Developing a raw metadata validator capable of parsing Web sites and outputting valid cc-related information about them in a human-readable fashion.
*Week 7 — Parsing cc-related information contained within syndication feeds (RSS 1.0, RSS 2.0, Atom 1.0).
*Week 8 — Ability to submit an URI to the Web site to be parsed and upload a file using a form. Auto-detection of the MIME content type of such submissions based on HTTP headers and file extension. Ability for the user to override the automatic choice.
*Week 9 — Traversal of embedded objects and special links to obtain more information about licensing terms.
*Week 10 — Generating and storing statistics about the effects of validation such as: content type, errors, means of input, types of licences.
*Week 11 — Extensive testing and providing automatic test suites covering all of the aforementioned capabilities.
*Week 12 — Writing the documentation summarising the architecture. Making the application more user-friendly. Cloning the layout of the Creative Commons Web site.
*'''Final milestone''' — A full-fledged Web application capable of parsing licensing information from a variety of sources.
== Related Web applications ==
* [http://validator.w3.org/ The W3C Markup Validation Service]
* [http://jigsaw.w3.org/css-validator/ The W3C CSS Validation Service]
* [http://www.w3.org/RDF/Validator/ The W3C RDF Validation Service]
* [http://www.w3.org/2007/08/pyRdfa/ RDFa Distiller]
* [http://hcs.science.uva.nl/projects/SWI-Prolog/packages/sgml/online.html Online SWI-Prolog RDF parser demo]
== Web framework and libraries ==
Once [http://www.python.org/ Python] becomes the programming language of the application, one has to decide about the software to be used in building a [http://en.wikipedia.org/wiki/Web_application Web application], so that not everything is written from scratch. First of all, the [http://en.wikipedia.org/wiki/Web_application_framework framework] which will handle the tasks that are typically found in Web applications such as the implementation of the [http://en.wikipedia.org/wiki/Model-view-controller MVC pattern], [http://en.wikipedia.org/wiki/Web_application_framework#URL_mapping URL mapping], and so on. There are several [http://wiki.python.org/moin/WebFrameworks Web application frameworks in Python], for instance [http://www.djangoproject.com/ Django] and [http://www.turbogears.org/ TurboGears]. As Creative Commons uses the BSD-like licensed [http://www.pylonshq.com/ Pylons], this is the preferred choice for the project. Pylons supports many [http://en.wikipedia.org/wiki/Web_template_system template systems], like [http://www.makotemplates.org/ Mako] and [http://jinja.pocoo.org/ Jinja], and it is our design choice to use the BSD-like licensed [http://genshi.edgewall.org/ Genshi] for this purpose.

Other than the above, it comes to choosing Python packages to facilitate parsing and extracting information from the documents. To detect the character encoding of files provided by the users of the validator, the dual-licensed (cc-by and LGPL 3 or later) [http://cthedot.de/encutils/ encutils] shall be used. It is powered by the [http://chardet.feedparser.org/ Universal Encoding Detector] (if present) and is designed to handle XML (including XHTML and RSS) and HTML. Next, one cannot expect that the users will provide only well-formed documents, therefore [http://utidylib.berlios.de/ µTidylib] (available under a MIT-style license) and the BSD-like licensed [http://www.crummy.com/software/BeautifulSoup/ Beautiful Soup] may be used to clean up the ill-formed mark-up, so that it can still be parsed.

Once the document is well-formed and decoded properly, one can proceed to extracting its embedded information relevant to the license terms. It can be represented in [[Extend Metadata|numerous different ways]]. For instance, RDF data can be provided inside comments or as elements inside head and body elements. Such data can also be encoded using a [http://en.wikipedia.org/wiki/Data:_URI_scheme data: URI scheme] or linked externally using the link element. To parse RDF data one can use the BSD-like licensed [http://rdflib.net/ RDFLib], the dual-licensed (W3C Software License and GNU GPL 2 or newer) [http://infomesh.net/2003/rdfparser/ rdfxml.py], and many others. To do the same with RDFa one can employ [http://rdfa.digitalbazaar.com/librdfa/ librdfa] (which has Python bindings and is licensed under GNU LGPL 2.1 or newer) or the MIT-style licensed [http://pypi.python.org/pypi/rdfadict rdfadict].
== Helper scripts ==
The following tools have been developed in order to facilitate the accomplishment of the goal, though they are not directly related to the project:
* [[Rewrite_Metadata_Validator/SoC_2008/Wine_Browsers_Setup|Wine browsers setup]] — a Bash script to install [http://www.apple.com/safari/ Safari] and [http://www.microsoft.com/windows/products/winfamily/ie/default.mspx Microsoft Internet Explorer] with the [http://www.microsoft.com/downloads/details.aspx?familyid=E59C3964-672D-4511-BB3E-2D5E1DB91038&displaylang=en Internet Explorer Developer Toolbar] under [http://www.winehq.org/ Wine],
* [[Rewrite_Metadata_Validator/SoC_2008/IRC_Scanner|IRC Scanner]] — a PHP 5 script that searches for the occurrences of a particular nickname in the logs of [[IRC|the creative commons' IRC channel]].

Rewrite Metadata Validator/SoC 2008/Wine Browsers Setup

2008-06-15T15:57:14Z

Hdworak: Initial writing

If one wants to use two popular browsers (namely [http://www.apple.com/safari/ Safari] and [http://www.microsoft.com/windows/products/winfamily/ie/default.mspx Microsoft Internet Explorer] with [http://www.microsoft.com/downloads/details.aspx?familyid=E59C3964-672D-4511-BB3E-2D5E1DB91038&displaylang=en Internet Explorer Developer Toolbar]) under [http://www.ubuntu.com/ Ubuntu] or [http://www.ubuntu.com/products/whatisubuntu/derivatives one of its flavours] to test the compatibility of his or her Web application, he or she must use [http://www.winehq.org/ Wine] for this purpose. It is important to note that in order to run Microsoft Internet Explorer under Ubuntu, one has to obtain a legal copy of [http://www.microsoft.com/windows/ Microsoft Windows]. One can also experience problems when installing Internet Explorer Developer Toolbar, because [http://www.tatanka.com.br/ies4linux/page/Main_Page IEs 4 Linux] installs Microsoft Internet Explorer 6 in such a fashion, that it runs under a simulated Microsoft Windows 98 environment, whereas the aforementioned toolbar requires Microsoft Windows 2000 or newer. If [http://www.wine-doors.org/ Wine-Doors] is used to accomplish this goal, it may suffer from [http://www.wine-doors.org/trac/ticket/723 a bug that has been closed just recently] (as of June 2008). The following public domain Bash script can be helpful to install both browsers:
<pre>#!/bin/bash
sudo apt-get install sed wget gedit wine binfmt-support msttcorefonts
cd ~
if [ ! -f ies4linux-latest.tar.gz ]
then
wget http://www.tatanka.com.br/ies4linux/downloads/ies4linux-latest.tar.gz
fi
if [ ! -f IEDevToolBarSetup.msi ]
then
wget http://download.microsoft.com/download/f/3/c/f3c93e70-ccdc-46c9-bbd4-70d94bdd0cc9/IEDevToolBarSetup.msi
fi
if [ ! -f SafariSetup.exe ]
then
wget http://appldnld.apple.com.edgesuite.net/content.info.apple.com/Safari3/061-4602.20080416.t5rGb/SafariSetup.exe
fi
if [ ! -f winetricks ]
then
wget http://www.kegel.com/wine/winetricks
fi
chmod u+x winetricks
cat > theme.reg <<'EOF'
Windows Registry Editor Version 5.00

[HKEY_CURRENT_USER\Control Panel\Colors]
"ActiveBorder"="212 208 200"
"ActiveTitle"="10 36 106"
"AppWorkSpace"="128 128 128"
"Background"="58 110 165"
"ButtonAlternateFace"="181 181 181"
"ButtonDkShadow"="64 64 64"
"ButtonFace"="212 208 200"
"ButtonHilight"="255 255 255"
"ButtonLight"="212 208 200"
"ButtonShadow"="128 128 128"
"ButtonText"="0 0 0"
"GradientActiveTitle"="166 202 240"
"GradientInactiveTitle"="192 192 192"
"GrayText"="128 128 128"
"Hilight"="10 36 106"
"HilightText"="255 255 255"
"HotTrackingColor"="0 0 128"
"InactiveBorder"="212 208 200"
"InactiveTitle"="128 128 128"
"InactiveTitleText"="212 208 200"
"InfoText"="0 0 0"
"InfoWindow"="255 255 225"
"Menu"="212 208 200"
"MenuBar"="212 208 200"
"MenuHilight"="0 0 0"
"MenuText"="0 0 0"
"Scrollbar"="212 208 200"
"TitleText"="255 255 255"
"Window"="255 255 255"
"WindowFrame"="0 0 0"
"WindowText"="0 0 0"
EOF
mkdir ~/.browsers
WINEPREFIX=~/.browsers/safari wineprefixcreate
WINEPREFIX=~/.browsers/safari ./winetricks corefonts vcrun2005
WINEPREFIX=~/.browsers/safari wine SafariSetup.exe # uncheck additional commands
WINEPREFIX=~/.browsers/safari wineserver -k
WINEPREFIX=~/.browsers/safari regedit theme.reg
tar zxvf ies4linux-latest.tar.gz
cd ies4linux-*
./ies4linux --no-gui --locale PL --basedir ~/.browsers
cd ~
rm -f ~/.browsers/ie6/.firstrun
sed -i 's/DllOverrides/DllOverridez/g;s/win98/win2k/g' ~/.browsers/ie6/user.reg
WINEPREFIX=~/.browsers/ie6 msiexec /i IEDevToolBarSetup.msi</pre>
The following commands have to be executed afterwards:
<pre>sed -i 's/DllOverridez/DllOverrides/g;s/win2k/win98/g' ~/.browsers/ie6/user.reg
WINEPREFIX=~/.browsers/ie6 regedit theme.reg
rm -rf ies4linux-* theme.reg IEDevToolBarSetup.msi ~/.browsers/ie6/.firstrun SafariSetup.exe</pre>

Rewrite Metadata Validator/SoC 2008/IRC Scanner

2008-06-15T15:30:13Z

Hdworak: Initial writing

If one wants to search for his or her activity at [[IRC|the IRC channel of Creative Commons]], he or she can use [http://www.google.com/search?&q=john+site%3Ahttp%3A%2F%2Fmirrors.creativecommons.org%2Firc%2Fcc%2F Google] or the following public domain script written in PHP 5:
<pre><?php

$nick = 'john';
$path = 'http://mirrors.creativecommons.org/irc/cc/';
preg_match_all('/%23cc\.'.date('Y').'\-\d\d\-\d\d\.log\.html/', file_get_contents($path), $matches);
$irrelevant = array();
if (file_exists('irrelevant.txt')) {
$irrelevant = unserialize(file_get_contents('irrelevant.txt'));
}
foreach ($matches[0] as $url) {
if (file_exists('relevant/'.($filename = str_replace('%23', '', $url)))
|| in_array($filename, $irrelevant)) {
echo 'Skipped ', $filename, PHP_EOL;
continue;
}
$contents = file_get_contents($path.$url);
if (!strstr($contents, $nick.'</th><td class="text"')) {
echo 'Irrelevant ', $filename, PHP_EOL;
$irrelevant[] = $filename;
continue;
}
file_put_contents('relevant/'.$filename, $contents);
echo 'Downloaded ', $filename, PHP_EOL;
}
file_put_contents('irrelevant.txt', serialize($irrelevant));
echo 'Saved irrelevant.txt', PHP_EOL;</pre>
Please note that the herewith enclosed script searches the logs saved the current year. All relevant logs are downloaded to the local machine.

Extend Metadata

2008-06-03T19:46:18Z

Hdworak: /* As an element attribute */

[[Category:Developer]]
[[Category:Technology]]
[[Category:metadata]]
[[Category:guide]]

== Extending Creative Commons Metadata ==

Familiar with RDF and want to know how this fits in with the larger picture?

* View our [http://creativecommons/schema.rdf RDF schema] for information about [http://creativecommons.org/ns our model] in a machine-processable form.
* Read our report on how to use [[#Defining dc:rights|dc:rights]].
* Take a look at [[#Embedding RDF in HTML|our conclusions on using RDF in HTML]].

If you've got comments or suggestions on any of this, don't hesitate to [[Contact|send us a note]].

== Defining <code>dc:rights</code> ==

Use of the Dublin Core dc:rights element has long been fraught with confusion and misunderstanding. A study of its use in practice shows a variety of different purposes and formats with little commonality. For the Creative Commons metadata specification, we wanted to define a profile of the rights element that was more useful and structured.

Most copyright statements consist of a statement of the date and copyright holder (i.e. "(C) 2002 Gnomovision Records. All rights reserved.") and the actual text of the legal license. The date can be described using dc:date, the copyright holder using dc:rights and the license can be linked to with cc:license.

We've decided to use dc:rights to hold a more structured description of the copyright holder. Here's an example:

<pre>
<dc:rights>
<Agent rdf:about="http://me.yoyo.dyne.name/">
<dc:title>Yo-Yo Dyne</dc:title>
<dc:date>1001-10-01</dc:date>
</Agent>

</dc:rights>
</pre>

As you can see the copyright holder is a person with a URL, a name and a date (of birth).

In this way we cover all the parts of the copyright notice in a way that RDF tools can deal with. As a full example, this plain text statement:

<pre>
(C) 2002 Gnomovision Records. All rights reserved.

This document has NO WARRANTY. You are permitted to
copy it, modify it, and place up to three (3) copies
of it on the White House lawn.
</pre>

becomes this chunk of RDF:

<pre>
<rdf:Description>
<dc:date>2002</dc:date>
<cc:license rdf:resource="http://flf.org/licenses/whiteHouseLawn" />
<dc:rights><Agent>
<dc:title>Gnomovision Records</dc:title>
</Agent></dc:rights>
</rdf:Description>
</pre>

'''Note:''' The actual text and description of the license ("...place up to three (3) copies...") is not included in this RDF, but would presumably be available at the URL given for the license.

== Embedding RDF in HTML ==

There may be lots of RDF out there, but it's no good if no one can find it. Since almost everything on the Web is HTML, we need a good way to hook RDF into the HTML Web. Here are the best ways to do so:

=== LINKing to an external file ===

RDF can be kept in a separate file (<code>filename.rdf</code>) and linked to using <code><link rel="meta" type="application/rdf+xml" ... /></code> or an equivalent a href tag.

* '''Pro:''' Doesn't break anything.
* '''Con:''' Requires users to make files. Not supported by all readers.

=== LINKing to a data: URL ===

A data: URL (see [http://www.ietf.org/rfc/rfc2397.txt RFC 2397]) encapsulates data into the URL itself. If used as the <code>href</code> attribute of a <code><link></code> element, it can include an RDF document verbatim.

<link rel="meta" type="application/rdf+xml" href="data:application/rdf+xml,%3Crdf%3ARDF%20xmlns%3D%22http%3A%2F%2Fweb.resource.org%2Fcc%2F%22%0D%0A%20%20%20%20xmlns%3Adc%3D%22http%3A%2F%2Fpurl.org%2Fdc%2Felements%2F1.1%2F%22%0D%0A%20%20%20%20xmlns%3Ardf%3D%22http%3A%2F%2Fwww.w3.org%2F1999%2F02%2F22-rdf-syntax-ns%23%22%3E%0D%0A%3CWork%20rdf%3Aabout%3D%22http%3A%2F%2Fexample.org%2Fgnomophone.mp3%22%3E%0D%0A%20%20%3Cdc%3Atitle%3ECompilers%20in%20the%20Key%20of%20C%3C%2Fdc%3Atitle%3E%0D%0A%20%20%3Cdc%3Adescription%3EA%20lovely%20classical%20work%20on%20compiling%20code.%3C%2Fdc%3Adescription%3E%0D%0A%20%20%3Cdc%3Acreator%3E%3CAgent%3E%0D%0A%20%20%20%20%3Cdc%3Atitle%3EYo-Yo%20Dyne%3C%2Fdc%3Atitle%3E%0D%0A%20%20%3C%2FAgent%3E%3C%2Fdc%3Acreator%3E%0D%0A%20%20%3Cdc%3Arights%3E%3CAgent%3E%0D%0A%20%20%20%20%3Cdc%3Atitle%3EGnomophone%3C%2Fdc%3Atitle%3E%0D%0A%20%20%3C%2FAgent%3E%3C%2Fdc%3Arights%3E%0D%0A%20%20%3Cdc%3Adate%3E1842%3C%2Fdc%3Adate%3E%0D%0A%20%20%3Cdc%3Aformat%3Eaudio%2Fmpeg%3C%2Fdc%3Aformat%3E%0D%0A%20%20%3Cdc%3Atype%20rdf%3Aresource%3D%22http%3A%2F%2Fpurl.org%2Fdc%2Fdcmitype%2FSound%22%20%2F%3E%0D%0A%20%20%3Cdc%3Asource%20rdf%3Aresource%3D%22http%3A%2F%2Fexample.net%2Fgnomovision.mov%22%20%2F%3E%0D%0A%20%20%3Clicense%20rdf%3Aresource%3D%22http%3A%2F%2Fcreativecommons.org%2Flicenses%2Fby-nc-nd%2F2.0%2F%22%20%2F%3E%0D%0A%20%20%3Clicense%20rdf%3Aresource%3D%22http%3A%2F%2Fwww.eff.org%2FIP%2FOpen_licenses%2Feff_oal.html%22%20%2F%3E%0D%0A%0D%0A%3C%2FWork%3E%0D%0A%0D%0A%3CLicense%20rdf%3Aabout%3D%22http%3A%2F%2Fcreativecommons.org%2Flicenses%2Fby-nc-nd%2F2.0%2F%22%3E%0D%0A%20%20%3Cpermits%20rdf%3Aresource%3D%22http%3A%2F%2Fweb.resource.org%2Fcc%2FReproduction%22%20%2F%3E%0D%0A%20%20%3Cpermits%20rdf%3Aresource%3D%22http%3A%2F%2Fweb.resource.org%2Fcc%2FDistribution%22%20%2F%3E%0D%0A%20%20%3Crequires%20rdf%3Aresource%3D%22http%3A%2F%2Fweb.resource.org%2Fcc%2FNotice%22%20%2F%3E%0D%0A%20%20%3Crequires%20rdf%3Aresource%3D%22http%3A%2F%2Fweb.resource.org%2Fcc%2FAttribution%22%20%2F%3E%0D%0A%20%20%3Cprohibits%20rdf%3Aresource%3D%22http%3A%2F%2Fweb.resource.org%2Fcc%2FCommercialUse%22%20%2F%3E%0D%0A%0D%0A%3C%2FLicense%3E%0D%0A%3C%2Frdf%3ARDF%3E" />

* '''Pro:''' It includes the data in the HTML document; it doesn't break old browsers; it's standard-compliant; it uses the proper element (link) for associating metadata
* '''Con:''' It requires the RDF to be URL-encoded; it's hard to read; not all URL libraries can recognize or decode a data: URL; Creative Commons tools ([[MozCC]], [[CcLookup]], etc) don't support it, likely browsers don't/won't support it; not a preferred method

See also: [http://software.hixie.ch/utilities/cgi/data/data data: URI kitchen] (tool for making data: URLs), [http://www.mozilla.org/quality/networking/docs/aboutdata.html about data: URLs] (from Mozilla)

=== In the HEAD ===

RDF can be included inline in the <code><head></code> tag of the document, along with the title and any <code><link></code>s.

* '''Pro:''' Should be ignored by browsers.
* '''Con:''' Not all users may be able to. Doesn't validate.

=== In the BODY ===

RDF can be included inline in the <code><body></code> of the document, like any other HTML.

* '''Pro:''' Super simple.
* '''Con:''' RDF needs to be formatted as attributes. May cause problems with some browsers. Doesn't validate.

=== In META ===

A reference to the license a web page is offered under may be included in a <code><meta></code> tag in the document's <code><head></code>.

Example:

<pre>
<meta name="DC.rights"
scheme="DCTERMS.URI"
content="http://creativecommons.org/licenses/by-sa/2.0/" />
<meta name="DC.rights"
content="(c) 2002 Gnomovision Records" />
</pre>

* '''Pro:''' Won't break anything.
* '''Con:''' Not all users may be able to. '''Not RDF''' (included in this list due to obviousness). Limited to statements about the current page.

Further reading: [http://dublincore.org/documents/dcq-html/ Expressing Dublin Core in HTML/XHTML meta and link elements]

=== As an element attribute ===

A <code>rel="license"</code> attribute may be added to an "<code>a</code>" anchor element in HTML/XHTML. Example:

<pre>
This work is licensed under the <a rel="license"
href="http://creativecommons.org/licenses/by/2.0/">Creative
Commons Attribution License</a>.
</pre>

* '''Pro:''' Super simple, validates.
* '''Con:''' '''Not RDF''' (however may be used to generate RDF in a GRDDL fashion). Limited to statements about the current page.

Further reading: [http://lists.ibiblio.org/pipermail/cc-metadata/2004-February/000290.html small-s semantic web] and CC.

=== In a comment ===

The RDF can be included in a <code></code> in the document. The TrackBack system embeds its RDF in this way.

Note that if a literal "--" is included in a comment, it must be encoded as "--".

* '''Pro:''' Won't break anything. Super simple.
* '''Con:''' Not supported by all readers. Makes people queasy.

=== Conclusions ===

RDF readers should support all of these methods.

We use and recommend the "in the comment" method, since it's easy and won't break anything. Hopefully tools that don't yet support it will be upgraded to do so. Two examples: [http://www.yergler.net/projects/ccvalidator/ ccValidator] and [http://www.yergler.net/projects/mozcc/ mozCC].

Further reading: [http://esw.w3.org/topic/EmbeddingRDFinHTML Embedding RDF in HTML]

Rewrite Metadata Validator/SoC 2008

2008-06-03T18:51:12Z

Hdworak: Related Web applications, Web framework and libraries

[[Category:Challenge]]
[[Category:Developer]]
[[Category:Metadata]]
[[Category:Opensource]]
[[Category:Project]]
== Introduction ==
Creative Commons [http://code.google.com/soc/2008/cc/about.html participates] in [http://code.google.com/soc/2008/ Google Summer of Code™] and has accepted a proposal (see the [http://code.google.com/soc/2008/cc/appinfo.html?csaid=3C284456287C63B9 abstract]) of [http://hugo.dworak.info Hugo Dworak] based on its description of a task to [[Rewrite Metadata Validator|rewrite]] its now-defunct [http://validator.creativecommons.org metadata validator]. [http://creativecommons.org/about/people/#83 Asheesh Laroia] has been assigned as a mentor of the project. The work began on May 26th, 2008 as per the [http://code.google.com/soc/2008/faqs.html#0.1_timeline project timeline].
== Synopsis ==
A licensor or a licensee wants to check whether a digitally embedded Creative Commons license associated with a file (a Web site in particular) is valid, does not use deprecated means to express it, and matches what the user has been expecting. Therefore, one opens the Web site and either pastes the source code as a direct input, uploads a file, or provides a link to retrieve it. The software displays the results of the analysis in a human-readable manner.
== Proposed timeline ==
*Week 1 — Preparing the SVN/Git or alike repository. Installing the required framework and libraries. Setting up temporary input (source code) and output (result) facilities. Testing the Python Web environment.
*Week 2 — Parsing cc-related RDFa information from well-formed XHTML files.
*Week 3 — Parsing cc-related RDF comments embedded in the XHTML code and those put directly in "head" and "body" elements.
*Week 4 — Parsing cc-related RDF files linked externally or embedded in the "link" element in the header section of the XHTML.
*Week 5 — Parsing cc-related dc-style XHTML-conforming information (embedded in "meta" elements or anchors).
*Week 6 — Ability to clean up the invalid XHTML code whenever possible.
*'''Midterm milestone''' — Developing a raw metadata validator capable of parsing Web sites and outputting valid cc-related information about them in a human-readable fashion.
*Week 7 — Parsing cc-related information contained within syndication feeds (RSS 1.0, RSS 2.0, Atom 1.0).
*Week 8 — Ability to submit an URI to the Web site to be parsed and upload a file using a form. Auto-detection of the MIME content type of such submissions based on HTTP headers and file extension. Ability for the user to override the automatic choice.
*Week 9 — Traversal of embedded objects and special links to obtain more information about licensing terms.
*Week 10 — Generating and storing statistics about the effects of validation such as: content type, errors, means of input, types of licences.
*Week 11 — Extensive testing and providing automatic test suites covering all of the aforementioned capabilities.
*Week 12 — Writing the documentation summarising the architecture. Making the application more user-friendly. Cloning the layout of the Creative Commons Web site.
*'''Final milestone''' — A full-fledged Web application capable of parsing licensing information from a variety of sources.
== Related Web applications ==
* [http://validator.w3.org/ The W3C Markup Validation Service]
* [http://jigsaw.w3.org/css-validator/ The W3C CSS Validation Service]
* [http://www.w3.org/RDF/Validator/ The W3C RDF Validation Service]
* [http://www.w3.org/2007/08/pyRdfa/ RDFa Distiller]
* [http://hcs.science.uva.nl/projects/SWI-Prolog/packages/sgml/online.html Online SWI-Prolog RDF parser demo]
== Web framework and libraries ==
Once [http://www.python.org/ Python] becomes the programming language of the application, one has to decide about the software to be used in building a [http://en.wikipedia.org/wiki/Web_application Web application], so that not everything is written from scratch. First of all, the [http://en.wikipedia.org/wiki/Web_application_framework framework] which will handle the tasks that are typically found in Web applications such as the implementation of the [http://en.wikipedia.org/wiki/Model-view-controller MVC pattern], [http://en.wikipedia.org/wiki/Web_application_framework#URL_mapping URL mapping], and so on. There are several [http://wiki.python.org/moin/WebFrameworks Web application frameworks in Python], for instance [http://www.djangoproject.com/ Django] and [http://www.turbogears.org/ TurboGears]. As Creative Commons uses the BSD-like licensed [http://www.pylonshq.com/ Pylons], this is the preferred choice for the project. Pylons supports many [http://en.wikipedia.org/wiki/Web_template_system template systems], like [http://www.makotemplates.org/ Mako] and [http://jinja.pocoo.org/ Jinja], and it is our design choice to use the BSD-like licensed [http://genshi.edgewall.org/ Genshi] for this purpose.

Other than the above, it comes to choosing Python packages to facilitate parsing and extracting information from the documents. To detect the character encoding of files provided by the users of the validator, the dual-licensed (cc-by and LGPL 3 or later) [http://cthedot.de/encutils/ encutils] shall be used. It is powered by the [http://chardet.feedparser.org/ Universal Encoding Detector] (if present) and is designed to handle XML (including XHTML and RSS) and HTML. Next, one cannot expect that the users will provide only well-formed documents, therefore [http://utidylib.berlios.de/ µTidylib] (available under a MIT-style license) and the BSD-like licensed [http://www.crummy.com/software/BeautifulSoup/ Beautiful Soup] may be used to clean up the ill-formed mark-up, so that it can still be parsed.

Once the document is well-formed and decoded properly, one can proceed to extracting its embedded information relevant to the license terms. It can be represented in [[Extend Metadata|numerous different ways]]. For instance, RDF data can be provided inside comments or as elements inside head and body elements. Such data can also be encoded using a [http://en.wikipedia.org/wiki/Data:_URI_scheme data: URI scheme] or linked externally using the link element. To parse RDF data one can use the BSD-like licensed [http://rdflib.net/ RDFLib], the dual-licensed (W3C Software License and GNU GPL 2 or newer) [http://infomesh.net/2003/rdfparser/ rdfxml.py], and many others. To do the same with RDFa one can employ [http://rdfa.digitalbazaar.com/librdfa/ librdfa] (which has Python bindings and is licensed under GNU LGPL 2.1 or newer) or the MIT-style licensed [http://pypi.python.org/pypi/rdfadict rdfadict].

Rewrite Metadata Validator

2008-05-26T22:03:05Z

Hdworak: Work in progress (Google Summer of Code™ 2008)

{{Challenge
|related_to=CcREL
|challenge_type=Developer
|tags=gsoc, metadata
|is_complete=no
}}

{{Infobox|'''Work in progress'''

A proposal based on this idea has been accepted for [http://code.google.com/soc/2008/ Google Summer of Code™ 2008] and the project is currently a work in progress. Read more about it on its [[Rewrite Metadata Validator/SoC 2008|dedicated page]].}}
Creative Commons historically ran a [http://validator.creativecommons.org metadata validator] (also developed in response to a challenge) which allowed users to verify they had made the correct assertions about their work. Since moving to [[RDFa]], the validator has become less relevant and is in need of a major overhaul (perhaps even rewrite).

A minimal implementation will:

* support all recommended methods, historical and current, of embedding CC license information in a web page (see the [[HTML|file format page]] for more information)
* alert users that embedding using historical methods has been deprecated
* support making assertions about embedded objects

An awesome implementation will:

* support validating CC license inclusion in non-HTML formats (ie, [[Syndication]] feeds), auto-detecting the validation suite to use basic on returned content-type and allowing users to override if necessary
* traverse special links (ie, [[CcPlus|cc:morePermissions]], [[ccREL|cc:attributionURL]], [[Web Statement|xmpRights:WebStatement]]) looking for additional metadata to provide a complete picture of the published metadata.

Rewrite Metadata Validator/SoC 2008

2008-05-26T21:37:49Z

Hdworak: New page: Category:Challenge Category:Developer Category:Metadata Category:Opensource Category:Project == Introduction == Creative Commons [http://code.google.com/soc/2008/cc/abo...