Trackbacks, and spammers, and DDoS, oh my!

The Idea

Before I give you all the dark news about this, let me set out my position: I really, really think that repositories communicating the papers that are cited and referenced to each other is a really good thing. If a paper was deposited in the Oxford archive, and it referenced a paper held in a different repository, say in Southampton's EPrints archive, I think that it is a really fantastic idea to let the Oxford archive tell the Southampton one about it.

And I decided to do something about it - I added two linkback facilities to the archive's user interface, allowing both trackbacks and pingbacks to be archived by the system. I adopted the pre-existing "standards" - really, they are just rough api's - because I think we have all learned our lessons about making up new APIs for basic tasks.

What is Trackback?

Trackback is an agreed technique from the blogging world. Many blogging systems have it built in, and it enables one blog post to explicitly reference and talk about another post, made on a remote blog somewhere. It does this by POSTing a number of form-encoded parameters to a specific URL, specific to the item that is being referenced. The parameters include things like title, abstract and URL of the item making the reference.

So on the surface, it appears that this trackback idea performs exactly what I was looking for.

BUT! Trackback has massive, gaping flaws, akin to the flaws in the email system which is full of spam. For one, all trackbacks are trusted in the basic specifications. No checking that the URL exists, no checking of the text for relevance, etc.

Pingback is a slightly different system, in that all that is passed, is the URL of the referencing item. It is then up to the remote server to go and get the requested page and parse that out to find the reference. (The next version of the specification is crying out to recommend microformats et al, in my opinion)

So, these systems, trackback and pingback, have been on trial in the live system for about 4 or 5 months, and I am sure you all want to hear my conclusions:

• Don't implement Trackback as it is defined in its specifications... seriously. It is a poorly designed method, with so much slack that it is a spammers goldmine.
• Even after adding in some safeguards to the Trackback method, such as parsing the supposed referencing page and checking for HTML and the presence of the supposed link, it was still possible for spammers to get through.
• When I implemented Trackbacks, I did so with the full knowledge that I might have to stand at a safe distance and nuke the lot. Here is the Trackback model used in the Fedora repository - A DC datastream containing the POSTed information mapped to simple dublin core and a RELS-EXT RDF entry asserting that this Fedora Object <info:fedora/trackback-id>, referenced <dcterms:references> the main item in the archive <info:fedora/item-id>. As the user interface for the archive gets the graph for that object, it was easy to get the trackbacks out as well. Having separate objects for the trackbacks and not changing the referenced item at all, made it very easy to remove the trackbacks at the end.
• The Trackback system did get hit, once the spammers found a way around my safeguards. So, yes, the trackbacks got 'nuked' and the system turned off.
• Currently, the system is under a sort of mini-DDoS, from the spammer's botnet trying to make trackbacks and overloading the session tracking system.
• The Pingback system, utilising XML-RPC calls, was never hit by spam. I still turned it off, as the safeguards on this system were equivalent to the Trackback system.

So, how do we go on from this quagmire of spam?

Well, for one, if I had time (and resources) to pass all requests through spamassassin or pay for akismet, that would have cut down the number drastically. Also, if I had time to sit and moderate all the linkbacks, again, spam would be nipped in the bud.

So, while I truly believe that this type of system is the future, it certainly isn't the case that it can be a system that can just be turned on and the responsibility for maintaining it added to an already full workload.

Alternatives?

White-listing sites may be one method. To limit the application to sharing references between institutions, you could use the PGP idea - a web of trust; a technique of encrypting the passed information with a private key that resolves to a public key from a white-listed institution. This would ensure that the passed reference really was from a given institution. This should be more flexible than requiring a single IP address to accept references from.

(There is always the chance that the private key could be leaked and made not-so-private by the institution, but that would have to be their responsibility. Any spam from a mistake of this sort would be directly attributed to those at fault!)

A slower, far less accurate but more traditional method, would be for a given institution to harvest references from all the other repositories it knows about. I really don't think this is workable, but has the pro that a harvester can be sure that a reference links to a given URL, (barring the more and more common DNS poisoning attacks)

Ditching the DB-based blog for a semantic one

Why Yet-another-blog-engine?

Well, blog engines tend to do the same things, their functionality is derived by simple views on a relational DB. To a large extent, I think that this RDB reliance has shaped the scope of what you can do with a blog and also I really feel it has guided how the blog (and related publishing) technology has developed.

Things like blog export, and interlinking of blogs and the bloggers who power the system are seen as extras, features added as plugins or as an additional service - this needs to change! Let's see what naturally happens when we try to build a system with a more interesting backend.

So, what to replace the RDB with?

Simply put, the data which a blog needs to function normally can be modelled in an objectstore, by linking items together by RDF predicates. Specifically, there is a namespace created by the SIOC project, aimed at defining social networks and their inter-linking in a semantic way - http://www.sioc-project.org/. I have a strong hunch that by using this work, a whole load of extra possibilities will emerge.

(Aside from the obvious benefits of simple export and reuse of objects, being able to make a single comment on more than one blog post from more than one blog, and so on.)

Okay... what's the plan?

So, my coding bias for objectstore and framework language is FedoraCommons and python, so no surprise there. It also means that I'll be reusing my code, so each object will have a OAI-ORE aggregration and good search intergration (via Apache Solr).

Modelling the blog:

Luckily, the good folks at the SIOC project have done a good load of the work for me, and having read through their work, I can say that I see no problem with it for my purposes. This means that using their namespace (http://rdfs.org/sioc/ns#), I can adopt the structure of classes, helpfully illustrated here

The first class objects from or subclassed from SIOC therefore will be as follows (a first class object has a 1-to-1 parity with the underlying Fedora objects):

User [contains or links to FOAF record] - Post [text and zero or more attachments] - Forum(Blog) [Dublin Core] - Post(Comment) [text] - Site [Dublin Core]

Other first class objects:

Link [DC record] and Petition [ Later ;) ]

Link speaks for itself. If a post contains a link, the link is promoted to an object and the post will connect to the link object. If someone else uses that link, the previous object is reused.

Petition is a social experiment which I will go into later ;)

Now for the behaviour of the blog - I am envisioning an academically focussed blog, so the social network, persistance, trust and discourse are important features to consider.

Users - accounts and blogs

Frankly, I am tired of writing authentication systems. So is everyone else. Users are tired of having an account per site too. Thank god for OpenID then :)

Right, so we have a working, live system for authenticating people, but what about authorising? Authenticate to comment, that's self-explanatory. But this is where we can do some interesting things:

The blog engine is 'seeded' by a blog author or authors, most likely the same people that installed the engine. Borrowing a common idea, these seeds can invite other people to have a publishing account. This is done by an indication of trust - at a technical level <uri-inviter> <trust:trust10> <uri-invitee>, with the <uri-invitee> being the User object created to correspond to a given OpenID. The next time they log in, the ability to create a blog should be apparent.

(trust namespace: http://trust.mindswap.org/ont/trust.owl)

Why not <foaf:knows>? Because that is saved for later :) <trust:trust10> is a predicate intended as a way for someone to fully vouch for someone else - you may know people, but that doesn't mean they should automatically gain blogging rights just because you have those rights.

Any post (post, comment or link) can be tagged as interesting to a User, via the <foaf:interest> predicate - declaring "I am interested in this thing" The payoff to the user is that their page (every User is an object, remember) will display the things they have marked as interesting. If a User declares that another User's blog is 'interesting' then all the latter User's posts will be accessible as well from here.

There are two forms of free-text tags; a trusted tag and a normal tag. Trusted tags are those placed on a Post, Blog or User, by the author/owner of that object - a statement about what the author feels is the subject of the object. An author can also tag themselves, and this is to give extra indication about what they normally blog or comment on, in addition to the tags they've put on their own items.

A normal tag can be placed by any authenticated User on any Post or Blog. Expected functionality, really.

I haven't tripped across a suitable ontology for these two, and I'd really like to use an existing one so feel free to add a comment if you have an idea of one.

Now, I mentioned a curious thing called a Petition - this is a second method to gain trust in the social network. A User can make a Petition, stating a brief summary of what they'll like to write about, and then tag the Petition with its subjects.

This is where it gets semantically fun - a Petition will be visible to existing blog posters with the following filters available:

• Show Petitions that have tags in common with mine, from Users I trust
• Show Petitions that have tags in common with mine, from Users I know
• Show Petitions that have tags in common with mine, from Users my friends know
• Show Petitions from Users who I've shown interest in
• Show Petitions from Users who my friends show interest in
• Show Petitions that have tags in common with mine
• Show most recent Petitions

It doesn't take much more time to see that the information already in the triplestore can be used to create some really interesting filters.

Now, a User can then decide to place a level of trust in a given Petitioner. The actual mechanics of what is required to elevate the trust placed in the Petitioner is up to the system installer, but a few interesting things can be used: A requirement for the combined <trust:trust> given to a User equivalent to a certain level, maybe a tiered system (2 trust9's are equivalent to 1 trust10, etc)

The bottom line is that this trust system means that there is no requirement for a super User as the network should be self-regulating - take the trust away, and the Petitioner loses their Blog. (It also doesn't mean that having a Super user is a bad idea!)

Posts and Posting

I hate the word blog... I've been using it as a crutch, as what I'd like this to be is a site to have a voice on. By using semantic relationships, it is quite possible to view it as you might a blog, but you might also view it like a forum, with posts and threaded comments. The underlying information and connections are the same, but the way you can view and present this become a lot more flexible.

So when I say Post, I mean it in a twitter/blogger/thread-reply kind of way ;)

The Post objects have a summary (200 char limit) and an optional body (a blog 'post') - no title!. They can be tagged of course and the post can also hold attachments for download, or embedding here or elsewhere. (All resources have dereferenceable URIs, and so can be linked to directly)

So a Post can be a 'tweet', or a 'blog' post - it's all the same thing. However, on the user's jumpoff page, the summaries are listed.

I'll have a go at putting something together, to see what works and what doesn't, so watch this space.

Creating a web application from scratch, backed by Fedora-Commons and Apache Solr (Part 1)

(Part 1 will detail the installation and setup of the basic system, services and libraries needed for a Fedora-Commons/Apache Solr backed web 'service'. Subsequent parts will deal with configuring and feeding the search engine, and constructing a web interface to handle article/blog/comment posting and using OpenID for authentication.)

Step 1 - Get a nice clean linux distribution focused on use on servers.

I am using Ubuntu JeOS, as it will be hosted on a VMware virtual machine. This also means that the walkthrough that follows will be very debian specific.

See the following pages for help:

This page aims at documenting how to create virtual appliance using Ubuntu Server Edition's JeOS.

This page has snapshots of the entire install process and also information on how to set up a LAMP stack but if you want to follow this guide, don't install any of the applications it asks. It is only useful for our purposes up until the first reboot, before the guide does things we don't need, like activating the root user account, or adding PHP.

Note that the user name I will be using in this guide is simply 'user' and I will refer to this as either 'user' or 'username'. Replace this with whatever the username was that you chose during installation.

Step 2 - install it and set up networking and firewalls

Now, firewalls aren't as critical as you might think, especially if you have installed something like JeOS which has nothing really running as default. But there are some very handy tricks to help stop abuse from malicious script kiddies.

(For example, my favorite two liner I always add to the iptables firewall script is a couple of lines that rate limits ssh access tries to 3 attempts every 180 seconds. (Note, that the following doesn't immediately ACCEPT it, it passes it into a iptables chain called 'TRUSTED' which deals with what may be genuine attempts at access. If you wanted to just accept it, change TRUSTED to ACCEPT.)

# Rate limit SSH attempts.
iptables -A INPUT -p tcp -m tcp --dport ssh -m state --state NEW \
-m recent --hitcount 3 --seconds 180 --update -j DROP

# Allow first attempts through
iptables -A INPUT -p tcp -m tcp --dport ssh -m state --state NEW \
-m recent --set -j TRUSTED

NB Something along these lines would be fine to rate-limit upload attempts to Fedora as well.

[Edit: seems there was something awry with the following script - as illustrated here. Thanks to the Rubric team. I've added their fix :) But their fix may not be enough, so I'd advise not applying this until everything is installed and working correctly! I'll test it out as soon as I can. ]
Full example firewall script, including this snippet and port opening for tomcat, http/https, and ssh. (As this is from my home server, it'll include a few other services that you may not need for this walkthrough).

If you wish to use the SSL connection to Fedora on port 8443, remember to open that port as well!

Step 3 - Install all updates and get the basic applications we will need

Get a root prompt on a commandline:

e.g.

[user@server]$ sudo -s
[root@server]#

Then make sure you can a) connect to the internet and that b) the server is up to date:

[root@server]# apt-get update
[..... lots of lines of stuff ....]
Hit http://gb.archive.ubuntu.com gutsy-updates/multiverse Sources [1708B]
Fetched 278kB in 0s (319kB/s)
Reading package lists... Done

[root@server]# apt-get upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be upgraded:
[ whatever packages that need to be upgraded will be listed here]
X upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 2497kB of archives.
After unpacking 16.4kB of additional disk space will be used.
Do you want to continue [Y/n]? Y
[ .... lots of lines of packages installing hopefully without error ..... ]

Hopefully, once those are installed, your machine will be up to date. Now to install on all the necessary packages:

[root@server]# apt-get install build-essential python-dev mysql-server sun-java5-jdk openssh-server python-mysqldb python-pysqlite2

Just let those install, but be aware that certain packages should ask you for information during installation, such as the required default root password for MySQL and a prompt will ask you if you agree with the Sun licence for Java.

Also, note that at this point, you should be able to SSH into the machine to continue working on it. It makes it a lot easier to cut and paste from guides if you do!

Step 4 - install some python libraries using Easy Install

Go here: http://peak.telecommunity.com/DevCenter/EasyInstall and download and install the ez_install.py script as it shows. Simply running it as root should do the trick:

[root@server]# wget http://peak.telecommunity.com/dist/ez_setup.py
[root@server]# python ez_install.py

[Edit: I did originally write the first part of this for Fedora 2.2, but since the REST api for Fedora 3.0 looks pretty damn usable, I've re-written this guide for version 3. Removing the installation/configuration for the SOAP client, drastically reduces the library dependancies this needs.]

So, we need to install some python libraries for later, iCalendar format (vobject) , OpenID consumer library (python-openid), and also install other miscellaneous things, such as a library that can generate UUIDs and a very good web framework called Pylons:

[root@server]# easy_install python-openid
[root@server]# easy_install uuid
[root@server]# easy_install vobject
[root@server]# easy_install pylons

(NB we have already installed the python libraries to interact with MySQL and SQLite with the apt-get install command earlier. It is best to install the latest stable packages for the items above, which is why they are installed through easy_install.)

Step 5 - Get Fedora-Commons and Apache Solr.

Either just blindly download the packages I tell you to:

[user@server]$ wget http://downloads.sourceforge.net/fedora-commons/fedora-3.0b1-installer.jar
[user@server]$ wget http://apache.rmplc.co.uk/lucene/solr/1.2/apache-solr-1.2.0.tgz

Or better, download them from the homepages of the projects themselves, using links2

Install a text-based web-browser and browse and download the packages that way (A manual page for links2):

[root@server]# apt-get install links2

When that has finished installing, you can drop out of your root session (press Ctrl+D, or type 'exit') and download the relevant applications:

[root@server]# exit
[user@server]$ links2

Don't be alarmed, it's meant to blank the screen! Press the letter 'g' and an location bar prompt will appear.

First let's go to the Fedora commons site so type in 'http://www.fedora-commons.org/' and press enter. Use the cursor keys to go down and click (press return) on the 'Download Fedora 3.0 beta 1' link (24/02/2008). Scroll down a bit, and you should see a link to download the installer. You will be presented with the 'save jar file' dialog, so save the fedora installer jar file.

Now, let's get the search appliance, Solr. Got to 'http://lucene.apache.org/solr/' and click on the 'download' link. Choose a mirror, go into the 1.2/ folder on that mirror and download the 'apache-solr-1.2.0.tgz' file. Press 'q' to quit links2.

Step 6 - Make the server environment ready for Fedora Commons

If you now list the home directory, you should see something like this:

[user@server]:~$ ls
apache-solr-1.2.0.tgz doc ez_setup.py fedora-3.0b1-installer.jar

We will need the following:

1. A directory to store Fedora's root directory (config files, logs, libraries, and default Tomcat instance)
2. A mysql database and account for Fedora to use
3. (Optional) A large filesystem to hold Fedora's data storage directory

Point 1 then - I chose to store the Fedora root directory at /opt/fedora30b1 -

[user@server]$ sudo -s
[root@server]# mkdir /opt/fedora30b1

Let the user own it: (Remember change 'user' to whatever your user is actually called!)

[root@server]# chown user:user /opt/fedora30b1

(Optional) And to aid upgrading, create a symlink at /opt/fedora to this folder:

[root@server]# ln -s /opt/fedora30b1 /opt/fedora

Fedora needs certain environment variables to be set up now, FEDORA_HOME and JAVA_HOME at the very least. Open up the system wide profile (/etc/profile) and add them in there. (I'm using the nano editor, vim is also available from a default JeOS install.)

[root@server]# nano -w /etc/profile

And add the following lines to the end of the file (also, note that there *must not* be any gaps either side of the '=' character, as tempting as it might be to press space to space it out to look better.):

# If you did not create the symlink, just point directly at your Fedora root
FEDORA_HOME=/opt/fedora30b1
# or if you did do the 'ln -s ...' step, use this instead:
FEDORA_HOME=/opt/fedora

export FEDORA_HOME

# If you did not create the symlink, just point directly at your tomcat root
CATALINA_HOME=/opt/fedora30b1/tomcat
# or if you did do the 'ln -s ...' step, use this instead:
CATALINA_HOME=/opt/fedora/tomcat

export CATALINA_HOME

JAVA_HOME=/usr/lib/jvm/java-1.5.0-sun
export JAVA_HOME

Save the file (Ctrl-X in nano)

Now, to check that this has worked, type the command 'exit' a few times to logout and then log back in again as your default user. If things have worked well, the following commands should work:

[user@server]$ echo $FEDORA_HOME
/opt/fedora30b1

[Or '/opt/fedora' depending on what you chose.]

[user@server]$ echo $JAVA_HOME
/usr/lib/jvm/java-1.5.0-sun

Now to sort out MySQL. Remember that default root password you set for MySQL? You'll need it now.

[user@server]$ mysql -uroot -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 10
Server version: 5.0.45-Debian_1ubuntu3.1-log Debian etch distribution

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql>

Now issue the following commands:

mysql> create database fedora30;
Query OK, 1 row affected (0.00 sec)

mysql> grant all on fedora30.* to 'fedoraAdmin'@'localhost' identified by 'PUTYOURPASSWORDHERE';
Query OK, 0 rows affected (0.00 sec)

mysql> flush privileges;
Query OK, 0 rows affected (0.00 sec)

mysql> ALTER DATABASE fedora30 DEFAULT CHARACTER SET utf8;
Query OK, 1 row affected (0.00 sec)

mysql> ALTER DATABASE fedora30 DEFAULT COLLATE utf8_bin;
Query OK, 1 row affected (0.00 sec)


mysql> exit
Bye

[user@server]$

(NB You may or may not need to add the utf-8 configuration lines for your particular version of MySQL, but as far as I know, the commands are harmless if you don't need them and utterly crucial if you do. Well, crucial unless you are dealing purely with ascii, but could you really guarantee that?)

Step 7 - install Fedora commons 3.0b1

(Note - Official installation guide is here)

Go to the location where you saved the fedora installer, probably the user's home directory and run the installer. I'll include the entire installation dialog here. Where the response is blank, I simply pressed enter to accept the default.

[user@server]$ cd /home/user
[user@server]$ java -jar fedora-3.0b1-installer.jar

***********************
Fedora Installation
***********************

To install Fedora, please answer the following questions.
Enter CANCEL at any time to abort the installation.
Detailed installation instructions are available at:
http://www.fedora.info/download/

Installation type
-----------------
The 'quick' install is designed to get you up and running with Fedora
as quickly and easily as possible. It will install Tomcat and an
embedded version of the McKoi database. SSL support and XACML policy
enforcement will be disabled.
For more options, including the choice of hostname, ports, security,
and databases, select 'custom'.
To install only the Fedora client software, enter 'client'.

Options : quick, custom, client

Enter a value ==> custom


Fedora home directory
---------------------
This is the base directory for Fedora scripts, configuration files, etc.
Enter the full path where you want to install these files.


Enter a value [default is /opt/fedora] ==>


Fedora administrator password
-----------------------------
Enter the password to use for the Fedora administrator (fedoraAdmin) account.

Enter a value ==> PUTTHEPASSWORDYOUDLIKEHERE


Fedora server host
------------------
The host Fedora will be running on.
If a hostname (e.g. www.example.com) is supplied, a lookup will be
performed and the IP address of the host (not the host name) will be used
in the default Fedora XACML policies.


Enter a value [default is localhost] ==>


Authentication requirement for API-A
------------------------------------
Fedora's management (API-M) interface always requires user authentication.
Require user authentication for Fedora's access (API-A) interface?

Options : true, false

Enter a value [default is false] ==>


SSL availability
----------------
Should Fedora be available via SSL? Note: this does not preclude
regular HTTP access; it just indicates that it should be possible for
Fedora to be accessed over SSL.

Options : true, false

Enter a value [default is true] ==>


SSL required for API-A
----------------------
Should API-A be accessible exclusively via SSL? If true, requests
to access API-A URLs will be automatically redirected to the secure port.

Options : true, false

Enter a value [default is false] ==>


SSL required for API-M
----------------------
Should API-M be accessible exclusively via SSL? If true, requests
to access API-M URLs will be automatically redirected to the secure port.

Options : true, false

Enter a value [default is true] ==> false


Servlet engine
--------------
Which servlet engine will Fedora be running in?
Enter 'included' to use the bundled Tomcat 5.5.23 server.
To use your own, existing installation of Tomcat, enter 'existingTomcat'.
Enter 'other' to use a different servlet container.

Options : included, existingTomcat, other

Enter a value [default is included] ==> included


Tomcat home directory
---------------------
Please provide the full path to your existing Tomcat installation, or
the path where you plan to install the bundled Tomcat.


Enter a value [default is /opt/fedora/tomcat] ==>


Tomcat HTTP port
----------------
Which HTTP port (non-SSL) should Tomcat listen on? This can be changed
later in Tomcat's server.xml file.


Enter a value [default is 8080] ==>


Tomcat shutdown port
--------------------
Which port should Tomcat use for shutting down? Make sure this doesn't
conflict with an existing service. This can be changed later in Tomcat's
server.xml file.


Enter a value [default is 8005] ==>


Tomcat Secure HTTP port
-----------------------
Which port (SSL) should Tomcat listen on? This can be changed
later in Tomcat's server.xml file.


Enter a value [default is 8443] ==>


Keystore file
-------------
For SSL support, Tomcat requires a keystore file.
If the keystore file is located in the default location expected by
Tomcat (a file named .keystore in the user home directory under which
Tomcat is running), enter 'default'.
Otherwise, please enter the full path to your keystore file, or, enter
'included' to use the the sample, self-signed certificate) provided by
the installer.
For more information about the keystore file, please consult:
http://tomcat.apache.org/tomcat-5.5-doc/ssl-howto.html.

Enter a value ==> included


Policy enforcement enabled
--------------------------
Should XACML policy enforcement be enabled? Note: This will put a set of
default security policies in play for your Fedora server.

Options : true, false

Enter a value [default is true] ==> false


Enable Resource Index
---------------------
Enable the Resource Index?

Options : true, false

Enter a value [default is false] ==> true


Enable REST-API
---------------
Enable the REST-API? The REST-API is an EXPERIMENTAL feature that exposes
the Fedora API with a REST-style interface. In particular, URL endpoints
should not be considered final, nor has policy enforcement been evaluated.
For more information about the REST-API, see
http://www.fedora.info/wiki/index.php/RESTful_Fedora_Proposal

Options : true, false

Enter a value [default is false] ==> true


Database
--------
Please select the database you will be using with
Fedora. The supported databases are McKoi, MySQL, Oracle and Postgres.
If you do not have a database ready for use by Fedora or would prefer to
use the embedded version of McKoi bundled with Fedora, enter 'included'.

Options : mckoi, mysql, oracle, postgresql, included

Enter a value ==> mysql


MySQL JDBC driver
-----------------
You may either use the included JDBC driver or your own copy.
Enter 'included' to use the included JDBC driver, or, enter the location
(full path) of the driver.


Enter a value [default is included] ==>


Database username
-----------------
Enter the database username Fedora will use to connect to the Fedora database.

Enter a value ==> fedoraAdmin


Database password
-----------------
Enter the database password Fedora will use to connect to the Fedora database.

Enter a value ==> PUTYOURDBPASSWORDHERE


JDBC URL
--------
Please enter the JDBC URL.


Enter a value [default is jdbc:mysql://localhost/fedora30?useUnicode=true&characterEncoding=UTF-8&autoReconnect=true] ==>


JDBC DriverClass
----------------
Please enter the JDBC driver class.


Enter a value [default is com.mysql.jdbc.Driver] ==>


Successfully connected to MySQL
Deploy local services and demos
-------------------------------
Several sample back-end services are included with this distribution.
These are required if you want to use the demonstration objects.
If you'd like these to be automatically deployed, enter 'true'.
Otherwise, the installer will put the files in your FEDORA_HOME/install
directory in case you want to deploy them later.

Options : true, false

Enter a value [default is true] ==>


Preparing FEDORA_HOME...
Configuring fedora.fcfg
Installing beSecurity
Installing Tomcat...
Preparing fedora.war...
Processing web.xml
Deploying fedora.war...
Deploying fop.war...
Deploying imagemanip.war...
Deploying saxon.war...
Deploying fedora-demo.war...
Installation complete.

----------------------------------------------------------------------
Before starting Fedora, please ensure that any required environment
variables are correctly defined
(e.g. FEDORA_HOME, JAVA_HOME, JAVA_OPTS, CATALINA_HOME).
For more information, please consult the Installation & Configuration
Guide, located online at
http://www.fedora.info/download/ or locally at
/opt/fedora/docs/userdocs/distribution/installation.html
----------------------------------------------------------------------

And that should merrily go away and install and setup Fedora and the bundled Tomcat server for you. Unlike other services you may install, this won't start the Fedora service, nor will it create a handy startup/shutdown script that integrates with you linux startup scripts in /etc/init.d. We will create one later on.

Step 8 - Further configuration of Fedora 3.0

!IMPORTANT! Fix the broken 'mail.jar' library! (Broken, as in the REST api will not work correctly with the version release in 3.0b1)

Get it from here: http://python-fedoracommons-webarchive.googlecode.com/files/mail.jar and use it to replace the mail.jar found in $FEDORA_HOME/tomcat/webapps/fedora/WEB-INF/libs/mail.jar. Restart Tomcat if you need to.

I am keen on UUIDs, and I cannot see a good reason for not using them. I suggest using the fedora id 'namespace' of uuid, so that a fedora URI will look like <info:fedora/uuid:d3733f61-1083-4a3e-b914-5a853c42189b>

It is also trivial to generate these in python, consider the following code:

[user@server]$ python
Python 2.5.1 (r251:54863, Oct 5 2007, 13:36:32)
[GCC 4.1.3 20070929 (prerelease) (Ubuntu 4.1.2-16ubuntu2)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from uuid import uuid4
>>> uuid4().urn[4:]
'uuid:d3733f61-1083-4a3e-b914-5a853c42189b'

To get Fedora to accept these though, the 'uuid' namespace needs to be added to the retainPID region in fedora's configuration file.

[user@server]$ nano -w /opt/fedora/server/config/fedora.fcfg

Press Ctrl-W and search for retainPID. Add in uuid to the list of namespaces (the ordering is not important):

<param name="retainPIDs" value="demo uuid test changeme ...


Step 9 - Installing Solr

[NB you will only have to follow the guide below, but here are the official docs, should you get in trouble
http://wiki.apache.org/solr/SolrInstall - Basic installation
http://wiki.apache.org/solr/SolrTomcat - Tomcat specific things to bear in mind]

Extract the whole archive somewhere on disc and you will see something like this in the apache-solr-1.2 folder:

~/apache-solr-1.2.0$ ls
build.xml CHANGES.txt dist docs example KEYS.txt lib LICENSE.txt NOTICE.txt README.txt src

~/apache-solr-1.2.0$ ls dist
apache-solr-1.2.0.jar apache-solr-1.2.0.war

The easiest thing is to install Solr straight into the instance of Tomcat that Fedora has installed. One thing to be aware of is that search applications eat RAM and Heap for breakfast, so make sure you install it onto a server with plenty of RAM and it would be wise to increase the amount of Heap space available to the Tomcat instance. This can be done by making sure that the environment variable CATALINA_OPTS is set to "-Xmx512m". This can be done inside the catalina.sh script in your /opt/fedora/tomcat/bin directory.

[i.e. just add CATALINA_OPTS="-Xmx512m" at the beginning of the file if it doesn't already exist.]

One final bit of advice before I point you at the rather good installation docs is that you might want to rename the .war file to match with the URL pathname you desire, as the guide relies on Tomcat automatically unpacking the archive:

So, a war called "apache-solr-1.2.0.war" will result in the final app being accessible at http://tomcat-hostname:8080/apache-solr-1.2.0/. We will rename ours when we copy it into Tomcat's webapps directory.

Finally, Solr needs a place to keep its configuration files and its indexes. The indexes themselves have the capability to get huge (1Gb is not unheard of) and need somewhere to be stored. The documentation linked to below will refer to this location as 'your solr home' so it would be wise to make sure that this location has the space to expand. (NB this is not the directory inside Tomcat where the application was unbundled.)

So, let's create a solr home in /opt as we did for fedora (NB change user):

[user@server]$ sudo -s
[root@server]# mkdir /opt/solr
[root@server]# chown user:user /opt/solr

Place the solr.war into Fedora's Tomcat instance:

[root@server]# exit
[user@server]$ pwd
/home/user/apache-solr-1.2.0
[user@server]$ cp dist/apache-solr-1.2.0.war $CATALINA_HOME/webapps/solr.war

Finally, we have to make sure a variable is available in Tomcat's environment; the location of the Solr home directory. Remember that CATALINA_OPTS line we added before? Amend that now to look like:

(E.g. via nano -w $CATALINA_HOME/bin/catalina.sh )

CATALINA_OPTS="-Xmx512m -Dsolr.solr.home=/opt/solr"

Now, as we will shape the Solr search service later on (i.e. choosing the fields to be indexed, and how to index them for faceted searching) we will just copy across the basic solr example, to make sure everything is running fine.

[Make sure you are in the unpacked solr directory:]
[user@server]$ pwd
/home/user/apache-solr-1.2.0
[user@server]$ cp -a example/solr/* /opt/solr
[user@server]$ ls /opt/solr
bin conf README.txt

Adding HTTP authentication to Solr update

First add a username/password to tomcat/conf/tomcat-users.xml:

<tomcat-users>
...
<user username="solradmin" password="XXXXXXXX" roles="solradmin">
...
</user>
Then, in your Solr context, in tomcat/webapps/solr/WEB-INF/web.xml, add the following:

<web-app>

.... usual stuff ....

<security-constraint>
<web-resource-collection>
<web-resource-name>
SolrUpdate
</web-resource-name>
<url-pattern>/update/*</url-pattern>
</web-resource-collection>
<auth-constraint>
<role-name>solradmin</role-name>
</auth-constraint>
</security-constraint>
<!-- Define the Login Configuration for this Application -->
<login-config>
<auth-method>BASIC
<realm-name>Auth needed
</login-config>

</web-app>

NB BASIC authentication sends the password over by plain-text, so this isn't too great but is suitable for a localhost updater. Change this to DIGEST to increase the security, but bear in mind you may need to set the Realm for the Tomcat container and Digest hash mechanism (SHA1, MD5, etc)

(Some good guides to securing Tomcat services are but a Google search away - for example: http://www.unidata.ucar.edu/projects/THREDDS/tech/reference/TomcatSecurity.html )

Step 10 - Test your foundation

Now, we need to start up Fedora, and hopefully, it will all go smoothly:

[user@server]$ cd /opt/fedora/tomcat/bin/
[user@server]$ ./startup.sh
Using CATALINA_BASE: /opt/fedora/tomcat
Using CATALINA_HOME: /opt/fedora/tomcat
Using CATALINA_TMPDIR: /opt/fedora/tomcat/temp
Using JRE_HOME: /usr/lib/jvm/java-1.5.0-sun

Now try these links:

http://localhost:8080/fedora/search
http://localhost:8080/fedora/describe - make sure 'uuid' is one of the retainPIDs
http://localhost:8080/solr/admin Should look like a whole heap of options and bells and whistles.

Any 404 or 500 Server errors means that something has come unstuck. But, if you've followed this guide, using an Ubuntu Gutsy you should be all set without a problem - I just followed it on my home computer without a hitch :)

Next up, we are going to build a pylons interface to do basic CRUD type functionality with the ability to link together items semantically, using the SIOC project's (http://sioc-project.org/ontology) namespace, at http://rdfs.org/sioc/ns#

Deposit by email

(This is a 'if I get time' implementation outline, based on my experiences with other online repositories such as YouTube and Flickr)

1. User registers to let them deposit material by email.
• This must be done to set up default licensing of the submitted material, and also to get their agreement to the standard deposit agreement. The submitter needs to be aware that if they choose this mechanism for a deposit, that it is, by its very nature, less rigorous as there is little opportunity for authentication.
2. They are given an email template and a unique email address to post to:
• The email template
• should be a text/plain file, that can be opened and edited. The format should be forgiving (not XML then) and should be suitable for simple dublin core. I was thinking of a simple (keyword) : ([.*])\n type format, discarding anything which doesn't match.
• Pass by reference should be acceptable for the submission of items.
  
• For Pass by Value, the files should be attached to the email.
  
• The email address target
• for that individual will be of the form (random16letteralphanum)@archivehost.com. Email sent to this within a pre-determined timeframe will be anticipated to be from that person. E.g. Kas3wPr3Ew9SQFer@submissions.ora.ouls.ox.ac.uk
• To help curb the inevitable problems faced by people who seek to type this in (rather than just click or copy and paste), avoid the letters I, O, L and the numbers 1 and 0. This should still be enough for a decent amount of randomised combinations.
  
• A step to harden this process might be to lock down submissions based on the email they are sent from in addition to given them a unique target. But I have a feeling that this may not be necessary.
  
3. An email is sent in reply either:
• giving them their persitent, citable URL to this item (not plural)
• [Thoughts: How about passing an ORE map to the system...? Might have some milage]
  
• Stating why their submission didn't make it (likely a metadata issue.)
  

Yahoo Pipes + Solr's API = RSS feeds for repository submitters

Quick post: Part of the reason for trying to get as many open APIs onto the services I put in place for the repository is so that I don't have to customise things for every department or use; the interested parties can do it for themselves.

As a little proof, I have set up a Pipe that takes repository search results, based on an author's name, and creates an RSS feed from it:

http://pipes.yahoo.com/oxfordrepo/authorfeeds

You should be able to clone this pipe, play with it, or simply use it for it's intended purpose.

Update

I am still here...

Just thought I'd put in a quick note to say that yes, i am still posting to this blog and yes, it is worth waiting for (I hope)

The posts that are sitting in my drafts at the moment are:

Image annotation and how to describe regions of images in a sound and stable manner. From talking to people on the #swig on freenode IRC, most notably Ben Nowack, I think that the format for doing this is pretty much crystallised.

Why is image region annotation important? It gives a consistent mechanism for recording information, such as:

- relating the area on a scanned page to words from an OCR'd text.

- it is easy to identify the same thing in multiple images - for example, identifying a person in multiple photos, labelling a burial site both in a photo and also in a topographical survey map, showing a particular phenotype being expressed on multiple slides of fruitflys, etc.

I think that is enough to whet your appetite for why this is handy to have sorted out early on.

And as this blog is called 'Less Talk, More Code', the answer to 'but how do we add this?' is that I have found a good GPL2 javascript library for drawing regions on images and the format is simple to write for.

Another thing lurking in my draft folder is all to do with "HOWTO: Building a fedora-commons backed web site from scratch, using open source tools" - Part 1 & 2.

My aim is that someone following both parts should end up with a web site that will allow them to do what is normally expected of an archive - handily, some points are listed in this photo from the Jisc-Crig unconference - http://www.flickr.com/photos/wocrig/2197484000/

"Put stuff in, get stuff out, Find stuff in, relationships between objects, annotations" plus content types, edit, event logging and visualisation and record pages built from one or more metadata sources.

Part 1 is the mundane details of installing an OS (Ubuntu JeOS), and then acquiring all the dependencies such as Sun Java 1.5 Fedora, Apache Solr, and the requisite python libraries (PyXML, Pylons, and the libraries I've written) for part 2, building the interface itself.

The key to it I hope is that the build of the python interface is described in such a way that it will help people do the same and allow them to think about what can be done further.

Progress? Part 1 is almost done and I am re-jigging part 2 to be more readable; I am trying to put part 2 into an order so that the simple things are dealt with first and the more complex things later, rather than in a 'function-by-function' structure.

Populating a search engine (Apache Solr) from Fedora Objects

I am going to move fairly fast on this one, with the following assumptions about the person(s) reading this:

• You can at least set up a linux server, with Fedora 2.2 or so using its bundled tomcat without help from me
  
• You have python, and have added the libraries indicated at the top of this post.
• You have pulled the SVN hosted libraries, also from the aforementioned post from:
• svn co https://orasupport.ouls.ox.ac.uk/archive/archive/lib

Let's get started then after I outline why I am using Solr:

Why Solr

From their front page:

Solr is an open source enterprise search server based on the Lucene Java search library, with XML/HTTP and JSON APIs, hit highlighting, faceted search, caching, replication, and a web administration interface. It runs in a Java servlet container such as Tomcat.

It has open APIs, faceted searching, replication and if you have a look at SOLR-303, the developers have added in distributed (i.e. federated) searching over HTTP between Solr instances, the precise functionality is still being refined (cleaner handling of tiebreakers, refinement queries, etc) but is functioning nonetheless.

I think from now on, I would no more code my own web server than I would code my own search engine service.

Getting and Installing Solr

First step is to grab Solr from this page. Pick a mirror that is close to you, and download the Solr package (I am currently using version 1.2), either as the .zip version or the gzipped tar, .tgz.

Extract the whole archive somewhere on disc and you will see something like this in the apache-solr-1.2 folder:

~/apache-solr-1.2.0$ ls
build.xml CHANGES.txt dist docs example KEYS.txt lib LICENSE.txt NOTICE.txt README.txt src

~/apache-solr-1.2.0$ ls dist
apache-solr-1.2.0.jar apache-solr-1.2.0.war

The easiest thing is to install Solr straight into an instance of Tomcat. One thing to be aware of is that search applications eat RAM and Heap for breakfast, so make sure you install it onto a server with plenty of RAM and it would be wise to increase the amount of Heap space available to the Tomcat instance. This can be done by making sure that the environment variable CATALINA_OPTS is set to "-Xmx512m" or even better "-Xmx1024m". This can be done inside the startup.sh script in your tomcat/bin directory is needed.

One final bit of advice before I point you at the rather good installation docs is that you might want to rename the .war file to match with the URL pathname you desire, as the guide relies on Tomcat automatically unpacking the archive:

So, a war called "apache-solr-1.2.0.war" will result in the final app being accessible at http://tomcat-hostname:8080/apache-solr-1.2.0/. I renamed mine to just solr.war.

Finally, Solr needs a place to keep its configuration files and its indexes. The indexes themselves have the capability to get huge (1Gb is not unheard of) and need somewhere to be stored. The documentation linked to below will refer to this location as 'your solr home' so it would be wise to make sure that this location has the space to expand. (NB this is not the directory inside Tomcat where the application was unbundled.)

Right, the installation instructions:

http://wiki.apache.org/solr/SolrInstall - Basic installation
http://wiki.apache.org/solr/SolrTomcat - Tomcat specific things to bear in mind

Now if you point your browser in the right place (http://tomcat_host:8080/solr/admin perhaps) you should see the admin page of the pre-built example application... which we do not want of course :)

Customising the Solr installation

The file solrconfig.xml is where you can configure the more mundane aspects of Solr - where it stores its indexes, etc. The documentation on what the options mean can be found here: http://wiki.apache.org/solr/SolrConfigXml but really, all you are initially going to change is the index save path, if you change that at all anyway.

The real file, the one that is pretty much crucial to what we are going to do, is the schema.xml. It has umpteen options, but you can get by, by just changing the example schema to hold the fields you want. Here is the wiki documentation on this file to help out and here (should) be a copy of the schema I am currently using

In fact, if you are using MODS or DC as your metadata, you may wish for the time being to just use the same schema as I am, just to progress.

Getting metadata into Solr

In my opinion, Solr has a really good API for remote management and updating the items ('documents' in solr lingo) in its indexes. Each 'document' has a unique id, and consists of one or more name-value pairs, like 'name=author, value=Ben'. To update or add a document to the Solr indexes, you HTTP POST a simple XML document of the following form to the update service, usually found at http://tomcathost:8080/solr/update.

The XML you would post looks like this:

<add>
<doc>
<field name="id">ora:1234</field>
<field name="title">The FuBar and me</field>
.... etc.
</doc>
</add>

Note that the unique identifier field is defined in the schema.xml near the end, by the element "uniqueKey", for example in my Solr instance, its <uniqueKey>id</uniqueKey>

The cunning thing about this is, is that to update a document in Solr, all I have to do is re-send the XML post above, having made any changes I wish. Solr will spot that it already has a document with id of 'ora:1234' and perform an update of the information in the index, rather than adding a second copy of this information.

One thing to note is that the service you are posting to is the update manager. No change is actually made to the indexes themselves until either the update manager is sent an XML package telling it to commit the changes, or the pre-defined (solrconfig.xml) maximum number of documents waiting to be indexed is hit.

Hooking into Solr from your library of choice

Handily, you are very unlikely to have to futz around writing code to connect to Solr, as there are a good number of libraries built for just such purposes - http://wiki.apache.org/solr/IntegratingSolr

As I tend to develop in python, I opted to start with the SolrPython script which I have made a little modification to and included in the libraries as libs/solr.py.

The BasicSolrFeeder class in solrfeeder.py and how it turns the Fedora Object into a Solr document

There are a number of different content types in the Oxford Research Archive, and ideally, there would be a type-specific method for indexing each one into the Solr service. As ORA is an emergent service, 99% of the items in the repository are text-based (journals, theses, book chapters, working/conference papers) and they all use either MODS (dsid: MODS) or simple Dublin Core (dsid: DC) to store their metadata.

I have written a python object called BasicSolrFeeder (inside the libs/solrfeeder.py file) which performs a certain script sequence of functions on a fedora object given its pid. (BasicSolrFeeder.add_pid in solrfeeder.py is the method I am describing.)

Using an array to hold all the resultant "<field name='foo'>bar</field>" strings

• Get the content type of the object (from objContentModel field in the FOXML)
• --> <field name="content_type">xxxxxxxx</field>
  
• Get the datastream with id 'MODS' and if it is XML, pass it through the xsl sheet at 'xsl/mods2solr.xsl'
• --> Lots of <field.... lines
• If there is no MODS datastream, then default to getting the DC, and pass that through 'xsl/dc2solr.xsl'
• --> Lots of <field.... lines
• As collections are defined in a bottom-up manner in the RELS-EXT, applying an xsl transformation to the RELS-EXT datastream, will yield the collections that this object is in.
  
• --> Zero or more < field name="collection">.... lines
• (NB there is some migrationary code here that will also deduce the content type, as certain collections are being used to group types of content.)
• Finally, if there is a datastream with id of FULLTEXT, this is loaded in as plain-text, and added to the 'text' field in Solr. This is how the searching the text of an object functions.
  
• (NB There are other services that extract the text from the binary attachments to an object, which collate these texts and adds them as a datastream called FULLTEXT.)
• This list of fields is then simply POSTed to the Solr update service, after which a commit may or may not be called.

So, if you have a fedora repository which uses either MODS or DC, and the collections are bottom-up, then the code should just work for you. You may need to tinker with the xsl stylesheets in the xsl/ directory to match what you want, but essentially it should work.

Here's an example script which will try to add a range of pids to a Solr instance:

from lib.solrfeeder import BasicSolrFeeder

sf = BasicSolrFeeder(fedora_url='http://ora.ouls.ox.ac.uk:8080/fedora',
                fedora_version="2.0",  # Supports either 2.0 or 2.2
                solr_base="/solr",     # or whatever it is for your solr app
                solr_url='orasupport.ouls.ox.ac.uk:8080')
                # Point at the tomcat instance for solr

# Now to scan through a range of pids, ora:1500 to ora:1550:

namespace = "ora"
start = "1500"
end = "1550"

# A simple hash to catch all the responses
responses = {}
for pid in xrange(start, end+1):
responses[pid] = sf.add_pid('%s:%s' % (namespace,pid), commit=False)

# Commit the batch afterwards to improve performance
sf.commit()

# Temporary variables to better report what went into Solr and
# what didn't
passed = []
failed = []

for key in responses:
if responses[key]:
   passed.append('%s:%s' % (namespace, key))
else:
   failed.append('%s:%s' % (namespace, key))
if passed:
print "Pids %s went in successfully." % passed
if failed:
print "Pids %s did not go in successfully." % failed

Adding security to Solr updates


Simple to add and update things in Solr, isn't it? A little too simple though, as by default anyone can do it. Let's add some authentication to the process. Solr does not concern itself with authenticating requests, and I think that is the right decision. The authentication should be either enforced by Tomcat, oor by some middleware.

The easiest mechanism is to use Tomcat's basic authentication scheme to password protect the solr/update url, to stop abuse by 3rd parties. It's pretty easy to do, and a quick google gives me this page - http://www.onjava.com/pub/a/onjava/2003/06/25/tomcat_tips.html - with 10 tips on running Tomcat. While most of the tips make for good reading, it is the 5th tip, about adding authentication to you Tomcat app, that is most interesting to us now.

Assuming that the password protection has been added, the script above needs a little change. The BasicSolrFeeder line needs to have two additional keywords, solr_user and solr_password and the rest of it should run as normal.

e.g.

sf = BasicSolrFeeder(fedora_url='http://ora.ouls.ox.ac.uk:8080/fedora',
                fedora_version="2.0",  # Supports either 2.0 or 2.2
                solr_base="/solr",     # or whatever it is for your solr app
                solr_username="your_username",
                solr_password="your_password",
                solr_url='orasupport.ouls.ox.ac.uk:8080')
                # Point at the tomcat instance for solr

Hopefully, this should be enough to get people to think about using Solr with Fedora, as Solr is a very, very powerful and easily embeddable search service. It is even possible to write a client in javascript to perform searches, as can be seen from the javascript search boxes in http://ora.ouls.ox.ac.uk/access/adv_search.php

I have purposefully left out how to format the queries from this post, but if people yell enough, I'll add some guidelines, more than I provide at http://ora.ouls.ox.ac.uk/access/search_help.php anyway

Conclusions on UUIDs and local ids in Fedora

I mentioned earlier about the possibility of using UUIDs as Fedora identifiers. I'll write my conclusion first and then the reasoning later for all you lazy people out there :)

Conclusions

Fedora repositories that wish to use UUIDs as identifiers should have the namespace 'uuid' added to the list of <retainPids> in fedora.fcfg.

The 32 character hex string representing the UUID is entered as the object id to the uuid namespace. For example:

Fedora pid - uuid:34b706b4-f080-4655-8695-641a0a8acb25

Benefits

1. Using the uuid scheme for identifiers, (those who retain the 'uuid' namespace), administrators will be able to painlessly transfer objects from one instance of Fedora to another, or even to have a distributed set of Fedora instances as a single 'repository'; No fiddling with pid changes, changing RELS-EXT datastreams or others, or changing metadata datastream identifiers.
   
2. Fedora pids will fit into the RFC 4122 mechanism for the uuid urn namespace easily -> urn:pid will be a valid URI.
3. The command 'getNextPID()' could be used to provide a local 'id', which can be added as a FOXML field, such as label or even added as the Alternate ID on ingest/migration.
   
4. Given a URI resolver that is updated when an object migrates from one fedora repository to another, a distributed set of Fedora instances could have cross-repository relationships in RDF that stay valid regardless of where the objects reside.
   
5. Use of a distributed set of Fedora instances with a federated search tool such as the Apache Solr is quite an attractive prospect for large scale implementations.

Reasoning

I've thought about the logistics of actually using them, and also of the fact that some people are happier with object ids that they can type (although for the life of me, I can't work out why; when was the last time that you, as a normal user, typed in a full URL and didn't just go to a discovery tool like google or a site search to get to a specific item? 99% of the time, I rely on my browser's address bar defaulting to google for anything that doesn't look like a url.) But I digress...

I prefer to deal with situations as they are, as opposed to what might be possible later, so let's recap what pids Fedora allows or needs:

Fedora pid = namespace : id

or more formally - (from http://www.fedora.info/definitions/identifiers/

object-pid    = namespace-id ":" object-id
namespace-id  = ( [A-Z] / [a-z] / [0-9] / "-" / "." ) 1+
object-id     = ( [A-Z] / [a-z] / [0-9] / "-" / "." / "~" / "_" / escaped-octet ) 1+
escaped-octet = "%" hex-digit hex-digit
hex-digit     = [0-9] / [A-F]

e.g. Anything that fits the following regular expression:

^([A-Za-z0-9]|-|\.)+:(([A-Za-z0-9])|-|\.|~|_|(%[0-9A-F]{2}))+$

As I said before, I am interested in using UUIDs (or something like them) because they need no real scheme to be unique and persistantly unique; UUIDs are designed so the chances of creating two ids that are the same is vanishingly small. So, what does one look like?

(from wikipedia:)

In its canonical form, a UUID consists of 32 hexadecimal digits, displayed in 5 groups separated by hyphens, in the form 8-4-4-4-12 for a total of 36 characters. For example:

550e8400-e29b-41d4-a716-446655440000

Regular expressions:

[A-Fa-f0-9]{8}-[A-Fa-f0-9]{4}-[A-Fa-f0-9]{4}-[A-Fa-f0-9]{4}-[A-Fa-f0-9]{12}
matches: 550e8400-e29b-41d4-a716-446655440000

^((?-i:0x)?[A-Fa-f0-9]{32}
matches: 0x550e8400e29b41d4a716446655440000

So a UUID can't be used as it is in place of a Fedora pid. However, according to RFC 4122, there is a uuid urn namespace which makes me more hopeful. The above uuid can be represented as urn:uuid:550e8400-e29b-41d4-a716-446655440000 for example.

So, how about if we make the reasonable assumption that a pid is a "valid" urn namespace, but a namespace that may or may not be registered yet? For example, I am currently using the fedora namespace ora for items in the Oxford repository. Would it be to far fetched to say that ora:1234 is understandable as urn:ora:1234?

So, all we need to do is make sure that the namespace 'uuid' is one of the ones in the <retainPid> element of fedora.fcfg and we are set to go. Looks like the pid format 'restriction' as I thought it, was quite handy after all :) So to state it clearly:

Fedora pids that follow the UUID scheme should be in the form of:

object-pid    = "uuid:" object-id
object-id     = 8-digit-hex '-' 4-digit-hex '-' 4-digit-hex '-' 4-digit-hex
                                                 '-' 4-digit-hex '-' 12-digit-hex
8-digit-hex   = ( hex-digit ) 8
4-digit-hex   = ( hex-digit ) 4
12-digit-hex  = ( hex-digit ) 12
hex-digit     = [0-9] / [A-F] / [a-f]

e.g. "uuid:34b706b4-f080-4655-8695-641a0a8acb25"

(NB forgive any syntactically slips above, I hope it's clear as it is.)

I mentioned before that some people want human-typeable fedora pids.... urgh. No really sure what purpose it serves. In fact, let me have a little rant...

<rant>

A 'Cool URL' is one that doesn't change. Short pids make for pretty URLs and guarantee little more than that.

</rant>

Right, that's out of my system. Now to accommodate the request...

Firstly, I'd just like to point out that I will ignore any external search and discovery services; essentially any type of resolver from search engine to the Handles system. This is because I feel that the format of the fedora pid is quite irrelevent to these services. (I am aware that certain systems made use of a nasty hack where the object id part of the fedora pid was used as the Handle id, after the institution namespace and believe me, this hack does have me quite worried. I can understand the reasoning behind this as the Handles system doesn't seem to have a simple way to query for the next available id, but I think this is a potential problem on the horizon.)

My suggestion is that the pid itself is the uuid as defined above, but that the repository system has a notion of local 'id'; the Fedora call of 'getNextPid()' could be used to create these 'tinypids' with whatever namespace is deemed pleasant.

They can be stored in the FOXML in fields such as Label or stored as an Alternate ID (fields which I personally have no use of). Fedora will index these for its basic search service, and could be used as a mechanism to look up the real pid given the local id.

For example, with the RDF triplestore turned on, the following iTQL query should be enough:

"select $object from <#ri>
      where $object <info:fedora/fedora-system:def/model#label> 'ora:1234'"

The tuple that is returned will be something like "uuid:34b706b4-f080-4655-8695-641a0a8acb25"

But it still isn't great, and I don't think the benefits outweigh the work involved in implementing it but it's a workable solution I think for those that need it.

Using python to play with a Fedora repository

Firstly, you'll need some extra libraries:

(If you are using Windows, I'm afraid you are on your own with problems. I can't help you, it's not a system I use.)

Get easy install from here: http://peak.telecommunity.com/DevCenter/EasyInstall
(If that site is slowed to a trickle, just install ez_install.py from somewhere else that is trustworthy)

Then, as root:

easy_install ZSI
easy_install uuid
easy_install 4Suite-xml
easy_install pyxml

(There may be more, I don't have a system set aside to try it out.)

Then create a clean working directory and grab the libraries from here:

svn co https://orasupport.ouls.ox.ac.uk/archive/archive/lib

These are of questionable quality, and are in a state of transistion from proof of concept jumbled structure into a more refined and refactored set of libraries. The main failing is that I have a mix of convenience methods which might be pretty specific in use, alongside more fundamental methods which are much more generic.

(PS, if you want to try the full archive interface out, you'll need to inject some objects into the repository to start with, specifically the resource objects that have the xsl for the view transforms. If anyone wants, I'll wrap these with a bow and add them when I have time.)

But, for now, they will at least provide the fundamentals to play with.

I will assume you have a Fedora repository set up somewhere, and that you know a working username and password that will let you create/edit/etc objects inside it. It also assumes the instance has an API like Fedora 2.2 especially for SOAP. I'll post up about making the FedoraClient multi-versioned with regards to SOAP later.

For the purposes of the rest of this post, fedoraAdmin is both the username and password for the repository, and that it lives at localhost:8080/fedora.

Inside the same directory that holds the lib/ directory, start the python commandline:

~/temp$ python
Python 2.5.1c1 (release25-maint, Apr 12 2007, 21:00:25)
[GCC 4.1.2 (Ubuntu 4.1.2-0ubuntu4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>

Now let's get a fedora client and poke around the repository

>>> from lib.fedoraClient import FedoraClient
(cue SOAP related chugging of CPU when loading the SOAP libs)
>>> help(FedoraClient) # This will show you all sorts about this class
>>> # But we are interested in the following:
>>> f = FedoraClient(serverurl='http://localhost:8080/fedora', username='fedoraAdmin', password='fedoraAdmin', version='2.2')

Now we have the client, let's try out a few things:

>>> print f.getDescriptionXML()
(XML related stuff in reply)

>>> f.doesObjectExist('namespace:pid')
True or False depending

>>> # For example, in my dev repo:
>>> f.getContentModel('person:1')
u'person'

>>> f.listDatastreams('ora:20')
[{'mimetype': u'image/png', 'checksumtype': u'DISABLED', 'controlgroup': 'M', 'checksum': u'none', 'createdate': u'2007-09-25T14:36:29.381Z', 'pid': 'ora:20', 'versionid': u'IMAGE.0', 'label': u'Downloadable stuff', 'formaturi': None, 'state': u'A', 'location': None, 'versionable': True, 'winname': u'ora_20-IMAGE.png', 'dsid': u'IMAGE', 'size': 0}, {'mimetype': u'text/xml', 'checksumtype': u'DISABLED', 'controlgroup': 'X', 'checksum': u'none', 'createdate': u'2007-09-25T14:37:02.882Z', 'pid': 'ora:20', 'versionid': u'DC.2', 'label': u'Dublin Core Metadata', 'formaturi': None, 'state': u'A', 'location': None, 'versionable': True, 'winname': u'ora_20-DC.xml', 'dsid': u'DC', 'size': 272}, {'mimetype': u'text/calendar', 'checksumtype': u'DISABLED', 'controlgroup': 'M', 'checksum': u'none', 'createdate': u'2007-09-25T14:37:03.391Z', 'pid': 'ora:20', 'versionid': u'EVENT.3', 'label': u'Events', 'formaturi': None, 'state': u'A', 'location': None, 'versionable': True, 'winname': u'ora_20-EVENT.ics', 'dsid': u'EVENT', 'size': 0}, {'mimetype': u'text/xml', 'checksumtype': u'DISABLED', 'controlgroup': 'X', 'checksum': u'none', 'createdate': u'2007-08-31T14:21:39.743Z', 'pid': 'ora:20', 'versionid': u'MODS.4', 'label': u'MODS Record', 'formaturi': None, 'state': u'A', 'location': None, 'versionable': True, 'winname': u'ora_20-MODS.xml', 'dsid': u'MODS', 'size': 1730}]

>>> f.doesDatastreamExist('ora:20','DC')
True
>>> f.doesDatastreamExist('ora:20','IMAGE')
True
>>> f.doesDatastreamExist('ora:20','IMAGE00123')
False

Creating new items:

The steps are as simple as creating a new blank FoXML object, and ingesting it, datastreams are uploaded and added afterwards. The first example will be trivial and the second will be more detailed.

First Demo:
http://pastebin.com/f7b1f21e7

Second Demo:
Look at the 'createBlankItem' method in FedoraClient. Plenty of scope for creating complex objects on the fly there.

Poking around the Triplestore:

Using the above libs:

>>> from lib.risearch import Risearch
>>> r = Risearch(server='http://localhost:8080/fedora') # This is the default, and equivalent to Risearch()

Then you can ask it fun things:

>>> # Retrieve a list of all the objects in the repository:
>>> pids = r.getTuples("select $object from <#ri> where $object <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <info:fedora/fedora-system:def/model#FedoraObject>", format='csv', limit='10000').split("\n")[1:-1]

>>> # Get a list of the pids in a given bottom up collection (ora:neeo):
>>> pids = r.getTuples("select $object from <#ri> where $object <fedora-rels-ext:isMemberOf> <info:fedora/ora:neeo>", format='csv', limit='10000').split("\n")[1:-1]

>>> # Test to see if a certain relationship exists:
>>> # May need to change the code in risearch.py to use the old method
>>> r.doesTripleExist('<info:fedora/ora:1> <person:hasStatus> <info:fedora/collection:open>')
False

Next post, I'll write up about how Solr can be fed from objects in a Fedora repository.

Content Types in the Oxford Research Archive

I started working on the Oxford research archive last year, and very early on, it seemed inevitable that a broad classification would be useful to distiguish between the types of things that were being submitted. I don't mean distiguish between authors or file types or even between items containing images and those that didn't.

What I mean is that a broad and somewhat rough characterisation of the item's nature is needed, something that can be found when the following type of questions are asked:

• What is the item's canonical metadata? (MODS, DC, Qual. DC, MIX, FoaF etc?)
  
• Is its metadata likely to comprise a certain set or profile of information, such as the sets of information you might get from a journal article, a book, an image, a video or a thesis? For example, while a book is likely to have 'author', 'publisher' and so on in its set of metadata, an image is likely to have 'photographer', 'camera' and 'exposure' as well.
  
• Is the item a metadata only item, or does it have binary attachments? (Is it a "by reference" item or "by value"?)
  
• Which attachments should be listed for download (PDFs, etc) and which should be shown inline (images, thumbnails, video) in the page?
• Is the item part of the repository to give it structure, a item that corresponds to a collection, an author, or a department rather than to the data and attachments from a actual submission?
  

So, what broad types has Oxford Research Archive got in the production server? Copying and pasting from this search: (07 Dec results taken)

(From near the bottom of the response:)
<int name="eprint">398</int> Content type name: eprint
<int name="basic">345</int> Content type name: basic
<int name="ethesi">54</int> Content type name: thesis
<int name="thesi">54</int> Content type name: thesis
<int name="general">52</int> Content type name: general
<int name="collect">6</int> Content type name: collection
<int name="confer">5</int> Content type name: conference and conferenceitem

NB The results are 'stemmed' (runs and running have the stem run, for example) but I have written the real names alongside them. Also, due to Fedora 3.0 expressing content types (content models to them) using RDF, rather than a metadata field in the object's FOXML, the above results are a combination of the old style of content model, combined with the new. In the new, I have dropped the e from ethesis as it was purposeless.

I should mention that there are universal datastreams, datastreams that are common to all of these types.

DC is the simple Dublin Core for an item and is equivalent to the only mandated(?) format for OAI-PMH, 'oai_dc'. This is also required by Fedora, at least up to version 2.2.1. If it is removed, or not included in an ingest, Fedora will make one.

There is also the optional FULLTEXT datastream which holds whatever textual content that can be harvested from the binary files, using applications such as antiword, pdftotxt, and others.

There is the EVENTS datastream, holding an iCal formatted simple log of actions taken on the item and actions that should take place, in future, on that item as well. This is not intended to replace PREMIS or a similar format. It simply allows for a much more pragmatic approach to help deal with the technicalities of performing, scheduling and logging events, rather than being the canonical data for the provenance of an item.

eprint - the bread and butter content type. I'll go through this one in depth:

Datastreams:

This includes a variety of actual content, but mainly the type of thing that you would see in a typical IR, journal articles and short article type reports. A single canonical metadata (we've chosen MODS) with two derived metadata, simple DC and MARCXML, both provided using XSLT. The item also has zero or more attached files, with the prefix of ATTACHMENT or JOURNAL (legacy) to the datastream ids and an number after the prefix, with the listing order desired to be 01, 02 , etc. There are likely to also be no inline images necessary, as producing thumbnails of the frontpages would be an utter waste - white box with black smudge or smudges in the middle. Most of these have a cover page as well, so the thumbnail would be even more useless.

Metadata:

Essential the metadata will be that provided by the MODS record. Optionally, the better catalogued items will have a mods:relatedItem element, with a type="host" and this will define the host journal and where the article came from. This information is simply filled in on a web-form normally, and the XML nature of it is hidden.

Presentation:

• Test to see if the item has MODS and get that. Transform it using the eprint specific xslt stylesheet.
• If not, get the DC instead and use the basic dc2html.xsl
• Get the list of acceptable datastreams and present links to these as downloads.
• Look for interesting RDF connections and present those too.

basic - a simple DC based atomic item. This will often be the default type of any item harvested from a EPrints.org or similar repository.

Datastreams: Ronseal item, a single 'thing' is archived with metadata no more complicated than simple DC can handle. The 'thing' can be in multiple formats, especially if it is initially submitted or harvested in a proprietary format such as MS Word.

Metadata: Simple DC

Presentation:

• Get DC and pass it through the dc2html.xsl
• Get the list of acceptable datastreams and present links to these as downloads.
• Look for interesting RDF connections and present those too.

thesis - rich metadata stored in MODS, with a good number of binary attachments (typically one PDF per chapter, but alongside whatever original files were uploaded.) Very similar to eprint type, but with some important differences: one author is expected, with zero or more supervisors. The role is indicated as per usual in MODS, /mods:mods/mods:name/mods:role/mods:roleTerm = 'author' for example. Also, etd metadata should be present in the mods:extension section.

Datastreams and Metadata: As for eprint, but with small differences as noted above.

Presentation: Currently, as for eprint, but with a slightly different template and vocab. This will be developed given time to make more of the fact that it will have a single author, zero or more supervisors, etc.

general - ephemera collection, and it's not exactly a content type as it holds everything else. Groups of items will get 'promoted' from this collection, once we have identified that there is enough of one type of item to warrant time and effort spent doing so. The one thing that unites these items is that MODS metadata format is capable of holding a good description of what the items are.

Datastreams, Metadata, and Presentation: As for eprint

collection - a metadata only, structural item. This provides an item which serves to enable collections and so forth. It provides a node in the triplestore, which other items can then relate to. Holds a very basic dublin core record, which

conference and conferenceitem - Conference item types. The conference type itself is intimately related to the collection type; it exists to provide a URI to hang some information upon. How it differs is in the metadata it has to describe itself. A conference item has MODS metadata indicating the location, the dates, editors and other associated information that a conference implies. The item can represent either a single instance of a conference (Open Repositories 08) or a series (Open Repositories).

A picture tells a thousand words; this is a link to an image that should help show how these conference item types are used: http://www.flickr.com/photos/oxfordrepo/2102829887/

(Errata: image should have been updated to show 'isMemberOfCollection' in preference to just 'isMemberOf')

Linking items together by using an RDF store

(Disclaimer: This is going to rapidly skip over functionality that is present in Fedora, and can be duplicated for other object stores given effort. The key pieces of functionality are that each item has a set of RDF triples that describe how it relates to other items, and that each item can be referenced by a URI.)

(If you wish to follow along without a pre-made environment, see the bottom of this post, where I'll stick some python code that'll create a rdf triplestore that you can tinker with - Not as a web service though, command line only at this point.)

Assumption #1 An item is the tangible object in your store. It is likely to hold both metadata and the data itself as part of it. So it could be a METS file or a Fedora Object easily.
Assumption #2 Items and their various important parts are identifiable using URIs, or at the very least, a hackable scheme.

So, considering Fedora, it's objects have URIs of the form , where the namespace and id are the determining part. A part of this object (a datastream, such as an attached pdf or even the Dublin Core of the object itself) can be written in URI form by using the datastream's id - e.g. will resolve to the Dublin Core xml that is attached to the object 'ora:1600'.

Skipping very lightly over questions of semantics and other things, the basic premise of using RDF in the way I shall show is to define relationships between nodes expressed as URIs or as text - more formally as 'Literals'. These relationships, or 'properties' are drawn from a host of namespaces where the properties themselves are defined.

Let's see a few hopefully familiar properties:

From http://purl.org/dc/elements/1.1/ (You may need to view the source of that page): {NB the following has been heavily cut, as indicated by [ snip ]}

[ snip ]
<rdf:Property about="http://purl.org/dc/elements/1.1/title">
<rdfs:label lang="en-US">Title</rdfs:label>
<rdfs:comment lang="en-US">A name given to the resource.</rdfs:comment>
<dc:description lang="en-US">Typically, a Title will be a name by which
the resource is formally known.</dc:description>
<rdfs:isdefinedby resource="http://purl.org/dc/elements/1.1/">
<dcterms:issued>1999-07-02</dcterms:issued>
<dcterms:modified>2006-12-04</dcterms:modified>
<dc:type resource="http://dublincore.org/usage/documents/principles/#element">
<dcterms:hasversion resource="http://dublincore.org/usage/terms/history/#title-005">
</dcterms:hasversion>
</rdf:Property>
<rdf:Property about="http://purl.org/dc/elements/1.1/creator">
<rdfs:label lang="en-US">Creator</rdfs:label>
<rdfs:comment lang="en-US">An entity primarily responsible for making
the resource.</rdfs:comment>
<dc:description lang="en-US">Examples of a Creator include a person, an
organization, or a service. Typically, the name
of a Creator should be used to indicate the entity.</dc:description>
<rdfs:isdefinedby resource="http://purl.org/dc/elements/1.1/">
[ snip ]

Familiar? Perhaps not, but now we can move onto what makes the RDF world go around, 'triples'. At a basic level, all that you will need to know is that a triple is a chunk of information that states that 'A → B → C' where A is a URI, C can be a URI or a literal, and the property is B which links A and C together.

Real example now as it is quite straightforward really. You will be able to understand the following triple for example!

<info:fedora/ora:909> <http://purl.org/dc/elements/1.1/creator> "Micheal Heaney"

The object 'ora:909' has a dc:creator of 'Micheal Heaney'. Simple enough, eh? Now, let's look at a more useful namespace, and for my ease, let's choose one used by Fedora: http://www.fedora.info/definitions/1/0/fedora-relsext-ontology.rdfs

You don't really need to read the definition file, as the meaning should be fairly obvious:

<info:fedora/ora:1600> <info:fedora/fedora-system:def/relations-external#isMemberOf> <info:fedora/ora:younglives> .

<info:fedora/ora:1600> <info:fedora/fedora-system:def/relations-external#isMemberOf> <info:fedora/ora:general> .

To make things a little more clear, I have put the important part of the triple in bold. (If you are really, really curious to see all the triples on this item, go to http://tinyurl.com/33z3rn )

This, as you may have guessed is how I am implementing collections of objects. ora:general is the object corresponding to the set of general items in the repository, and ora:younglives is the collection for the 'Young Lives' project.

As this snapshot should show, this is not the extent to how I am using this. (Picture of the submission form for the archive - http://www.flickr.com/photos/oxfordrepo/2102829891/)

Next Steps:

The mechanics of how the triples get into the rdf triplestore (think of it as a db for triples) is entirely up to you. What I will say is that if you use Fedora, and put the RDF for an object in a datastream called 'RELS-EXT', this RDF is automatically added to an internal triplestore. For an example of this, see http://ora.ouls.ox.ac.uk:8080/fedora/get/ora:1600/RELS-EXT

So, I did warn that I was going to skip over the mechanics of getting triples into a triplestore, and I will now talk about the 'why' we are doing this; querying the triplestore.

Querying the store

The garden variety query is of the following form:

"Give me the nodes that have some property linking it to a particular node" - i.e. return all the objects in a given collection, find me all the objects that are part of this other object, etc.

Let's consider finding all the things in the Young Lives collection mentioned before

In iTQL:

"select $objects from <#ri> where $objects <fedora-rels-ext:ismemberof> <info:fedora/ora:younglives>" - http://tinyurl.com/3a642k

or equally, in SPARQL for a different triplestore:
http://purl.org/dc/elements/1.1/
"select ?objects where { ?objects <fedora-rels-ext:ismemberof> <info:fedora/ora:younglives> . }"

I think it should be straightforward to see that this is a very useful feature to have.

Python Triplestore backed by MySQL:

Install rdflib - either by using easy_install, or from this site: http://rdflib.net/
Install the MySQL libraries for python (debian pkg - python-mysqldb)

Then, to create a triplestore and add things to it:

#!/usr/bin/python
# -*- coding: utf-8 -*-

import rdflib
from rdflib.Graph import ConjunctiveGraph as Graph
from rdflib import plugin
from rdflib.store import Store
from rdflib import Namespace
from rdflib import Literal
from rdflib import URIRef

from rdflib.sparql.bison import Parse

graph = Graph()

# Now, say you have some RDF you want to add to this. If you wanted to add a file of it:
# May I suggest http://ora.ouls.ox.ac.uk:8080/fedora/get/ora:1600/RELS-EXT as this will make the
# query work :)
graph.parse(path_or_url_to_rdf_file)

# Adding a single triple needs a little prep, (namespace definitions, etc)
dc = Namespace("http://purl.org/dc/elements/1.1/")
fedora = Namespace("info:fedora/") # Or whatever

# the format is -> graph.add( (subject,predicate,object) )
graph.add( (fedora['ora:1600'], dc['title'], Literal('Childhood poverty, basic services and cumulative disadvantage: an international comparative analysis') ) )

# And to look at all of your RDF, you can serialize the graph:
# in RDF/XML
print graph.serialize()

#in ntriples
print graph.serialize(format='nt')

# Now to query it:
parsed_query = Parse("select ?objects where { ?objects <fedora-rels-ext:isMemberOf> <info:fedora/ora:younglives> . }"

response = graph.query(parsed_query).serialize() # in XML

# or directly as python objects:
response = graph.query(parsed_query).serialize('python')

To make it persistant, you need a store. This is how you add MySQL as a store:

# replace the graph = Graph() line with the following:
# Assumes there is a MySQL instance on localhost, with an empty, writable db 'rdfstore' owned by user 'rdflib' with password 'asdfasdf'

default_graph_uri = "http://host/rdfstore"

# Get the mysql plugin. You may have to install the python mysql libraries
store = plugin.get('MySQL', Store)('rdfstore')

# Create a new MySQL backed triplestore. Remove the 'create=True' part to reopen the store
# at a later date.
resp = store.open("host=localhost,password=asdfasdf,user=rdflib,db=rdfstore", create=True)
graph = Graph(store, identifier = URIRef(default_graph_uri))