OSCELOT Open Source Day III - views

The event was held at the Nevada gaming institute, and was overall, a well-structured day. The driving ideology was that of the unconference - "... a facilitated, face-to-face, and participant-driven conference centered around a theme or purpose."

However, it seemed that the theme or purpose of the event was not about Open Source - it was as if it were a Blackboard self-help group, trying to solve the issues and failings of this proprietary software. Some of the issues were a little shocking - someone proposed that they had "a need to search the content of [Blackboard Vista] repository" - it came as some surprise to me that this wasn't already possible in such a mature product.

I was pleased that we were able to help and inform the other attendees about more open technology and standards, such as OAuth, resource-orientated architecture, creative commons licensing and more.

One session I lead on was titled - controversially - "Why [bother with] Portals?" - in which I wanted to get a discussion on what students actually use. The point I wanted to make was that URLs are the base currency of the internet - search engines produce lists of them, people bookmark them, and URLs are used when sharing information between people.

This means that there is a very large responsibility on the content providers not to change URLs, or they will devalue the very resources they are trying to get people to use. This is the reason why persistent URLs are a crucial thing to aim for.

I hope that we were able to bring extra value to the meeting, due to the fact that, unlike the vast majority of attendees, we do not have a Blackboard background.

However, I do think that the event needed to have more emphasis on real-world open source projects such as Sakai and Moodle, and examine how best to intergrate their systems with external systems.

Open replacements for Twitter and more importantly, Tinyurl

I hope that you all already know about http://identi.ca and the software stack laconi.ca that it runs - in short, it's a Twitter-like micro-blogging service, that is geared to be open. It provides the possibility of a distributed micro-blogging set of services that can talk to each other. Pretty cool.

But the less well known release, was that of the lilurl service, a Tinyurl replacement, again, geared to be very open. For example, the database of links the service holds can be downloaded by any user! BUT it lacks an API to create these links on the fly...

I think you've already guessed the end of that statement, I've made an API for it as the base code for the service is open source. Hearty thanks to Evan Prodromou!

So, changes from the source (which is at: http://ur1.ca/ur1-source.tar.gz)

Firstly, change the .htaccess rewrite rules:

From:

• RewriteRule (.*) index.php

To:

• RewriteRule s/(.*) index.php

This requires a few cosmetic changes to the index.php to serve correct lil'urls:

Line 41 in index.php:
From:

• $url = 'http://'.$_SERVER['SERVER_NAME'].'/'.$lilurl->get_id($longurl);

To:

• $url = 'http://'.$_SERVER['SERVER_NAME'].'/s/'.$lilurl->get_id($longurl);

And then, in the root directory for the app, add in api.php, which is currently pastebinned:

http://pastebin.com/f29465399 - api.php

How it works - Creating lilurls:

POST to /api.php with parameters of longurl=desired url

This will be the response:

HTTP/1.1 201 Created
Date: Tue, 08 Jul 2008 16:51:13 GMT
Server: Apache/2.0.52 (Red Hat)
X-Powered-By: PHP/5.1.4
Content-Length: 36
Connection: close
Content-Type: text/html; charset=utf-8

http://somehost.com/s/1

The message body of the response will contain the URL. Similarly, you can lookup a given lilurl, by GET /api.php?id=lilurl_id

GET /api.php?id=1 HTTP/1.1
Host: somehost.com
content-type: text/plain
accept-encoding: compress, gzip
user-agent: Basic Agent

HTTP/1.1 200 OK
Date: Tue, 08 Jul 2008 16:55:00 GMT
Server: Apache/2.0.52 (Red Hat)
X-Powered-By: PHP/5.1.4
Content-Length: 24
Connection: close
Content-Type: text/html; charset=utf-8

http://ora.ouls.ox.ac.uk

So... er, yeah. Job done ;)

Archiving Webpages with ORE

(Idea presented is from the school of "write it down, and then see how silly/workable it is")


Following on from the example by pkeane on the OAI-ORE mailing list, about constructing an Atom 'feed', listing the resources linked to by a webpage. Well, it was more a post wondering what ORE provides that we didn't have before, which for me is the idea of an abstract model with multiple possible serialisations. But anyway, I digress.

(pkeane++ for an actual code example too!)

For me, this could be the start of a very good, incremental method for archiving static/semi-static (wiki) pages.

Archiving:

1. Create 'feed' of page (either as an Atom feed or an RDF serialisation)
• It should be clearly asserted in the feed which one of the resources are the (X)HTML resource that is the page being archived.
  
2. Walk through the resources, and work out which ones are suitable for archiving
• Ignore adverts, video perhaps and maybe also some remote resources (but decisions here based on policy and the process is an incremental one.Step 2 can be revisited with new policy decisions, such as remote PDF harvesting and so on.)
3. For each resource selected for archiving,
   
1. Copy it by value to a new, archived location.
2. Add this new resource to the feed.
   
3. Indicate that this new resource is a direct copy of the original in the feed as well (using the new rdf-in-atom syntax, or just plain rdf in the graph.)

Presentation: (Caching-reliant)

1. A user queries the service to give a representation for an archived page.
2. Service recovers ORE map for requested page from internal store
   
3. Resource determination Again, policy based - suggestions:
   
• Last-Known-Good: Replaces all URIs in (X)HTML source
  with their archived duplicates, and sends the page to the user.
  (Assumes dupes are RESTful - archived URIs can be GET'ed)
• Optimistic: Wraps embedded resources with javascript, to attempt to get the original resources or timeout to get the archived versions.

Presentation: (CPU-reliant)

1. Service processes ORE map on completion of resource archiving
   
2. Resource determination Again, policy based - suggestions:
   
• Last-Known-Good: Replaces all URIs in (X)HTML source
  with their archived duplicates, and sends the page to the user.
  (Assumes dupes are RESTful - archived URIs can be GET'ed)
• Optimistic: Wraps embedded resources with javascript, to attempt to get the original resources or timeout to get the archived versions.
3. Service stores a new version of the (X)HTML with the URI changes, adds this to the feed, and indicates that this is the archived version.
4. A user queries the service to give a representation for an archived page and gets it back simply.

So, one presentation method relies on caching, but doesn't need a lot of CPU power to get up and running. The archived pages are also quick to update, and this route may even be a nice way to 'discover' pages for archiving - i.e. the method for archive url submission is the same as the request. Archiving can continue in the background, while the users get a progressively more archived view of the resource.

The upshot of having a dynamic URI swapping on page request is that there can be multiple copies in potentially mobile locations for each resource, and the service can 'round-robin' or pick the best copies to serve as replacement URIs. This is obviously a lot more difficult to implement with static 'archived' (X)HTML, and would involve URI lookup tables embedded into the DNS or resource server.

Internet Explorer can go to hell

Download a PDF file from the archive through a browser


[Just to get the headers here, I used: "curl -I http://ora.ouls.ox.ac.uk/objects/uuid%3A4af22069-ec0e-4407-b42d-2926c5a6c9ca/datastreams/ATTACHMENT01"]

Server Response:
HTTP/1.1 200 OK
content-length: 53760
content-disposition: attachment; filename="uuid4af22069-ec0e-4407-b42d-2926c5a6c9ca-ATTACHMENT01.doc"
accept-ranges: bytes
last-modified: Wed, 11 Jun 2008 13:54:45 GMT
content-range: 0-53759/53760
etag: 1213192485.0-53760
content-type: application/msword
x-pingback: http://ora.ouls.ox.ac.uk/pingback
Date: Wed, 11 Jun 2008 13:54:45 GMT
Server: CherryPy/3.0.2

• Firefox (any version, any OS) downloads this and passes it to an external app fine, job done.
• Safari, same deal.
• Camino, ditto.
• Opera, not a problem.
• Even Internet Explorer 7, downloads it and opens it fine.

But:

Internet Explorer 6 -> Adobe Acrobat says "Error, file not found"

Internet Explorer 5 -> Adobe still says "Error, file not found"

I'll post up the fix to IE5/6 not being able to download the file properly. It's all in the response headers, and I'll let you play spot the difference:

Server Response (works with IE5/6):
HTTP/1.1 200 OK
content-length: 53760
content-disposition: attachment; filename="uuid4af22069-ec0e-4407-b42d-2926c5a6c9ca-ATTACHMENT01.doc"
accept-ranges: bytes
last-modified: Wed, 11 Jun 2008 13:54:45 GMT
content-range: 0-53759/53760
etag: 1213192485.0-53760
pragma:
content-cache:
content-type: application/msword
x-pingback: http://ora.ouls.ox.ac.uk/pingback
Date: Wed, 11 Jun 2008 13:54:45 GMT
Server: CherryPy/3.0.2

Found it? No?

etag: 1213192485.0-53760
pragma:
content-cache:
content-type: application/msword

That's right, adding empty fields into the response headers magically fixes the download issue for IE6 and 5. One day, I'll find an IE engineer and if I do, when I do... well... my bail money will be with my lawyer. in that case.

Google App Engine SDK - How to work out if you are running deployed or locally

Short post this one:

os.environ['SERVER_SOFTWARE'] is your variable to see.

• Deployed, it reads 'Google Apphosting/1.0'
• Running locally, it reads 'Development/1.0'

So doing "if os.environ['SERVER_SOFTWARE'].startswith('Development'):" should be enough to deal with the differences between local and deployed.

Release 0.2 of the Python REST Client

The update is due to improvements made to the Google App Engine flavour of the restful lib code - I've added automatic Basic and Digest authentication to the code, so that apps deployed to the App engine can now use services that require either form of authentication, such as Twitter (Basic auth needed) or the Talis Platform (Digest auth only).


http://code.google.com/p/python-rest-client/

OAI-ORE reaches beta

Beta release of specs - http://www.openarchives.org/ore/0.9/

At a cursory glance through the documents, there seems to be a few refinements on version 0.3:

1. Resource map metadata is now (IMO) handled better
2. The inclusion of resource typing into the aggregation data model
   - Yeah, wonder where they got that idea ;) But seriously, it suggests to type the objects at a conceptual level. I would have liked to see inclusion of more physical typing alongside this, such as metadata standard adherence (dcterms:conformsTo) and a 'mimetype' of some sort (dc:format but this is open to debate).
   
3. Proxies are more fleshed out (I still think that this is a solution looking for a problem though)

So, aside from the resource map metadata changes, I don't think this will cause me undue work to update to. The alpha to beta spec transition was just refinement and not an overhaul.