JISCMarkr: Prototype RSS reader for teachers

Title of Primary Project Output: JISCMarkr: Prototype RSS reader for teachers

Screenshots or diagram of prototype:

Description of Prototype: Markr is a tool to help educators at all levels manage, track, evaluate, analyze, visualize and assess information from student blogs. This prototype provides an RSS reader front end and word cloud interface to student blog content pulled live from the web via the gdata API.

Link to working prototype: http://www.cems.uwe.ac.uk/markr/

Link to end user documentation: http://www.cems.uwe.ac.uk/markr/

Link to code repository or API: http://code.google.com/p/markr/source/browse/

Link to technical documentation: http://code.google.com/p/markr/source/browse/

Date prototype was launched: 30/11/09

Project Team Names, Emails and Organisations: Dan Dixon, dan.dixon@uwe.ac.uk, University of the West of England; Prakash Chatterjee, prakash.chatterjee@uwe.ac.uk, University of the West of England

Project Website: http://markr.digitaldust.org/

PIMS entry: No project entry found.

Table of Content for Project Posts:

Figure and Ground

Negative space

Technical standards and stuff we can’t live without

User participation, undiscovered use cases and spanners

The chart that dare not speak its name

Wins and Fails with SysAdmins

Reflections on attempting an Agile project in academia

Project Evaluation and Future Planning

Project Evaluation and Future Planning

As it draws to the end of the project it is time to start thinking about what and where we are going next. Even if this project doesn’t get funded it is worth taking forward and we’ll be spending time on it over the current year continuing development and seeing where we get to.

As part of that it really is worth reflecting on what we’ve achieved and looking for opportunities, because we have surfaced a few interesting things as part of this project.

SWOT analysis

As we are not trying to create a commercial application, most of the possible threats seem to end up being opportunities. If we’ve built the wrong thing, or it can’t compete then we know that and either change the direction of the tool or do something different. At this stage there isn’t a massive investment in this, it is merely some initial prototyping. And all the other threats and weaknesses become challenges rather than problems.

One of the things we set out to do was prove that we could use blogs as a distributed CMS or data store for student content. And especially using Blogger and the Google GDataAPI we found that we could do it reliably and quickly. This is a useful finding because it justifies some of the decisions we made in architecting this the way we did. There might be some downsides to doing it this way, in that not all the content is stored centrally, but this was one of the aspects we were investigating.

One fascinating idea that has come up recently is to see if we could recreate this app in a more lightweight manner. Could we do it using another reader/aggregator, merge feeds using Yahoo Pipes and hook those feeds into google chart or Many Eyes, maybe use Greasemonkey scripts to add metadata or bring in google charts. It would still be worth looking into the feasibility of this however google chart is limited to some simple graphs. Many Eyes only takes uploads of public data sets, so has no API. Greasemonkey scripts are brittle and not something to plan an app around. Google Reader has no API, doesn’t generate feeds and doesn’t look like something that one could hook into. And yahoo pipes doesn’t do the maths that we need to generate stats. Pity, it would be nice to just generate pretty graphs on the fly, out there in the cloud.

Also looked into Daytum and whether that could do what we want, but unfortunately it doesn’t work the way we need. It collects individual pieces of data. Though with a pro account one could pipe everything through into Twitter and have Daytum pick those up. Which is worth trying to see if that will work, but although it is very pretty, the visualizations are targeted at a particular use, a very personal and OCD use.

Feedback from the interface so far shows that it is understandable, that we’ve used common interface patterns, and seems easy to use. Because we use similar layout and interface to common tools it is learnable. There are certainly some tweaks that will need to happen before use.

Reflections on attempting an Agile project in academia

I have a lot of experience of running Agile projects in commercial situations, so I thought I could easily do this in a slightly different setting. Agile methodologies have been used in many different situations. Agile processes get used for lots of different project types, from event organization, to marketing and even to build software. They’ve been scaled beyond the small teams where it works very well (though I’m yet to be convinced that this isn’t because of very talented individuals rather than the innate nature of the process). Why wouldn’t it work in an academic setting?

One key reason, and that is scheduling. In a standard academic setting the team members are never together. Classes are scheduled, other projects get in the way, holidays happen and other deadlines loom. This means even our little team of two only ever had one chance to spend any length of time together over the project.

Individuals and Interactions

I’m going back to the Agile Manifesto to understand why this didn’t work as well as intended. And the biggest reason it hasn’t worked so well is the first item on the Agile Manifesto, “Individuals and Interactions over process and tools.” Which, in my opinion, is the most important item on the list, and deserves to go first.

The first tool to deal with this issue is co-location of team members and I’ve always been amazed at how much of a difference this makes. Two metres distance rather than twenty has a noticeable impact on speed and collaborative potential.

Any future project I would try to actively plan in time for collaborative work. To the extent of taking it up with the timetable schedulers to make space for this. This needs decent lead time to organise, and because of this I can see why there aren’t any/many short term projects in academia.

Working Software

Most academic projects are slow burn, a day a week, or even a few hours here and there. So unless there is some form of dedicated technical resource on the project nothing gets built quickly. When it might normally take a couple of weeks to get a prototype application together it instead might take three months or more. This has the knock on effect that other team members might have to wait for that whole length of time before carrying out activities they need to do, and any marginal delay has an amplified effect.

Again the working software tenet can only be used when there is full time resource. Otherwise the more classic software engineering approach is better because the project can effectively be split into sections and dealt with with less dependencies.

Documentation is a safety net, but sometimes you need a safety net.

Our process

So in trying to think about and  implement something like Scrum we came up with the following changes.

• Two weekly scrums, not daily
• Less rigid sprints with Andon based breakpoints to replan
• Abandoned burn down

As we are only spending on average one to two days a week, we only need a weekly or bi-weekly scrum catch up. The time of the entire project would be one to two sprints in Scrum, so no need to have many repeated meetings to replan and we abandoned this. However we did need to heavily replan once and do some minor course correction. Something like the Andon card/display system from Lean Manufacturing/Management is worth implementing, and this ties in nicely to the “Respond to change over following a plan” aspect of the Agile Manifesto.

Wins and Fails with SysAdmins

Here’s one from the notebook. It’s both a win and a fail as it has gone from epic fail status to a tired win.

So we got the code live on the 6th, but had problems with Markr working in the live environment. As we are trying to get RSS and do HTTP requests out through the proxy we hit problems. HTTP out from inside was an issue. It took a good week to find this and it finally got solved, but there are obviously many problems internally trying to support these kinds of applications.

Ultimately the sysadmins are there to support systems being used by students, not researchers and the servers are set up to deal with security issues around that, not support apps and researchers developing apps. So this makes it difficult to deploy apps quickly and easily. In the future I would look much more closely at hosting apps externally.

Code live :)

We’ve got some code up on the live server after typical sys admin wrangling.

http://www.cems.uwe.ac.uk/markr/

We’ve still got a few feed problems with it. But Yay!

The chart that dare not speak its name

The chart/graph has a name! What I’ve been calling a bubble chart is called something else on IBM’s Many Eyes. It is a Matrix Chart!

I was inspired by bubble charts (or Scatter Plots) and rather than mapping three continuous values, wanted a discrete axis with a longitudinal track. Hence this layout rather than a normal bubble chart.

Not that I expected to be the first one too it, but last time I looked at Many Eyes I didn’t see this one. So this form of chart is new to me. And doing a quick google search I can’t see too many other examples of this chart, except as references to Many Eyes. So maybe they came up with it.

It is unfortunate that Many Eyes doesn’t have any form of API or easy to use interface for getting data in. Otherwise this would make our lives much easier.

User participation, undiscovered use cases and spanners

One set of (prospective) users for this tool are the teachers on a level 1 module who have been using Blogger for teaching some basic HTML/CSS with. In fact these guys are the inspiration in many ways. Managing a large module that now running across UWE and three local colleges looks tedious and difficult… step in Markr… after it develops to the point where it can relieve workload, not add confusion.

One thing that the teachers on this module do is get the students to change their blogger blogs, update their templates with their new found CSS-foo. They then check in regularly to see changes, something that an RSS feed just doesn’t alert us too.

We have one or two other modules that provide smaller users groups who are only interested in the content (and what we can easily get at with RSS feeds and GData). But this provides an interesting and chunky use case to deal with.

Use Case

User: Teacher

Scenario:

1. A teacher is made aware that a student’s blog template has been changed
2. The blog is eyeballed manually
3. Markr records the result

Technical standards and stuff we can’t live without

The Google Data API and the Zend framework, hand-in-hand, have bootstrapped the project to where it is now.

One of the original research questions was whether we could effectively use blogs as a distributed data store, ie not have to cache everything in our databases, and thus possibly get out of synch with local content. And the GData Blogger API does give us that, and with remarkably quick response times. Currently we are fetching content, or checking for freshness, from individual blogs when a user looks at a specific blog post and it is working very well. Obviously one part of that equation is Google’s API.

The second part is the Zend PHP framework. We’ve gone through a process of refactoring most of the code and moving to a slightly more three-tier approach, but that isn’t the most important thing for us so far with Zend. The best bit to date is that the framework just deals with the XML we fetch and gives us nice data to play with. We don’t need to worry whether it is RSS or ATOM or deal with different versions. Zend just makes it happen, as well as managing our feed and API caching nicely. It helps that all Google’s XML comes back as ATOM.

links for 2009-09-14

• LingPipe: Database Mining Tutorial
  (tags: mysql datamining text-extraction)
• Feed Informer: Mix, convert, and republish feeds
  (tags: rss feed)
• RSSCloud Vs. PubSubHubbub: Why The Fat Pings Win
  Editor's note: With all of the debate lately between RSSCloud versus PubSubHubbub, we wanted to hear from a developer who could actually tell …
  (tags: rsscloud rss pubsubhubbub)
• SimplePie: Super-fast, easy-to-use, RSS and Atom feed parsing in PHP.
  PHP RSS Parser
  (tags: php rss)

links for 2009-09-08

• WordPress Just Made Millions of Blogs Real-Time With RSSCloud
  (tags: rss wordpress rsscloud)
• HowTo: Rebooting the RSS cloud
  (tags: rsscloud rss cloud)
• RSS in the Clouds « Blog « WordPress.com
  (tags: rss feed wordpress rsscloud)
• Speed in the Feed: Google Reader and PubSubHubbub
  (tags: rss pubsubhubbub google feed api)