Rapid Moodle development using Git

Recently I've been doing a lot of Moodle development, and every time I start a feature, or work on a bug, I've been creating a new branch. I often also create a new database, and a fresh install just to make sure that there's nothing fruity going on. All this has meant that I've had branches and databases coming out of my ears.

Since I've just taken possession of a new desktop for work, I've taken the opportunity to start afresh with my Moodle branches and I'm trying to be more organised in my branch creation. To that end, I've got the following system going:

• Generic moodle bug: MDL-<bug number>-<revision>
• Version for master: MDL-<bug number>-master-<revision>
• Version for 2.0: MDL-<bug number>-MOODLE_20_STABLE-<revision>
• Version for 2.1: MDL-<bug number>-MOODLE_21_STABLE-<revision>

This allows me to start work on a bug, and have relevant revisions to my patches in a sane and reasonably sensible (if not a touch long) fashion.

To make life simpler still, I've added to my moodle/config.php. This selects my database (and optionally database username which is sometimes handy) based on my branch name.

<?php

$branch = exec("git branch --no-color | grep '^* '| sed 's/^* //'");
$dbuser = 'moodle';

// First check for generic branch parents
if (preg_match('/master/', $branch)) {
    $newbranch = 'master';
} else if (preg_match('/MOODLE_20_STABLE/', $branch)) {
    $newbranch = 'MOODLE_20_STABLE';
} else if (preg_match('/MOODLE_21_STABLE/', $branch)) {
    $newbranch = 'MOODLE_21_STABLE';
} else if (preg_match('/MDL-/', $branch)) {
    // Any remaining MDL- matches which don't specify a branch will be
    // assumed to be on master
    $newbranch = 'MOODLE_21_STABLE';
}

// And now more specific parents
switch ($branch) {
    case 'example':
        $dbuser = 'some-other-dbuser';
        $newbranch = 'master';
        break;
    default:
        break;
}

$branch = $newbranch;

...

$CFG->dbname = 'moodle-' . $branch;
$CFG->dbuser = $dbuser;

I guess I'll see how it goes, but so far it's working well and I intend to replicate this with Mahara too.

Mahara 1.4 Cookbook Review

A short while ago I was asked to review the Mahara 1.4 Cookbook, written by Ellen Murphie. I was quite excited to see what suggestions she had to offer on the many differing ways to use Mahara.

The book is split into eight chapters covering different types of user and use-cases with each chapter being made of a number of recipes (well, it is a cookbook!). There are suggestions utilising many of the features of Mahara, plus in depth steps on how to carry out each recipe. Although the book does assume that you've got a little prior knowledge on how to use Mahara, there are some very basic recipes for those who have not.

The first chapter, Mahara for the Visual Arts, focuses primarily on visual arts and makes some great suggestions, such as combining the Collections feature with Audacity to create an audio-guided tour which I particularly liked.  I was especially pleased to see discussion of Creative Commons licensing -- a subject which more authors and artists should be aware of!  The second chapter, Literature and Writing, gives some good ideas for using Pages to present journals in a variety of different ways. I liked the combination of RSS feeds and Journals within a Page to present a newspaper page (A Daily Gazette) complete with topical and up-to-date external content such as YouTube.  I was also pleased to see that the book covered ePortfolio for professional use and not just for students with the inclusion of chapter three, The Professional Portfolio. I liked the way that a combination of Secret URLs were suggested as a way to give different potential employers access to a personalised Curriculum Vitae. The suggestion of uploading HTML, and Copying Pages were very useful to avoid duplicating effort and I'm glad that they were included.

Chapters four and five focused more on using Mahara as a teaching tool than for users wanting to create portfolios and I was surprised to see them in the middle of the book rather than at the end.  Chapter four, Working with Groups, gives ideas on different ways of using groups - primarily for collaboration, but also for assessment. It gives an introduction to the possible uses of groups, and details some of the basic operations (adding and removing members), and also goes on to detail some ideas on how to use groups to engage students more.  Primary education and teaching is a topic I'm not overly familiar with and I was surprised by the number of recipes in chapter five, The Primary Education Portfolio. I was intrigued by many of the ideas - I'd previously assumed that primary school aged students wouldn't typically work heavily with Mahara.  I liked the recipe on creating a reading list with book reports.

I felt that chapter six, The Social Portfolio, gave lots of good ideas to help users to organise their profile page and include all sorts of external content (e.g. Twitter, and external blogs) and it was good to see mention of the RSS export features of Mahara.

Chapters seven and eight were more focused on higher education and the recipes focused on using features of Mahara to exhibit work and information for college applications. Several of the recipes suggested similar ideas but used the techniques in different ways. I liked that the topic of archiving portfolio content was covered, though disappointed that LEAP2A was not actually discussed.

Overall, I found the book very interesting and it gave me some thoughts on how others might be using Mahara. I was a little disappointed that some of the tips at the end of the book weren't included earlier (notably the ability to upload a zip file and unzip it in Mahara, rather than uploading each file individually -- this wasn't touched upon until chapter 7), but I don't think that this detracts from the book as a whole as they were still covered.

I think that this book would be ideally suited to users wanting to be able to make their work stand out and be seen, but also to teachers and advisors looking for ideas to give their students. I shall definitely be recommending this book to others.

Mahara, Mahara, and more Mahara

Just a quick post really, I've been working really hard on all sorts of bits and pieces so not had much time to blog about several of the things I've been planning to blog on. Very frustrating but something I intend to rectify.

I've been a core Mahara developer for about 18  months, and working heavily with it for about two years now -- time flies when you're having fun! It's good fun working for a Mahara Partner and it's really good to be allowed time for open source contributions.

Sadly, I've been really busy at work recently so not had a huge amount of time for OSS work, but in my spare I've been working on some new features for Mahara which should be really uber cool. First off, I've been working with a patch that Penny Leach wrote about 2 years ago but which never got integrated. It's to add phpunit unit testing to Mahara. This should really help us to produce much better code and the plan is to run the unit tests on upload of every single patch, for both MySQL and Postgres.
I'm still trying to iron out some issues with MySQL and this, but hopefully soon I'll have that finished.

In a similar vain, I've then written a Command Line Interface (CLI) library for Mahara which should allow for much easier creation of CLI tools. I've then used that to create a CLI installer for Mahara, and a CLI upgrade script. Although this probably won't actually affect many end users, I do think that there are a certain group of users who will use this. There's also software such as CPanel which includes Mahara - but has typically had issues installing it for users without a CLI interface.
The main reason for writing these was again for testing purposes. Not only does it make life much easier for those of us developing mahara (I have a different database for every mahara branch so installing mahara at the drop of a hat is much appreciated), but it also means that we can machine test both installation, and upgrade of mahara as part of patch submission.

I've also been asked to review the upcoming Mahara 1.4 Cookbook (published by packt Publishing). Looking forward to reading it and seeing what suggestions other people have for using it. I'll be doing a quick demo of Mahara to a group of users within Lancaster University so I plan to use some of it's suggestions.

Server Distributions...

I've been thinking quite a bit over the past couple of weeks as to what is the 'best' server distribution. We use Debian at work and I think that it's healthy to re-evaluate decisions once in a while, and see if they still make sense.

I'm a strong believer in using a distribution which is well supported, has plenty of software pre-packaged in a sensible format, but still allows you to roll your own packages without pulling teeth. It shouldn't include X11 as standard, but should include useful things such as LVM and the software raid stack.

In my mind, the key players of the Linux distribution world, which most people seem to consider fo servers are:

• Debian;
• Ubuntu;
• RHEL;
• CentOS;
• Fedora; and
• SUSE.

Of course, there are a many many other distributions which people use for their servers, but I think I'll limit my thoughts to these six. I've excluded Solaris and BSD for this post - perhaps I'll cover them in another. I'm not going to look at SUSE for the moment - partially because I've had some frustrating experience with it, but partially because it's a similar model to CentOS with both it's support model and it's packaging model.

RHEL, Fedora and CentOS

RHEL

While it can't be denied that RHEL is well supported, it's hardly an affordable level of support. You buy support on a per-processor and virtual-guest level with prices from around $399/year. Of course, there will doubtless be a variety of discounts, bulk discounts, educational discounts, yadda yadda, etc, etc, but it hardly makes for an affordable model if you're running servers which aren't mission critical. If you're a bank, and downtime costs you serious money, then I'm sure that it's worth buying, but for most, it's very difficult to justify.

Fedora

As a result, many people seem to use Fedora. My main reason against using Fedora is it's bias towards a desktop system.  I'm not someone who wants their servers to have X11 installed, or all of the junk that you get with a window manager such as Gnome, or KDE. Fedora loses my vote primarily for this reason. The other real gripe that I have with Fedora is it's lack of support. The release cycle for Fedora is every six months, with that release being supported until your release + 2 has been out for a month. That means that, given the 6-monthly release cycle, you have security support for approximately 13 months if you install that server on day 1 of the release. Sure, you can jump from release, to release, to release, upgrading constantly, but the upgrade 'process' (if you can really call it that) is somewhat convoluted, and it's not really polite to cause a deliberate and avoidable downtime to all of your users every 6-12 months is it just because you choose to use a Desktop distribution for your server.

CentOS

I imagine that many users who are perfectly happy running Fedora on their desktops, but don't want to pay out for the exorbitant cost of RHEL support, therefore run CentOS. CentOS seems to be the mythical beast which perfectly encapsulates the issues I've already raised. It's a free system, with community support; and the support life cycle for each major version is seven years with releases made available 4-8 weeks after Red Hat publish the Source RPMs for RHEL. Minor releases seem to be made available approximately every 5-10 months.

So CentOS seems to be a really viable solution. I do have a couple of issues with it, mostly related to how they handle packages, or rather how they just don't seem to exist! Of course, a number of core packages do exist - things like Postgres, Apache, Perl, etc. But centos.org doesn't have a method for searching the list of available packages. It doesn't have many perl modules available (as far as I can tell) for example.

Of course, if you want to run software which isn't available out of the standard distribution packages, you probably want to roll your own packages anyway. You can, of course, build your own RPMs but again, it doesn't seem to be something that many people do. Instructions are available from Fedora and are valid for CentOS too.

Debian and Ubuntu

Debian

I should admit now, that I'm already a Debian convert. I use it for my work desktops, and all of my servers. The release cycle has recently changed to use a bi-annual freeze with the distribution released once that release is considered stable. Support for a release is available for about 1 year after it's moved to old-stable and, thus far, the support has been pretty good IMO. DSAs tend to be addressed pretty quickly, with packages generally released quickly too (sorry no stats for this) and few regressions caused by these.

On the packaging side, over 25,000 packages are available and cover a wide variety of software. Perl modules are well-catered for, as are python modules. Debian developers may only become an official developer after going through a pretty stringent process involving having your GPG key signed by other developers, and having a period of sponsorship by another developer. All packages are signed and verified which adds that warm-fuzzy feeling too.

Creating packages is also pretty well supported and very well documented and really is a breeze.

Ubuntu

Ubuntu is a derivitive of Debian which was created back in 2004 when Debian wasn't creating release cycles frequently enough for many. It still uses many of the same packages and much of the work done feeds back into the Debian project. Releases are every 6 months, but a Long-Term Support option is available which is released every second year and has support for five years.

Packages are the same as in Debian, and often newer versions of packages are available than in Debian.

However, I have a few niggles with Ubuntu which do put me off it a little. Only the core repository is supported, and only some of these packages are themselves supported. There have also been a fair few regressions in Ubuntu security releases which concern me.

My Summary

I think that, in summary, I'm pretty happy with my current distribution choice of Debian. I think that of the distributions I've looked at writing this post, it meets my requirements for security support, and release lifecycle. The availability of packages is very good (in my opinion) and it's really pretty easy to roll your own packages.

Debugging Mahara

As I've mentioned before, I do quite a bit of development work on Mahara.
There are some really handy debugging features which it's worth knowing about.


Configuration
As Mahara ships, debug, info, warn an environ error messages are sent to your error log.
Environ message are also sent to screen.
The default settings for logging are in lib/config-defaults.php under the Logging Configuration section.
I tend to have my log levels set to LOG_TARGET_SCREEN | LOG_TARGET_ERRORLOG as below:

$cfg->log_dbg_targets     = LOG_TARGET_SCREEN | LOG_TARGET_ERRORLOG;
$cfg->log_info_targets    = LOG_TARGET_SCREEN | LOG_TARGET_ERRORLOG;
$cfg->log_warn_targets    = LOG_TARGET_SCREEN | LOG_TARGET_ERRORLOG;
$cfg->log_environ_targets = LOG_TARGET_SCREEN | LOG_TARGET_ERRORLOG;

Functions
You can then use the various log functions to log anything. Variables are printed in a sane format which is easy to read. It's really quite a breath of fresh air when you compare it to something as primitive as var_dump:

log_debug('hello');
log_info($object);
log_warn($othervar);
log_environ($yetanothervar);

Notes

Mahara uses javascript form submission all over the place and if you're trying to debug a form using javascript, the log mesages won't be printed to screen until you next load the page.

As a result, it's often really helpful to tail the apache error log a lot of the time.

The power of git - splitting one file into multiple commits

Another really handy thing with Git, which I do use regularly, is it's ability to split lots of changes to the same file into separate logical commits.

As far as I know, any of the other VCSs I've used in the past haven't supported this and the only sane way I know of doing so would be to copy the file away, and use vimdiff to copy the logical commits one-by-one, committing each in turn.

Thankfully, git makes this really easy with the git add command.

Set up an example repository

# Create a new git repository for this example
cd /tmp/
mkdir git-add-pi
cd git-add-pi/
git init

# Make our first commit
echo "Example line" > example-file
git add example-file
git commit -m "First part of the example file"

Make some changes to our example file
Make a change to the beginning of the file:

(echo "Some big sweeping change to the file"; cat example-file) > tmp; mv tmp example-file

and make a change to the end of the file:

echo "Another big sweeping change to the same file" >> example-file

The magic
We actually wanted to split that file into two logical commits. With subversion, that would be a pain, but with git it's really easy with git add --pi:

523 git-add-pi:master> git add -pi example-file
diff --git a/example-file b/example-file
index d503a0c..82d4c17 100644
--- a/example-file
+++ b/example-file
@@ -1 +1,3 @@
+Some big sweeping change to the file
 Example line
+Another big sweeping change to the same file
Stage this hunk [y/n/a/d/s/?]?

Because the lines are so close to one another, git hasn't automatically split them up. Specify s to split them:

Stage this hunk [y/n/a/d/s/?]? s
Split into 2 hunks.
@@ -1 +1,2 @@
+Some big sweeping change to the file
 Example line
Stage this hunk [y/n/a/d/j/J/?]?

Well, we want to commit this hunk, so hit y

Stage this hunk [y/n/a/d/j/J/?]? y
@@ -1 +2,2 @@
 Example line
+Another big sweeping change to the same file
Stage this hunk [y/n/a/d/K/?]? n

We didn't want this hunk as part of this logical commit, so choose n

Checking what we've done
We can double check what we've added so far:

524 git-add-pi:master> git diff --cached
diff --git a/example-file b/example-file
index d503a0c..8bca897 100644
--- a/example-file
+++ b/example-file
@@ -1 +1,2 @@
+Some big sweeping change to the file
 Example line

And commit as normal:

525 git-add-pi:master> git commit -m "First change"
Created commit ad01f32: First change
 1 files changed, 1 insertions(+), 0 deletions(-)

What about the rest of the file?
You can repeat the add -pi as much as you like, or since we only have one more line to commit:

526 git-add-pi:master> git add .
git commi527 git-add-pi:master> git commit -m "final commit"
Created commit 16722ed: final commit
 1 files changed, 1 insertions(+), 0 deletions(-)

The power of git - splitting a commit

Every blog has to have a customary first post - or at least, almost every blog seems to. Rather than mine just saying that I'm starting a new blog, and I'll try and keep it up to date and blah blah blah, I thought I'd actually post some content.

For the past 9 months or so, I've been fairly heavily involved in an open source project called Mahara. Mahara is an ePortfolio system and I'm sure that I'll post about it more in the near future so I won't go into lots of detail now. Needless to say though, the work I've been doing on it has involved a variety of customisations and any sane developer stores, manages and tracks their work using a version control system or VCS. The mahara project uses git.

Git is probably the most powerful version control system I've come across and enables you to manage your source and commits with minimal effort and confusion - it's probably the only VCS I've ever used which works for you rather than making you work around it. That said, some of the really powerful features of git take a bit of understanding.

Earlier today I was telling someone how you can use git in some really cool and interesting ways. One of the things I do frequently is to take a commit and split it into a series of logical commits and I thought that I'd share this.

Setting up a quick git repository
If you want to try and follow with my examples, I've pushed my sample repository to gitorious.org. You can skip the first part by cloning it with:

git clone git://gitorious.org/thamblings/git-split-commits.git

Say that you've created a quick git repository with a few files in it:

# Create a new git repository
cd /tmp
mkdir git-example
cd git-example/
git init

# Commit the first file
echo "First File" >> one 
git add one 
git commit -m "First file committed"

# Now add a few more commits/files
echo "Second File" >> two 
echo "Third File" >> three
git add .
git reset
git add two 
git commit -m "Second File"
git add three
git commit -m "Third File"

# Now add two files in the same commit
echo "Fourth File" >> four
echo "Fifth File" >> five
git add .
git commit -m "Fourth and Fifth Files"

# And another file
echo "Sixth File" >> six 
git add .
git commit -m "Sixth File"

# Let's see what we've done
git log

But we made a mistake...
But wait a second - files four and five should have been in different files. Let's split them out.

Let's rewrite history
First we'll check out a new branch, and then we'll do an interactive rebase to edit the commit:

git checkout -b fixcommits
git rebase -i HEAD~2

We should get something like:

pick 0d3abd7 Fourth and Fifth Files
pick feaeea8 Sixth File

We want to edit commit 0d3abd7, so change that to an edit - you can use 'e' instead:

edit 0d3abd7 Fourth and Fifth Files
pick feaeea8 Sixth File

That takes us to the point after the commit 'Fourth and Fifth Files' and before 'Sixth File'.
We can then use git reset the state of our current HEAD.

git reset HEAD~1

We can then re-add and re-commit each file:

git add four
git commit -m "Fourth File"
git add five
git commit -m "Fifth File"

We can then continue our rebase and we'll be left with our finished article:

git rebase --continue

Let's get things back to master
And if we're happy with what we have, we can check it back in to our master branch with another rebase

git checkout master
git rebase fixcommits

The same also works for other git additions. So you could add parts of a file with an interactive git add (git add -pi) for example.