tag:blogger.com,1999:blog-91840957194955879952024-03-06T03:21:13.100+00:00Thoughts and ramblingsThoughts and rambling of the developer withinAndrewhttp://www.blogger.com/profile/00624839922000831199noreply@blogger.comBlogger9125tag:blogger.com,1999:blog-9184095719495587995.post-39459770342760368952013-07-11T14:26:00.001+01:002013-07-11T14:26:58.574+01:00Deploying Moodle - continued<div dir="ltr" style="text-align: left;" trbidi="on">
<div>
One thing that I didn't really touch upon in my previous blog post was
the method in which we actually deploy Moodle to our live servers.
</div>
<div>
Our production Moodle system is... well, complicated. We only have
14,000 students at present, but we're trying to really push use of the
Moodle and the last thing that we want to happen is for it to go down.
</div>
<div>
To the same aim, we want to be able to patch servers without any
downtime, and be able to take servers out of our load balancing pools
to enable us to prepare updates seamlessly to our end users.
<br />
<br />
I will try and improve this blog posting a bit more when I have some time...</div>
<h2>
Systems</h2>
<h3>
Architecture</h3>
<div>
We run our deployment on a purely virtualised infrastructure, currently
that's VMWare VSphere 5.1. We have a separate infrastructure team who
provide that infrastructure. The VMs sit on a pair of fully-redundant
and replicate Storage Area Networks (SANs), and our moodledata is
served over NFS by a NAS (Network Attached Storage) server.
</div>
<div>
Rather than having on beefy server to handle all of the load, we've
found it's more efficient to have lots of smaller servers. Our VMWare
specialists (Matt, and Graham) inform me that it's far better for
scheduling if we have fewer processor cores than too many. If we have
more than 4 processor cores on a VM, then VMWare has to try a lot
harder to allocate the resources, and it's harder to migrate those VMs
around the various blades.
</div>
<div>
At present, our architecture looks something like this (sorry, it's a little out of date, but should give you a fair idea):
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh3X0Mui19r7CUzh-Mm7cizxjkUb_wORwzUC6ggBt_ps018EJU-XuyR1kUUsaSMVLbrpaYwjkYAgJbf8sd-wkpp2g2Ugnd6uxT6H16EGuLrx_-LyGtvDQ21P25zjBUgeQrF0TvE4fy54w8/s1600/mle.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="237" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh3X0Mui19r7CUzh-Mm7cizxjkUb_wORwzUC6ggBt_ps018EJU-XuyR1kUUsaSMVLbrpaYwjkYAgJbf8sd-wkpp2g2Ugnd6uxT6H16EGuLrx_-LyGtvDQ21P25zjBUgeQrF0TvE4fy54w8/s320/mle.png" width="320" /></a></div>
<span id="goog_65986787"></span><span id="goog_65986788"></span><br /></div>
<h3>
Web servers</h3>
<div>
We have five live web servers (moodle-web[0,1,2,3,4]). These currently
have two cores allocated, and 3GB RAM. They only need a small disk
(20GB). Looking at our statistics throughout the year, we're likely to
relinquish 1G of this memory on each VM as it's largely going
unutilised.
</div>
<div>
We went for five servers because we want to be able to theoretically
lose a whole blade which may have a couple of web servers on it, and
not lose service. Theoretically VMWare should handle this
automagically, but we've seen cases where this hasn't happened as it
should.
</div>
<div>
Futhermore, we frequently pull one or two of these servers out of the pool to
perform maintenance. I'll be doing this next week in our Moodle 2.5
upgrade. I'll take two of the servers out of the pool, prepare them
with the upgrade and make sure that everything is all funky dory, and
then I'll perform the upgrade, swing over to the new servers, and
upgrade the old ones.
</div>
<div>
These web servers serve their content using Apache2 and we currently
use mod_php5 rather than fastcgi. We didn't find any particularly
stunning performance improvements with any of the CGI methods, but this
may be something we re-evaluate in the future.
</div>
<div>
We currently ues APC, but we may consider switching to PHP Opcache when
we upgrade to PHP 5.5 at some point. We follow Debian releases so we're
unlikely to see PHP 5.5 for a couple of years yet.
</div>
<h3>
Load balancers</h3>
<div>
To handle all of these web servers, we have a pair of load balancers. I
should point out that we're Debian nuts and we love Open Source. We do
our load balancing in Software with two software VMs.
</div>
<div>
These are also low-powered with 2 cores, and 1GB RAM (actually, one of
them has 2GB but we intend to reduce this back to 1GB).
</div>
<div>
We currently terminate our SSL connections on these load balancers with
nginx. When we made this decision, we were in two minds as to whether
it was the 'right' thing to do, but in retrospect it has worked very
well for us. It's a toss-up between being able to scale vertically at
the web server, or at the load balancer. The web servers don't need
public IP addresses, whilst the load balancers do. However, our web
servers cost more (in terms of resources), and Lancaster University is
extremely fortunate enough to have access to an entire slash-16 address
range with 65534 globals.
</div>
<div>
In addition to terminating the SSL with nginx, we also do an amount of
caching. Any images, and javascript, served by certain intelligient
endpoints in Moodle are cached more aggressively in memory on the load
balancer. This way, we don't even hit the load balancing software to
serve them, and we don't hit the web servers either. Handy!
</div>
<div>
We also use X-Accel-Redirect to serve many of the files from our NFS
server using nginx directly; rather than having PHP buffer them from
disk. This is much more efficient and saw a huge drop in our CPU
usage on the web servers with only a minor increase in CPU usage on our
frontends. Basically, web serving software is designed for serving
bytes off disk, whilst php is not. Again, it may seem a bit strange to
serve the files on the load balancer rather than the web servers, and
this is something that we may change in due course, but at present
we are forced to use Apache on our web servers, and the mod_sendfile
Module for Apache2 which does the same thing as X-Accel-Redirect for
nginx is much less mature.
</div>
<div>
After traffic has been terminated, and cached content retrieved and
served, it's then passed to our load balancing software, haproxy.
</div>
<div>
haproxy is an awesome little tool, which supports a range of really
powerful features including different allocation methods, session
stickiness, and it is also protocol aware for some protocols. It also
has handy logging.
</div>
<div>
I'll briefly mention that we use keepalived to manage the VRRP layer of
our stack. Each of our load balancers has a dedicated management IP,
and a virtual service IP. The management IP never changes and reflects
the name of the server providing service (e.g. moodle-fe0.lancs.ac.uk).
Meanwhile, the VIPs are free to fly wherever they need.
</div>
<div>
We also use round-robin DNS to direct traffic to the two load
balancers. I'd consider something like multicast DNS, but in our
current environment all of our servers are in the same pair of
datacentres and are only a mile apart and use the same IP range as
everything else on campus. There's really very little point at this
time.
</div>
<div>
At any point, we can take a load balancer out of service for
maintenance. We frequently do so and our end users shouldn't notice at
all. They'll still get sent to the same web servers that were handling
their request before.
</div>
<h2>
Software distribution</h2>
<div>
As I mentioned before, we're Debian nuts. We love Debian. We use it for
pretty much everything (I think we have one Ubuntu box for
BigBlueButton, but that's a Debian derivative anyway).
</div>
<h3>
Server configuration</h3>
<div>
We also have a configuration management suite called configutil. It
was written by a former employee, Chris Allen, and was originally based
on Oxford University's configtool. However, we've pretty much rewritten
it now and it does some pretty cool stuff. This includes distributed
iptables, user management, package management, service management, and
file deployment. A large chunk of this is actually handed over to
puppet, but we build the puppet manifest with configbuild, and deploy
the files with configutil/luns-config.
</div>
<div>
We keep all of our server configuration in git too (did I mention, we
like git), build the configuration using configbuild. Servers have a
deployment tool called luns-config which syncs against the
configuration server.
</div>
<div>
In addition to liking Moodle, Mahara, Debian, and git, we also like
security. In fact, we really really like security.
</div>
<div>
I'm not just talking about security in getting onto our systems (all of
our servers are behind our corporate firewall, plus have strict
iptables. We then enforce ssh keys on all servers). We also like our
configuration to be safe. Our configuration is served over SSL, with
client-side key verification too using our internal certificate
authority. We generate revocation lists frequently and if the list goes
out of date (6 monthly IIRC), we stop serving any configuration. A
server can only retrieve configuration for the server named in it's
configuration management certificate. On our package management server,
we employ the same type of client-side certificate requirement so only
systems with a valid SSL certificate and key-pair can access our
configuration.
</div>
<h3>
Software deployment</h3>
<div>
So now we've got all of that out of the way... I did mention that we
really like Debian right? Right, good. Because we deploy all of our
software in the form of Debian Packages. I mean all of it.
</div>
<div>
We've gone down this route for a number of reasons, some of them
theoretical advantages, and some of them learned from experiences. They
really come down to these though:
<br />
<ul>
<li>we want to be sure that we know what software is on a server;</li>
<li>we want to be sure that each server in a group is identical;</li>
<li>we want to be able to install a new server quickly and easily;</li>
<li>we want the ability to roll back to a previous version if we screw up; and</li>
<li>we have a tendency to twitch if we come across things out of place.</li>
</ul>
</div>
<div>
Basically, what it comes down to, is that we want to be able to quickly
and easily build replacement servers, add new servers, re-install
servers, etc. Most of our servers are entirely disposable. We try to
keep all data on dedicated storage. As I mentioned before, our
moodledata is on NFS. In reality, most of our data across all servers
is stored on our NAS and served over NFS.
</div>
<div>
So if we discover that we're breaching the limits of our server
configuration, we can scale horizontally (that is the right one isn't
it?) and have a server built with a known configuration in a very short
period of time (typically about an hour).
</div>
<div>
To this aid, we package all of our software. So each Moodle
installation is a separate Debian package. Debian packages are awesome.
</div>
<div>
When we upgrade Moodle, we update the code using git (see <a href="http://www.blogger.com/thamblings.blogspot.com/2013/07/upgrading-moodle-from-git.html">thamblings.blogspot.com/2013/07/upgrading-moodle-from-git.html</a>
for my post on that topic). Once we've done that, we merge our new
deployment branch into a new git branch - luns-moodle-lu_2.5.
</div>
<div>
This has the debian packaging information in it and this is where we
create our package from.
</div>
<div>
Why not just keep our packaging data in our LUVLE-2-5.deployment
branch? Well, we could do, but we feel that this is cleaner. I mean
cleaner both in terms of separation of processes, and history.
</div>
<div>
For example, if we discover a bug in our package (like a missing
dependency), then we want to make that change on our packaging branch.
We don't want that change mixed up with the history of our Moodle
codebase.
</div>
<div>
If you're interested in our package skeleton, I've put it in a gist at
<a href="https://gist.github.com/andrewnicols/ae439676d116e9a6582f">https://gist.github.com/andrewnicols/ae439676d116e9a6582f</a>.
These files all go into the debian directory, and then you can run dch
--create to create an empty Changelog. You will, of course, need to
update the control file to reflect your package name.
</div>
<div>
So once we've made our chnages, we merge the deployment branch into our
packaging branch; we incremement the version number; build the package;
add it to our package server; and deploy each of the frontends. Here's
a summary of that process:
</div>
<script src="https://gist.github.com/andrewnicols/b17ddeaec5ef36f161fc.js?file=process.sh"></script>
<br />
<div>
Hmm - that looks very long winded, but in manys ways it's just lots of
small repetitive tasks which separate concerns, and make our lives
easier in the long-run.
</div>
<div>
Now we've done that, to deploy on our five web servers, and our cron
server we just run:
</div>
sudo apt-get update; sudo apt-get upgrade
<br />
<div>
Nice, and easy.
</div>
</div>
Andrewhttp://www.blogger.com/profile/00624839922000831199noreply@blogger.com0tag:blogger.com,1999:blog-9184095719495587995.post-37660872050037289622013-07-06T22:31:00.002+01:002013-07-06T22:51:49.723+01:00Upgrading Moodle from Git<div dir="ltr" style="text-align: left;" trbidi="on">
<h1>
Upgrading Moodle with Git</h1>
<h2>
Background</h2>
<span class="Apple-style-span" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">
I've been working on a new deployment of Moodle for Lancaster
University for the past two years or so. Our project started out with
Moodle 2.1, and we upgraded for our initial pilot to Moodle 2.2.
<br />
Since then, we've upgraded from Moodle 2.2 to 2.3; and we're now
planning the upgrade from 2.3 to 2.5.
<br />
We manage all of our upgrades with git, and our deployment using Debian
packages.
<br />
I've been asked a couple of times to write about our upgrade methodology
and reasoning so hopefully others will find this useful.
<br />
We use a variety of git features, but new features are added to git
all the time which change our deployment methodology from time-to-time.
At present we use:
<br />
</span><br />
<ul><span class="Apple-style-span" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">
<li>branches</li>
<li>tags</li>
<li>multiple remotes</li>
<li>merges</li>
<li>cherry-picks</li>
<li>rebase</li>
</span></ul>
<span class="Apple-style-span" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">
I'll try and cover how we use all of the above. We probably use other
features in addition to these, but these are the ones that spring to
mind.
<br />
In addition to the git side of things, we also make use of <a href="http://www.redmine.org/">Redmine</a> for issue tracking.
</span><br />
<h2>
Branches, Tags, and Remotes</h2>
<span class="Apple-style-span" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">
We have quite a few of these, but they really do make our life easier.
Git is a fantastic tool, and if used to it's full extent, having a lage
number of branches actually makes your life much easier, and less
complicated. Having multiple remotes helps to separate various
concerns too so you can ensure that it's harder to inadvertantly
publish your institution's IP.
</span><br />
<h3>
Branches</h3>
<span class="Apple-style-span" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">
In summary, we have one branch per feature, hack, or change in core
Moodle - no matter how small. These are named in a (hopefully) sensible
naming scheme to help identify them from one another easily and
quickly. The name describes the project/customer (usually LUVLE in our
case), the version, the type of the change, and the frankenstyle name
for that change. For some changes, we have an optional short name to
describe the branch further. Our naming scheme works out as:
</span><br />
<code>
{customer}-{major-version}-{change type}-{frankenstyle}[-{shortname}]
</code>
<span class="Apple-style-span" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">
Where we have several related features which must co-exist and cannot
be used without one another, we use a custom frankenstyle name of
set_{name}.
<br />
As an example, these are some fo the branches for our impending Moodle
2.5.0 upgrade:
</span><br />
<ul>
<li><code>LUVLE-2.5-feature-mod_ouwiki</code></li>
<li><code>LUVLE-2.5-feature-block_panopto</code></li>
<li><code>LUVLE-2.5-feature-local_luassignment</code></li>
<li><code>LUVLE-2.5-feature-set_bigbluebuttonbn</code></li>
<li><code>LUVLE-2.5-hack-mod_resource-singlefiles</code></li>
</ul>
<span class="Apple-style-span" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">
All of these branches are based on the same upstream tag - in this
case, <i>v2.5.0</i> for the 2.5.0 version of Moodle. We always use
this tag. Even when 2.5.32 has come out we will still use 2.5.0 (though
hopefully we won't ever get that far behind!). This may seem a touch
strange at first, but when it comes to merging all of our features and
changes together into a single testing or deployment branch, we want to
avoid any merge conflicts created by different versions. It's also much
easier when it comes to subsequent newer versions of Moodle in the
future.
<br />
By having each feature in it's own branch, we're able to develop, and
test that branch entirely in isolation.
<br />
In the rare cases that we are working with a feature which needs a
minimum release version which includes a minor increment (e.g. 2.3.1),
we check out from that tag instead, but we try to avoid this to make
things simpler.
<br />
In addition to all of the feature and hack branches, we also have a
range of testing and deployment branches. Generally, we have a main
test branch which contains the same branches as our deployment
environment. This is updated frequently when we want to test an upgrade
to a whole branch in combination with the rest of our installation, or
to test an upgrade to Moodle. Meanwhile, we typically only have a
single deployment branch - <i>LUVLE-2.5-deployment</i>. This is to
avoid any confusion and potential for dropped branches.
</span><br />
<h3>
Tags</h3>
<span class="Apple-style-span" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">
Similarly Tags are great too - we tag for each of our releases. Tags
offer a few really handy features. Obviously, they can just be used to
jump between version of Moodle; but we can also use them to compare a
particular file, or directory with a previous release to see how things
have changed. For development, this can be invaluable.
<br /><br />
Again, we have a pretty defined naming policy for tags:
</span><br />
<code>
{ourname}-moodle-{customer-name}_v{moodle-version}-{local-release}
</code>
<span class="Apple-style-span" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">
As an example we have some of the following:
</span><br />
<ul>
<li><code>lu-moodle-lu_v2.3.4-0</code></li>
<li><code>lu-moodle-lu_v2.3.4-7</code></li>
<li><code>lu-moodle-lu_v2.5.0-0</code></li>
</ul>
<span class="Apple-style-span" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">
If we wanted to see how a particular file had changed against our
previous release we can then run a command such as:
</span><br />
<pre> git diff lu-moodle-lu_v2.3.4-0 enrol/lusi/lib.php
</pre>
<h3>
Remotes</h3>
<span class="Apple-style-span" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">
As I mentioned before, one great reason for multiple remotes is to give
you a separation of concerns. There are times where you don't wish to
push some of your branches to the public, other times where you're
working on bug fixes you don't really need to push to an internal
repository, and all manner of other reasons besides.
<br /><br />
I have the following remotes to make my life easier:
</span><br />
<ul>
<li>
<b>origin - git.moodle.org/moodle.git </b> - my main upstream;
</li>
<li>
<b>integration - git.moodle.org/integration.git </b> - the moodle.org
integration branch. Useful when fixing issues that crop up during
integration;
</li>
<li>
<b>public - github.com/andrewnicols/moodle.git </b> - the repository I
push any bug fixes and new features for the community to; and
</li>
<li>
<b>cis - ciggit.lancs.ac.uk/moodle.git </b> - our main internal
repository.
</li>
</ul>
<span class="Apple-style-span" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">
As a general policy, and from experience of making oopsies, I've found
it best to have each remote start with a different letter - it also
makes tab completion much less frustrating.
</span><br />
<h2>
The process</h2>
<span class="Apple-style-span" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">
When a new release of Moodle comes out, we've typically taken a bit of
laborious approach to things. Whilst there are a lot of steps, I feel
that in the long run they've been less frustrating than trying to
resolve any merge conflicts; and we've saved time trying scratching
heads trying to work out where this change, or that whitespace conflict
came from.
</span><br />
<h3>
Initial set-up</h3>
<span class="Apple-style-span" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">
Once we've checked out a new branch for every single feature, and hack,
we begin to bring them all together. That's nice and easy when you're
just starting out - just git merge a lot:
</span><br />
<script src="https://gist.github.com/andrewnicols/af4c97a8edbb3ebcc2fb.js?file=initialsetup.sh"></script>
<span class="Apple-style-span" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">
And hey-presto... we <em>should</em> have our Moodle 2.5 installation ready
for testing and deployment. Once we're happy with our installation, we
usually then create a deployment branch from that testing branch.
</span><br />
<h3>
Grabbing a fix from upstream</h3>
<span class="Apple-style-span" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">
We frequently come across issues which have already been fixed in
upstream Moodle, or which we ourselves have helped to fix. Sometimes we
also backport features from a newer branches onto our production branch
if we really really want it.
<br /><br />
We do all of this with the fantastic git cherry-pick command which
allows you to pick a commit, or a number of commits, and apply them to
your current branch.
</span><br />
<script src="https://gist.github.com/andrewnicols/af4c97a8edbb3ebcc2fb.js?file=cherrypick_upstream.sh"></script>
<h3>
Updating our local branches</h3>
<span class="Apple-style-span" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">
For updating one of our feature branches, we simply make our changes to
that specific branch, and then merge them back in again:
</span><br />
<script src="https://gist.github.com/andrewnicols/af4c97a8edbb3ebcc2fb.js?file=update_local.sh"></script>
<h3>
Upgrading Moodle - Minor releases</h3>
<span class="Apple-style-span" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">
In reality, a minor update to Moodle is just the same as an update to
one of our local branches. If anything, it's probably simpler:
</span><br />
<script src="https://gist.github.com/andrewnicols/af4c97a8edbb3ebcc2fb.js?file=update_remote.sh"></script>
<h3>
Upgrading Moodle - Major releases</h3>
<span class="Apple-style-span" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">
This is where things get much more complicated, and where the number of
steps and the complication rapidly increases. That said, in my opinion
they also reduce the confusion later on.
</span><br />
<h4>
Externally provided code</h4>
<span class="Apple-style-span" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">
We start by grabbing the latest version of the externally provided
plugins and starting brand new branches for them. There's usually very
little point in keeping the history for those branches as it doesn't
contain any of the upstream commit messages.
</span><br />
<script src="https://gist.github.com/andrewnicols/af4c97a8edbb3ebcc2fb.js?file=updatebranch_external.sh"></script>
<h4>
Local branches which need updating</h4>
<span class="Apple-style-span" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">
For our local branches, we want to preserve this history of our
changes. We also want to remove any confusion with merges to newer
versions to keep the history as clear as possible.
<br /><br />
To do so, we use the wonderful <code>git rebase --onto</code> command.
<br /><br />
With a normal rebase command, git takes every commit since your branch
diverged from the new upstream, and attempts to replay each of them on
top of the new head.
<br /><br />
The <em>--onto</em> tells rebase where to take the commits from for the
re-application. That's to say, that if you only have one commit since
you branches from the tag, it grabs that commit, and immediately tries
to replay on top of the target version. Ordinarily, it would attempt to
reapply your commit on top of every other commit.
</span><br />
<script src="https://gist.github.com/andrewnicols/af4c97a8edbb3ebcc2fb.js?file=updatebranch_local.sh"></script>
<span class="Apple-style-span" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">
That's it. It's really simple, but it needs to be done for each and
every branch that you have.
<br /><br />
Finally, once all branches have been updated, we merge them into a new
testing branch and begin our testing phase.
</span>
<h3>Tracking things</h3>
<span class="Apple-style-span" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">
In order to make sure that we don't lose track of anything, or forget a branch during an upgrade, we make use of our issue tracking software <b>Redmine</b>.
<br />
When we start to put a version together, for example Moodle 2.2, we create a new task for the next upgrade - in this case, 2.3.
<br />
As we include each of our feature branches, we create a new subtask under the 2.3 task.
<br />
When we came to upgrading to Moodle 2.3, we then go through each of those subtasks and make any relevant notes from the upgrade process. We also create a new task for the subsequent upgrade (e.g. Moodle 2.5).
<br />
If we are no longer including a branch because it is now redundant or we have decided to change the functionality offered, we also note this in the relevant issue.
<br />
This all helps to ensure that we don't forget an issue and that we keep a record of all changes.
</span>
</div>
Andrewhttp://www.blogger.com/profile/00624839922000831199noreply@blogger.com1tag:blogger.com,1999:blog-9184095719495587995.post-41750354249514020722011-10-19T10:39:00.001+01:002011-10-19T10:40:51.442+01:00Rapid Moodle development using GitRecently I've been doing a lot of Moodle development, and every time I start a feature, or work on a bug, I've been creating a new branch. I often also create a new database, and a fresh install just to make sure that there's nothing fruity going on. All this has meant that I've had branches and databases coming out of my ears.<br />
<br />
Since I've just taken possession of a new desktop for work, I've taken the opportunity to start afresh with my Moodle branches and I'm trying to be more organised in my branch creation. To that end, I've got the following system going:<br />
<br />
<ul>
<li>Generic moodle bug: MDL-<bug number>-<revision></li>
<li>Version for master: MDL-<bug number>-master-<revision></li>
<li>Version for 2.0: MDL-<bug number>-MOODLE_20_STABLE-<revision></li>
<li>Version for 2.1: MDL-<bug number>-MOODLE_21_STABLE-<revision></li>
</ul>
<div>
This allows me to start work on a bug, and have relevant revisions to my patches in a sane and reasonably sensible (if not a touch long) fashion.</div>
<div>
<br /></div>
<div>
To make life simpler still, I've added to my moodle/config.php. This selects my database (and optionally database username which is sometimes handy) based on my branch name.</div>
<div>
<br /></div>
<div>
<pre class="brush: php"><?php
$branch = exec("git branch --no-color | grep '^* '| sed 's/^* //'");
$dbuser = 'moodle';
// First check for generic branch parents
if (preg_match('/master/', $branch)) {
$newbranch = 'master';
} else if (preg_match('/MOODLE_20_STABLE/', $branch)) {
$newbranch = 'MOODLE_20_STABLE';
} else if (preg_match('/MOODLE_21_STABLE/', $branch)) {
$newbranch = 'MOODLE_21_STABLE';
} else if (preg_match('/MDL-/', $branch)) {
// Any remaining MDL- matches which don't specify a branch will be
// assumed to be on master
$newbranch = 'MOODLE_21_STABLE';
}
// And now more specific parents
switch ($branch) {
case 'example':
$dbuser = 'some-other-dbuser';
$newbranch = 'master';
break;
default:
break;
}
$branch = $newbranch;
...
$CFG->dbname = 'moodle-' . $branch;
$CFG->dbuser = $dbuser;
</pre>
<br />
I guess I'll see how it goes, but so far it's working well and I intend to replicate this with Mahara too.</div>Andrewhttp://www.blogger.com/profile/00624839922000831199noreply@blogger.com0tag:blogger.com,1999:blog-9184095719495587995.post-64669974115263607382011-10-09T16:00:00.000+01:002011-10-09T16:00:06.985+01:00Mahara 1.4 Cookbook Review<div class="separator" style="clear: both; text-align: center;">
<a href="http://www.packtpub.com/sites/default/files/5061OS_Mahara%20Cookbook_0.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><span class="Apple-style-span" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><img border="0" height="200" src="http://www.packtpub.com/sites/default/files/5061OS_Mahara%20Cookbook_0.jpg" width="161" /></span></a></div>
<span class="Apple-style-span" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">A short while ago I was asked to review the <a href="http://www.packtpub.com/mahara-1-4-cookbook-for-training-education/book">Mahara 1.4 Cookbook</a>, written by Ellen Murphie. I was quite excited to see what suggestions she had to offer on the many differing ways to use Mahara.</span><br />
<br />
<span class="Apple-style-span" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">The book is split into eight chapters covering different types of user and use-cases with each chapter being made of a number of recipes (well, it is a cookbook!). There are suggestions utilising many of the features of Mahara, plus in depth steps on how to carry out each recipe. Although the book does assume that you've got a little prior knowledge on how to use Mahara, there are some very basic recipes for those who have not.</span><br />
<br />
<span class="Apple-style-span" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">The first chapter, <i>Mahara for the Visual Arts,</i> focuses primarily on visual arts and makes some great suggestions, such as combining the Collections feature with <i><a href="http://audacity.sourceforge.net/">Audacity</a></i> to create an audio-guided tour which I particularly liked. I was especially pleased to see discussion of <i><a href="http://creativecommons.org/">Creative Commons licensing</a> --</i> a subject which more authors and artists should be aware of! </span><span class="Apple-style-span" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">The second chapter, <i>Literature and Writing</i>, gives some good ideas for using Pages to present journals in a variety of different ways. I liked the combination of RSS feeds and Journals within a Page to present a newspaper page (<i>A Daily Gazette</i>) complete with topical and up-to-date external content such as <a href="http://www.youtube.com/">YouTube</a>. </span><span class="Apple-style-span" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">I was also pleased to see that the book covered ePortfolio for professional use and not just for students with the inclusion of chapter three, <i>The Professional Portfolio.</i> I liked the way that a combination of Secret URLs were suggested as a way to give different potential employers access to a personalised Curriculum Vitae. The suggestion of uploading HTML, and Copying Pages were very useful to avoid duplicating effort and I'm glad that they were included.</span><br />
<br />
<span class="Apple-style-span" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">Chapters four and five focused more on using Mahara as a teaching tool than for users wanting to create portfolios and I was surprised to see them in the middle of the book rather than at the end. </span><span class="Apple-style-span" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">Chapter four, <i>Working with Groups</i>, gives ideas on different ways of using groups - primarily for collaboration, but also for assessment. It gives an introduction to the possible uses of groups, and details some of the basic operations (adding and removing members), and also goes on to detail some ideas on how to use groups to engage students more. </span><span class="Apple-style-span" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">Primary education and teaching is a topic I'm not overly familiar with and I was surprised by the number of recipes in chapter five, <i>The Primary Education Portfolio</i>. I was intrigued by many of the ideas - I'd previously assumed that primary school aged students wouldn't typically work heavily with Mahara. I liked the recipe on creating a reading list with book reports.</span><br />
<br />
<span class="Apple-style-span" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">I felt that chapter six, <i>The Social Portfolio</i>, gave lots of good ideas to help users to organise their profile page and include all sorts of external content (e.g. Twitter, and external blogs) and it was good to see mention of the RSS export features of Mahara.</span><br />
<br />
<span class="Apple-style-span" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">Chapters seven and eight were more focused on higher education and the recipes focused on using features of Mahara to exhibit work and information for college applications. Several of the recipes suggested similar ideas but used the techniques in different ways. I liked that the topic of archiving portfolio content was covered, though disappointed that LEAP2A was not actually discussed.</span><br />
<br />
<span class="Apple-style-span" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">Overall, I found the book very interesting and it gave me some thoughts on how others might be using Mahara. I was a little disappointed that some of the tips at the end of the book weren't included earlier (notably the ability to upload a zip file and unzip it in Mahara, rather than uploading each file individually -- this wasn't touched upon until chapter 7), </span><span class="Apple-style-span" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">but I don't think that this detracts from the book as a whole as they were still covered.</span><br />
<br />
<span class="Apple-style-span" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">I think that this book would be ideally suited to users wanting to be able to make their work stand out and be seen, but also to teachers and advisors looking for ideas to give their students. I shall definitely be recommending this book to others.</span>Andrewhttp://www.blogger.com/profile/00624839922000831199noreply@blogger.com0tag:blogger.com,1999:blog-9184095719495587995.post-56795326969190500042011-09-15T10:59:00.000+01:002011-09-15T10:59:02.263+01:00Mahara, Mahara, and more MaharaJust a quick post really, I've been working really hard on all sorts of bits and pieces so not had much time to blog about several of the things I've been planning to blog on. Very frustrating but something I intend to rectify.<br />
<br />
I've been a core <a href="http://www.mahara.org/">Mahara</a> developer for about 18 months, and working heavily with it for about two years now -- time flies when you're having fun! It's good fun working for a <a href="http://www.luns.net.uk/">Mahara Partner</a> and it's really good to be allowed time for open source contributions.<br />
<br />
Sadly, I've been really busy at work recently so not had a huge amount of time for OSS work, but in my spare I've been working on some new features for Mahara which should be really uber cool. First off, I've been working with a patch that Penny Leach wrote about 2 years ago but which never got integrated. It's to add <a href="https://github.com/sebastianbergmann/phpunit/">phpunit</a> unit testing to Mahara. This should really help us to produce much better code and the plan is to run the unit tests on upload of every single patch, for both MySQL and Postgres.<br />
I'm still trying to iron out some issues with MySQL and this, but hopefully soon I'll have that finished.<br />
<br />
In a similar vain, I've then written a Command Line Interface (CLI) library for Mahara which should allow for much easier creation of CLI tools. I've then used that to create a CLI installer for Mahara, and a CLI upgrade script. Although this probably won't actually affect many end users, I do think that there are a certain group of users who will use this. There's also software such as CPanel which includes Mahara - but has typically had issues installing it for users without a CLI interface.<br />
The main reason for writing these was again for testing purposes. Not only does it make life much easier for those of us developing mahara (I have a different database for every mahara branch so installing mahara at the drop of a hat is much appreciated), but it also means that we can machine test both installation, and upgrade of mahara as part of patch submission.<br />
<br />
I've also been asked to review the upcoming <a href="http://goo.gl/pZsQa">Mahara 1.4 Cookbook</a> (published by packt Publishing). Looking forward to reading it and seeing what suggestions other people have for using it. I'll be doing a quick demo of Mahara to a group of users within Lancaster University so I plan to use some of it's suggestions.Andrewhttp://www.blogger.com/profile/00624839922000831199noreply@blogger.com0tag:blogger.com,1999:blog-9184095719495587995.post-11542184738743061722010-12-29T21:12:00.002+00:002010-12-29T21:28:03.291+00:00Server Distributions...<span class="Apple-style-span" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">I've been thinking quite a bit over the past couple of weeks as to what is the 'best' server distribution. We use Debian at work and I think that it's healthy to re-evaluate decisions once in a while, and see if they still make sense.</span><br />
<br />
<span class="Apple-style-span" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">I'm a strong believer in using a distribution which is well supported, has plenty of software pre-packaged in a sensible format, but still allows you to roll your own packages without pulling teeth. It shouldn't include X11 as standard, but should include useful things such as LVM and the software raid stack.</span><br />
<br />
<span class="Apple-style-span" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">In my mind, the key players of the Linux distribution world, which most people seem to consider fo servers are:</span><br />
<ul><li><span class="Apple-style-span" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">Debian;</span></li>
<li><span class="Apple-style-span" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">Ubuntu;</span></li>
<li><span class="Apple-style-span" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">RHEL;</span></li>
<li><span class="Apple-style-span" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">CentOS;</span></li>
<li><span class="Apple-style-span" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">Fedora; and</span></li>
<li><span class="Apple-style-span" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">SUSE.</span></li>
</ul><span class="Apple-style-span" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">Of course, there are a many many other distributions which people use for their servers, but I think I'll limit my thoughts to these six. I've excluded Solaris and BSD for this post - perhaps I'll cover them in another. I'm not going to look at SUSE for the moment - partially because I've had some frustrating experience with it, but partially because it's a similar model to CentOS with both it's support model and it's packaging model.</span><br />
<br />
<h3 class="Apple-style-span" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">RHEL, Fedora and CentOS</h3><h4 class="Apple-style-span" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">RHEL</h4><span class="Apple-style-span" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">While it can't be denied that RHEL is well supported, it's hardly an affordable level of support. You buy support on a per-processor and virtual-guest level with prices from around $399/year. Of course, there will doubtless be a variety of discounts, bulk discounts, educational discounts, yadda yadda, etc, etc, but it hardly makes for an affordable model if you're running servers which aren't mission critical. If you're a bank, and downtime costs you serious money, then I'm sure that it's worth buying, but for most, it's very difficult to justify.</span><br />
<br />
<h4 class="Apple-style-span" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">Fedora</h4><span class="Apple-style-span" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">As a result, many people seem to use<i> </i>Fedora. My main reason against using Fedora is it's bias towards a desktop system. I'm not someone who wants their servers to have X11 installed, or all of the junk that you get with a window manager such as Gnome, or KDE. Fedora loses my vote primarily for this reason. The other real gripe that I have with Fedora is it's lack of support. The <a href="http://fedoraproject.org/wiki/LifeCycle">release cycle</a> for Fedora is every six months, with that release being supported until your release + 2 has been out for a month. That means that, given the 6-monthly release cycle, you have security support for approximately 13 months if you install that server on day 1 of the release. Sure, you can jump from release, to release, to release, upgrading constantly, but the <a href="http://fedoraproject.org/wiki/YumUpgradeFaq">upgrade 'process'</a> (if you can really call it that) is somewhat convoluted, and it's not really polite to cause a deliberate and avoidable downtime to all of your users every 6-12 months is it just because you choose to use a Desktop distribution for your server.</span><br />
<br />
<h4 class="Apple-style-span" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">CentOS</h4><span class="Apple-style-span" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">I imagine that many users who are perfectly happy running Fedora on their desktops, but don't want to pay out for the exorbitant cost of RHEL support, therefore run <a href="http://www.centos.org/">CentOS</a>. CentOS seems to be the mythical beast which perfectly encapsulates the issues I've already raised. It's a free system, with community support; and the support life cycle for each major version is seven years with releases made available <a href="http://wiki.centos.org/Download">4-8 weeks after Red Hat publish the Source RPMs for RHEL</a>. Minor releases seem to be made available <a href="http://en.wikipedia.org/wiki/CentOS#Release_history">approximately every 5-10 months</a>.</span><br />
<br />
<span class="Apple-style-span" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">So CentOS seems to be a really viable solution. I do have a couple of issues with it, mostly related to how they handle packages, or rather how they just don't seem to exist! Of course, a number of core packages do exist - things like Postgres, Apache, Perl, etc. But centos.org doesn't have a method for searching the list of available packages. It doesn't have many perl modules available (as far as I can tell) for example.</span><br />
<br />
<span class="Apple-style-span" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">Of course, if you want to run software which isn't available out of the standard distribution packages, you probably want to roll your own packages anyway. You can, of course, build your own RPMs but again, it doesn't seem to be something that many people do. Instructions are available from <a href="http://fedoraproject.org/wiki/How_to_create_an_RPM_package">Fedora</a> and are valid for CentOS too.</span><br />
<br />
<h3 class="Apple-style-span" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">Debian and Ubuntu</h3><h4 class="Apple-style-span" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">Debian</h4><span class="Apple-style-span" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">I should admit now, that I'm already a Debian convert. I use it for my work desktops, and all of my servers. The release cycle has <a href="http://www.debian.org/News/2009/20090729">recently changed</a> to use a bi-annual freeze with the distribution released once that release is considered stable. Support for a release is available for <a href="http://www.debian.org/security/faq#lifespan">about 1 year</a> after it's moved to old-stable and, thus far, the support has been pretty good IMO. DSAs tend to be addressed pretty quickly, with packages generally released quickly too (sorry no stats for this) and few regressions caused by these.</span><br />
<br />
<span class="Apple-style-span" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">On the packaging side, over <a href="http://www.debian.org/">25,000 packages</a> are available and cover a wide variety of software. Perl modules are well-catered for, as are python modules. Debian developers may only become an official developer after going through a pretty <a href="http://www.debian.org/devel/join/newmaint">stringent process</a> involving having your GPG key signed by other developers, and having a period of sponsorship by another developer. All packages are signed and verified which adds that warm-fuzzy feeling too.</span><br />
<br />
<span class="Apple-style-span" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">Creating packages is also pretty well supported and <a href="http://www.debian.org/doc/maint-guide/">very well documented</a> and really is a breeze.</span><br />
<br />
<h4 class="Apple-style-span" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">Ubuntu</h4><span class="Apple-style-span" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">Ubuntu is a derivitive of Debian which was created back in 2004 when Debian wasn't creating release cycles frequently enough for many. It still uses many of the same packages and much of the work done feeds back into the Debian project. Releases are every 6 months, but a Long-Term Support option is available which is released every second year and has support for five years.</span><br />
<br />
<span class="Apple-style-span" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">Packages are the same as in Debian, and often newer versions of packages are available than in Debian.</span><br />
<br />
<span class="Apple-style-span" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">However, I have a few niggles with Ubuntu which do put me off it a little. Only the core repository is supported, and <a href="https://help.ubuntu.com/community/ServerFaq#What packages and repositories are maintained (supported)?">only some</a> of these packages are themselves supported. There have also been <a href="http://www.google.co.uk/search?q=ubuntu+security+upgrade+regression+site:ubuntu.com">a fair few</a> regressions in Ubuntu security releases which concern me.</span><br />
<br />
<h3 class="Apple-style-span" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">My Summary</h3><span class="Apple-style-span" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">I think that, in summary, I'm pretty happy with my current distribution choice of Debian. I think that of the distributions I've looked at writing this post, it meets my requirements for security support, and release lifecycle. The availability of packages is very good (in my opinion) and it's really pretty easy to roll your own packages.</span>Andrewhttp://www.blogger.com/profile/00624839922000831199noreply@blogger.com0tag:blogger.com,1999:blog-9184095719495587995.post-53151467055382675412010-10-29T16:42:00.004+01:002010-10-29T16:49:15.191+01:00Debugging Mahara<span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif; font-size: small;">As I've mentioned before, I do quite a bit of development work on Mahara.<br />
There are some really handy debugging features which it's worth knowing about.<br />
</span><br />
<span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif; font-size: small;"><br />
<b>Configuration</b><br />
As Mahara ships, debug, info, warn an environ error messages are sent to your error log.<br />
Environ message are also sent to screen.</span><br />
<span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif; font-size: small;"></span><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">The default settings for logging are in lib/config-defaults.php under the <i>Logging Configuration</i> section.</span><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
I tend to have my log levels set to LOG_TARGET_SCREEN | LOG_TARGET_ERRORLOG as below:</span><br />
<span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif; font-size: small;"><pre class="brush: shell">$cfg->log_dbg_targets = LOG_TARGET_SCREEN | LOG_TARGET_ERRORLOG;
$cfg->log_info_targets = LOG_TARGET_SCREEN | LOG_TARGET_ERRORLOG;
$cfg->log_warn_targets = LOG_TARGET_SCREEN | LOG_TARGET_ERRORLOG;
$cfg->log_environ_targets = LOG_TARGET_SCREEN | LOG_TARGET_ERRORLOG;
</pre></span><br />
<br />
<span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><b>Functions</b></span><br />
<span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif; font-size: small;">You can then use the various log functions to log anything. Variables are printed in a sane format which is easy to read. It's really quite a breath of fresh air when you compare it to something as primitive as var_dump:<br />
</span><br />
<span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif; font-size: small;"><pre class="brush: shell">log_debug('hello');
log_info($object);
log_warn($othervar);
log_environ($yetanothervar);
</pre><br />
<b>Notes</b></span><br />
<div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif; font-size: small;">Mahara uses javascript form submission all over the place and if you're trying to debug a form using javascript, the log mesages won't be printed to screen until you next load the page.</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif; font-size: small;">As a result, it's often really helpful to tail the apache error log a lot of the time.</span></div>Andrewhttp://www.blogger.com/profile/00624839922000831199noreply@blogger.com0tag:blogger.com,1999:blog-9184095719495587995.post-46352517721423102592010-08-20T14:36:00.001+01:002010-08-20T14:36:38.056+01:00The power of git - splitting one file into multiple commits<span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="font-size: small;"><br />
Another really handy thing with Git, which I do use regularly, is it's ability to split lots of changes to the same file into separate logical commits.<br />
<br />
As far as I know, any of the other VCSs I've used in the past haven't supported this and the only sane way I know of doing so would be to copy the file away, and use vimdiff to copy the logical commits one-by-one, committing each in turn.<br />
<br />
Thankfully, git makes this really easy with the git add command.<br />
<br />
<b>Set up an example repository</b><br />
<pre class="brush: shell"># Create a new git repository for this example
cd /tmp/
mkdir git-add-pi
cd git-add-pi/
git init
# Make our first commit
echo "Example line" > example-file
git add example-file
git commit -m "First part of the example file"
</pre><br />
<b>Make some changes to our example file</b><br />
Make a change to the beginning of the file:<br />
<pre class="brush: shell; collapse: false">(echo "Some big sweeping change to the file"; cat example-file) > tmp; mv tmp example-file
</pre>and make a change to the end of the file:<br />
<pre class="brush: shell; collapse: false">echo "Another big sweeping change to the same file" >> example-file
</pre><br />
<b>The magic</b><br />
We actually wanted to split that file into two logical commits. With subversion, that would be a pain, but with git it's really easy with git add --pi:<br />
<br />
<pre class="brush: shell; collapse; false">523 git-add-pi:master> git add -pi example-file
diff --git a/example-file b/example-file
index d503a0c..82d4c17 100644
--- a/example-file
+++ b/example-file
@@ -1 +1,3 @@
+Some big sweeping change to the file
Example line
+Another big sweeping change to the same file
Stage this hunk [y/n/a/d/s/?]?
</pre><br />
Because the lines are so close to one another, git hasn't automatically split them up. Specify <b>s</b> to split them:<br />
<pre class="brush: shell; collapse; false">Stage this hunk [y/n/a/d/s/?]? s
Split into 2 hunks.
@@ -1 +1,2 @@
+Some big sweeping change to the file
Example line
Stage this hunk [y/n/a/d/j/J/?]?
</pre>Well, we want to commit this hunk, so hit y<br />
<pre class="brush: shell; collapse; false">Stage this hunk [y/n/a/d/j/J/?]? y
@@ -1 +2,2 @@
Example line
+Another big sweeping change to the same file
Stage this hunk [y/n/a/d/K/?]? n
</pre>We didn't want this hunk as part of this logical commit, so choose n<br />
<br />
<b>Checking what we've done</b><br />
We can double check what we've added so far:<br />
<pre class="brush: shell; collapse; false">524 git-add-pi:master> git diff --cached
diff --git a/example-file b/example-file
index d503a0c..8bca897 100644
--- a/example-file
+++ b/example-file
@@ -1 +1,2 @@
+Some big sweeping change to the file
Example line
</pre>And commit as normal:<br />
<pre class="brush: shell; collapse; false">525 git-add-pi:master> git commit -m "First change"
Created commit ad01f32: First change
1 files changed, 1 insertions(+), 0 deletions(-)
</pre><br />
<b>What about the rest of the file?</b><br />
You can repeat the add -pi as much as you like, or since we only have one more line to commit:<br />
<pre class="brush: shell; collapse; false">526 git-add-pi:master> git add .
git commi527 git-add-pi:master> git commit -m "final commit"
Created commit 16722ed: final commit
1 files changed, 1 insertions(+), 0 deletions(-)
</pre></span></span>Andrewhttp://www.blogger.com/profile/00624839922000831199noreply@blogger.com1tag:blogger.com,1999:blog-9184095719495587995.post-59614819576815105162010-08-19T21:31:00.007+01:002010-08-20T14:00:14.126+01:00The power of git - splitting a commit<span class="Apple-style-span" style="font-size: small;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
Every blog has to have a customary first post - or at least, almost every blog seems to. Rather than mine just saying that I'm starting a new blog, and I'll try and keep it up to date and blah blah blah, I thought I'd actually post some content.<br />
<br />
For the past 9 months or so, I've been fairly heavily involved in an open source project called Mahara. Mahara is an ePortfolio system and I'm sure that I'll post about it more in the near future so I won't go into lots of detail now. Needless to say though, the work I've been doing on it has involved a variety of customisations and any sane developer stores, manages and tracks their work using a version control system or VCS. The mahara project uses git.<br />
<br />
Git is probably the most powerful version control system I've come across and enables you to manage your source and commits with minimal effort and confusion - it's probably the only VCS I've ever used which works for you rather than making you work around it. That said, some of the really powerful features of git take a bit of understanding.<br />
<br />
Earlier today I was telling someone how you can use git in some really cool and interesting ways. One of the things I do frequently is to take a commit and split it into a series of logical commits and I thought that I'd share this.<br />
<br />
<b>Setting up a quick git repository</b><br />
If you want to try and follow with my examples, I've pushed my sample repository to gitorious.org. You can skip the first part by cloning it with:<br />
<pre class="brush: bash">git clone git://gitorious.org/thamblings/git-split-commits.git
</pre><br />
Say that you've created a quick git repository with a few files in it:<br />
<pre class="brush: bash"># Create a new git repository
cd /tmp
mkdir git-example
cd git-example/
git init
# Commit the first file
echo "First File" >> one
git add one
git commit -m "First file committed"
# Now add a few more commits/files
echo "Second File" >> two
echo "Third File" >> three
git add .
git reset
git add two
git commit -m "Second File"
git add three
git commit -m "Third File"
# Now add two files in the same commit
echo "Fourth File" >> four
echo "Fifth File" >> five
git add .
git commit -m "Fourth and Fifth Files"
# And another file
echo "Sixth File" >> six
git add .
git commit -m "Sixth File"
# Let's see what we've done
git log
</pre><br />
<b>But we made a mistake...</b><br />
But wait a second - files four and five should have been in different files. Let's split them out.<br />
<br />
<b>Let's rewrite history</b><br />
First we'll check out a new branch, and then we'll do an interactive rebase to edit the commit:<br />
<pre class="brush: shell">git checkout -b fixcommits
git rebase -i HEAD~2
</pre><br />
We should get something like:<br />
<pre class="brush: shell">pick 0d3abd7 Fourth and Fifth Files
pick feaeea8 Sixth File
</pre><br />
We want to edit commit 0d3abd7, so change that to an edit - you can use 'e' instead:<br />
<pre class="brush: shell">edit 0d3abd7 Fourth and Fifth Files
pick feaeea8 Sixth File
</pre><br />
That takes us to the point after the commit 'Fourth and Fifth Files' and before 'Sixth File'.<br />
We can then use git reset the state of our current HEAD.<br />
<pre class="brush: shell">git reset HEAD~1
</pre><br />
We can then re-add and re-commit each file:<br />
<pre class="brush: shell">git add four
git commit -m "Fourth File"
git add five
git commit -m "Fifth File"
</pre><br />
We can then continue our rebase and we'll be left with our finished article:<br />
<pre class="brush: shell">git rebase --continue
</pre><br />
<b>Let's get things back to master</b><br />
And if we're happy with what we have, we can check it back in to our master branch with another rebase<br />
<pre class="brush: shell">git checkout master
git rebase fixcommits
</pre><br />
The same also works for other git additions. So you could add parts of a file with an interactive git add (git add -pi) for example.<br />
<br />
</span></span>Andrewhttp://www.blogger.com/profile/00624839922000831199noreply@blogger.com1