All posts by mark

Migrating YourNextRepresentative from PopIt to django-popolo

This post was originally intended for the mySociety blog, but because of its length and technical content people suggested to me it might be more suitable for my own blog. This about the development of the YourNextRepresentative project (GitHub link), and in particular why and how we migrated its storage system from using the PopIt web service to instead use PostgreSQL via django-popolo. Hopefully this might be of interest to those who have been following the development of YourNextRepresentative (previously YourNextMP).


YourNextMP for 2015

For the 2015 General Election in the UK, we developed a new codebase to crowd-source the candidates who were standing in every constituency; this site was YourNextMP. (Edmund von der Burg kindly let us use the domain name, which he’d used for his site with the same intent for the 2010 election.) We were developing this software to help out Democracy Club, who ran the site and built up the community of enthusiastic contributors that made YourNextMP so successful.

At the time, we saw this crowd-sourcing challenge as a natural fit for another technology we had been developing called PopIt, which was a data store and HTTP-based API for information about people and the organizations they hold positions in. PopIt used the Popolo data model, which is a very carefully thought-out specification for storing open government data. The Popolo standard helps you avoid common mistakes in modelling data about people and organisations, and PopIt provided interfaces for helping people to create structured data that conformed to that data model, and also made it easily available.

The original intention was that YourNextMP would be quite a thin front-end for PopIt but, as development progressed, but it became clear that this would provide a very poor user experience for editors. So, the YourNextMP wrapper, even in the very early versions, had to provide a lot of features that weren’t available in PopIt, such as:

  • A user experience specific to the task of crowd-sourcing election candidate data, rather than the very generic and unconstrained editing interface of PopIt’s web-based UI.
  • Versioning of information about people, including being able to revert to earlier versions.
  • Lookup of candidates by postcode.
  • Summary statistics of progress and completion of the crowdsourcing effort.
  • Logging of actions taken by users, so recent changes could be tracked and checked.
  • CSV export of the data as well as the JSON based API.
  • etc. etc.

Later on in development we also added more advanced features such as a photo moderation queue. To help support all of this, the YourNextMP front-end used a traditional RDBMS (in our case usually PostgreSQL), but effectively PopIt was the primary data store, which had all the information about people and the posts they were standing for.

This system worked fine for the general election, and I think everyone considered the YourNextMP project to be a great success  – we had over a million unique users, and the data was extensively reused, including by Google for their special election information widget. (You can see some more about the project in this presentation.)

Turning YourNextMP into YourNextRepresentative

There had been a considerable demand to reuse the code from YourNextMP for other elections internationally, so our development efforts then focussed on making the code reusable for other elections. The key parts of this were:

  • Separating out any UK-specific code from the core application
  • Supporting multiple elections generically (the site was essentially hard-coded to only know about two elections – the 2010 and 2015 UK general elections)
  • Internationalizing all text in the codebase, so that it could be localized into other languages.

We worked on this with the target of supporting a similar candidate crowd-sourcing effort for the general election in Argentina in 2015, which was being run by Congreso Interactivo, Open Knowledge Argentina and the Fundación Legislativo. This new version of the code was deployed as part of their Yo Quiero Saber site.

Since the name “YourNextMP” implies that the project is specific to electoral systems with  Members of Parliament, we also changed the name to YourNextRepresentative. This name change wasn’t just about international re-use of the code – it’s going to be supporting the 2016 elections in the UK as well, where the candidates won’t be aspiring to be MPs, but rather MSPs, MLAs, AMs, mayors, PCCs and local councillors.

Problems with PopIt

It had become apparent to us throughout this development history that PopIt was creating more problems than it was solving for us. In particular:

  • The lack of foreign keys constraints between objects in PopIt, and across the interface between data in PostgreSQL and PopIt meant that we were continually having to deal with data integrity problems.
  • PopIt was based on MongoDB, and while it used JSON schemas to constrain the data that could be stored in the Popolo defined fields, all other data you added was unconstrained. We spent a lot of time of time writing scripts that just fixed data in PopIt that had been accidentally introduced or broken.
  • PopIt made it difficult to efficiently query for a number of things that would have been simple to do in SQL. (For example, counts of candidates per seat and per party were calculated offline from a cron-job and stored in the database.)
  • Developers who wanted to work on the codebase would have to set up a PopIt locally or use our hosted version; this unusual step made it much more awkward for other people to set up a development system compared to any conventional django-based site.
  • PopIt’s API had a confusing split between data that was available from its collection-based endpoints and the search endpoints; the latter meant using Elasticsearch’s delightfully named “Simple Query String Query” which was powerful but difficult for people to use.
  • It was possible (and common) to POST data to PopIt that it could store in MongoDB but couldn’t stored in Elasticsearch, but no error would be returned. This long-standing bug meant that the results you got from the collections API (MongoDB-backed) and search API (Elasticsearch-backed) were confusingly inconsistent.
  • The latency of requests to the API under high load meant we had to have a lot of caching. (We discovered this on the first leaders’ debate, which was a good indication of how much traffic we’d have to cope with.) Getting the cache invalidation correct was tricky, in accordance with the usual aphorism.

At the same time, we were coming to the more broad conclusion that the PopIt project hadn’t been achieving the goals that we’d hoped it would, despite having putting a lot of development time into it, and we decided to stop all development on PopIt. (For many applications of PopIt we now felt the user need was better served by the new EveryPolitician project, but in YourNextRepresentative’s case that didn’t apply, since EveryPolitician only tracks politicians after they’re elected, not when they’re candidates.)

As developers we wanted to be able to use a traditional RDBMS again (through the Django ORM) while still getting the benefits of the carefully thought-out Popolo data model. And, happily, there was a project that would help use to do exactly that – the django-popolo package developed by openpolis.

Migrating YourNextRepresentative to django-popolo

django-popolo provides Django models which correspond to the Popolo Person, Organisation, Membership, Post and Area classes (and their associated models like names and contact details).

We had used django-popolo previously for SayIt, and have been maintaining a fork of the project which is very close to the upstream codebase, except for removing the requirement that one uses django-autoslug for managing the id field of its models. We opted to use the mySociety fork for related reasons:

  • Using the standard Django integer primary key id field rather than a character-based id field seems to be closer to Django’s “golden path”.
  • There are interesting possibilities for using SayIt in a YourNextRepresentative site (e.g. to capture campaign speeches, or promises) and using the same version of django-popolo will make that much easier

Extending django-popolo’s models

Perhaps the biggest technical decision about how to use the django-popolo models (the main ones being Person, Organization, Post and Membership) is how we extended those models to add the various extra fields we needed for YourNextRepresentative’s use case. (With PopIt you could simply post a JSON object with extra attributes.) The kinds of extra-Popolo data we recorded were:

  • The one or many elections that each Post is associated with.
  • The particular election that a Membership representing a candidacy is associated with.
  • The set of parties that people might stand for in a particular Post. (e.g. there’s a different set of available parties in Great Britain and Northern Ireland).
  • The ‘versions’ attribute of a Person, which records all the previous versions of that person’s data. (We considered switching to a versioning system that’s integrated with Django’s ORM, like one of these, but instead we decided to just make the smallest incremental step as part of this migration, which meant keeping the versions array and the same JSON serialization that was used previously, and save switching the versioning system for the future.
  • Multiple images for each Person and Organization. (django-popolo just has a single ‘image’ URL field.)
  • Whether the candidates for a particular Post are locked or not. (We introduced a feature where you could lock a post once we knew all the candidates were correct.)
  • To support proportional representation systems where candidates are elected from a party list, each Membership representing a candidacy needs a “party_list_position” attribute to indicate that candidate’s position on the party list.
  • etc.

Perhaps the most natural way of adding this data would be through multi-table inheritance; indeed, that is how SayIt uses django-popolo. However, we were wary of this approach because of the warnings in Two Scoops of Django and elsewhere that using multi-table inheritance can land you with difficult performance problems because queries on the parent model will use OUTER JOINs whether you need them or not. We decided instead to follow the Two Scoops of Django suggestion and make the one-to-one relationship between parent and child table explicit by creating new models called PersonExtra, PostExtra, etc. with a `base` attribute which is a OneToOneField to Person, Post, etc., respectively. This means that the code that uses these models is slightly less clear than it would be otherwise (since sometimes you use person, sometimes person.extra) but we do have control over when joins between these tables are done by the ORM.

Data migration

Once the Extra models were created, we wrote the main data migration. The idea of this was that if your installation of YourNextRepresentative was configured as one of the known existing installations at the time (i.e. the ELECTION_APP setting specified the St Paul, Burkina Faso, Argentina or the UK site) this data migration would download a JSON export of the data from the corresponding PopIt instance and load it into the django-popolo models and the *Extra models that extend them.

As the basis for this migration, we contributed a PopIt importer class and management command upstream to django-popolo. This should make it easier for any project that used to use PopIt to migrate to django-popolo, if it makes sense for them to do so. Then the data migration in YourNextRepresentative subclassed the django-popolo PopItImporter class, extending it to also handle the extra-Popolo data we needed to import.

(A perhaps surprising ramification of this approach is that once an installation has been migrated to django-popolo we should change the name of that country’s ELECTION_APP, or otherwise someone setting up a new site with that ELECTION_APP configured will have to wait for a long time for out-of-date data to be imported on running the initial migrations. So we will shortly be renaming the “uk_general_election_2015” application to just “uk”. To support people who want that feature (cloning an existing site to a development instance) we’ve added a new “candidates_import_from_live_site” management command that uses its new API to mirror the current version of an existing instance.)

Another issue that came up in working on this data migration is that we needed to preserve any identifiers that were used in URLs on the site previously so that after upgrading each site every bookmarked or search-engine-indexed URL would still show the same content. In the case of the Person model, this was easy because it used an integer ID previously. Parties and posts, however, used strings as their IDs. We migrated these IDs to fields called ‘slug’ (perhaps a slightly misleading name) on the OrganisationExtra and PostExtra models.

This turns out to be quite a slow migration to run – as well as importing the core data, it also downloads every image of people and parties, which is pretty slow even on a fast connection.

Updating views, tests and management commands

The next part of the migration was updating all the code that previously used PopIt’s API to instead use the new Django models. This was a significant amount of work, which left very little code in the project unchanged by the end. In general we tried to update a test at a time and then change the core code such that the test passed, but we knew there was quite a bit of code that wasn’t exercised by the tests. (One nice side-effect of this work is that we greatly improved the coverage of the test suite.)

We did think about whether we could avoid doing this update of the code essentially in one go – it felt rather like a “stop-the-world refactoring”; was there an incremental approach that would have worked better? Maybe so, but we didn’t come up with one that might reasonably have saved time. If the old code that interacted with the PopIt API had been better encapsulated, perhaps it would have made sense to use proxy models which in the migration period updated both PopIt’s API and the database, but this seemed like it would be at least as much work, and it was lucky that we did have a period of time we could set aside for this work during which the sites weren’t being actively updated.

Moving other code and configuration to the database

We also took the opportunity of this migration to introduce some new Django models for data that had previously been defined in code or as Django settings. In particular:

  • We introduced Election and AreaType models (previously the elections that a site supported and the types of geographical boundary they were associated with were defined in a dictionary in the per-country settings).
  • We introduced a PartySet model – this is to support the very common requirement that different sets of parties can field candidates in different areas of the country.
  • We replaced the concept of a “post group” (definied in code previously) with a “group” attribute on PostExtra

These all had the effect of simplifying the setup of a new site – more of the initial setup can now be done in the Django admin interface, rather than needing to be done by someone who’s happy to edit source code.

Replacing PopIt’s API

One of the nice aspects of using PopIt as the data store for the site was that it supplied a RESTful API for the core data of the site, so we previously hadn’t had to put much thought into the API other than adding some basic documentation of what was there already. However, after the migration away from PopIt we still wanted to provide some HTTP-based API for users of the site. We chose to use Django REST framework to provide this; it seems to be the most widely recommended tool at the moment for providing a RESTful API for a Django site (and lots of people and talks at djangocon EU in Cardiff had independently recommended it too). Their recommendations certainly weren’t misplaced – it was remarkably quick to add the basic read-only API to YourNextRepresentative. We’re missing the more sophisticated search API and various other features that the old PopIt API provided, but there’s already enough information available via the API to completely mirror an existing site, and Django REST framework provides a nice framework for extending the API in response to developers’ needs.

The end result

As is probably apparent from the above, this migration was a lot of work, but we’re seeing the benefits every day:

  • Working on the codebase has become a vastly more pleasant experience; I find I’m looking forward to it working on it much more than I ever did previously.
  • We’ve already seen signs that other developers appreciate that it’s much more easy to set up than previously.
  • Although the tests are still far from perfect, they’re much more easy to work with than previously. (Previously we mocked out a large number of the external PopIt API requests and returned JSON from the repository instead; this would have been a lot better if we’d used a library like betamax instead to record and update these API responses, but not having to worry about this at all and just create test data with factory_boy is better yet, I think.)

I think it’s also worth adding a note that if you’re starting a new site that deals with data about people and their memberships of organizations, then using the Popolo standard (and if you’re a Django developer, django-popolo in particular) can save you time – it’s easy to make mistakes in data modelling in this domain (e.g. even just related to names). It should also help with interoperability with other projects that use the same standard (although it’s a bit more complicated than that – this post is long enough already, though :))

The UK instance of YourNextRepresentative (at has been using the new codebase for some time now, and that will be relaunched shortly to collect data on the 2016 elections in the UK.

If you’re interested in starting up an candidate crowd-sourcing site using YourNextRepresentative, please get in touch with our international team or email for any other issues.

Printing out GitHub issues for triage or estimation

On one of the projects I’ve been working on at mySociety had a large number of open issues, most of which hadn’t been looked at for some time, and didn’t have estimates of difficulty. To address this we tried a variation of an exercise that’s suggested in “The Scrum Field Guide” in the chapter called “Prioritizing and Estimating Large Backlogs”, which involves printing out all the stories in the product backlog onto index cards and arranging them on a wall roughly in order of difficulty from left to right. (A second stage is to let the product owner adjust the height of each story on the wall to reflect its priority.) It seemed as if this might be a helpful way of estimating our large number of open issues, even though they’re not written as users stories.

An obvious problem here was that we didn’t have an easy way of GitHub tickets en masse, so I wrote a script that would generate a PDF for each open issue (excluding those that are pull requests) in a repository’s issue tracker using Pandoc. That script is here in case it’s of use to anyone:

As it turned out, this session was pretty successful – as well as the value of generating lots of difficulty estimates, it was a good way of getting a picture of the history of the project, and starting lots of discussions of issues that only one member of the team knew about.

Cryptic Crossword for Sarah

This is a cryptic crossword that Jenny and I wrote for my sister for her birthday, and I thought it might be worth putting up here (with her permission) in case anyone wants to give it a try. Many of the answers were chosen because the word might have some particular meaning for her, but I think they should all be generally accessible words. 7a relies on a reference which may be unfamiliar to some, however. The intention was to make the clues pretty easy, since she’s a relatively new solver, and the grid we ended up with to fit in all the themed words is quite a tough one.

Here’s a link to a PDF of the crossword.

Books in 2012

This is a rough list of the books I read in 2012 with some brief comments – I’ve seen other people do this on their blogs and enjoyed reading their summaries, so thought that I would have a go. (Originally I added a mention of each person who recommended one of these to me, but that turned out to be problematic, both in privacy and completeness terms – let me just say that I’m always grateful for the wonderful recommendations I get from friends and colleagues.)

Fear and Loathing in Las Vegas – Hunter S. Thompson

I’ve read it at least twice before – this time I didn’t feel the momentum of the writing bowling me along quite as it did when I first read it, but that’s probably to be expected.

Bury the Chains – Adam Hochschild

An excellent account of the movement in Britain to abolish slavery. It’s still very relevant today if you’re interested in activism and campaigning, and, as the author points out, it’s the story of the first mass movement where a group campaigned for the rights of people other than themselves.

Mort – Terry Pratchett

This is the first time I’d re-read it in many years, although I usually suggest it (as do many people) as the first Terry Pratchett novel to try if you’ve never read any of his Discworld novels before.  I was glad to find still every bit as enjoyable as the first time. (At the end of the previous year I’d finished the Discworld series, after not reading any of them roughly between the ages of 18 and 32.)

Mockingjay – Suzanne Collins

I’d listened to the first two of the Hunger Games novels as audiobooks when I still had Emusic‘s “one audiobook a month” deal. It’s worth finishing the trilogy if you’ve read the first two, which I also enjoyed despite finding the character of Katniss terribly frustrating (as is probably intended).  The nature of the resistance movement is interesting, and I found the conclusion satisfying.

Knots and Crosses – Ian Rankin

Hide And Seek – Ian Rankin

Tooth and Nail – Ian Rankin

Strip Jack – Ian Rankin

The Black Book – Ian Rankin

Mortal Causes – Ian Rankin

Let It Bleed – Ian Rankin

Black and Blue – Ian Rankin

The Hanging Garden – Ian Rankin

Dead Souls – Ian Rankin

Set in Darkness – Ian Rankin

The Falls – Ian Rankin

These are the first 12 of Ian Rankin’s Rebus series of detective novels, and I’ll probably read the last few of them in 2013. I found them quick to read, and I loved the portrait of Edinburgh – it’s much more true to my experiences of the city both as a child and an adult than virtually any other books I’ve read that are set there.  I found the character of Rebus compelling as well.  The first two novels (Knots and Crosses and Hide and Seek) have some points where the writing really jarred for me, but from the third onwards they’re consistently well-written.

The Player of Games – Iain M. Banks

I seem to re-read this about once a year – it’s one of my favourite of Iain M. Banks’ novels (along with Consider Phlebas and Excession). I’ve actually stopped reading his new books, the last straw being The Steep Approach to Garbadale, where I spent most of the book thinking “oh, please don’t let the twist be the one that I think is coming, [a disturbing theme from several previous novels]”, and of course it was…

The Open-Focus Brain – Les Fehmi

This was a kind present from a student in Zürich who was concerned about my style of concentration. The idea of the book is that one can switch from a very intense “fight or flight” style of concentration to something the author describes as “open focus” by using various techniques, in particular thinking about volumes of space. It comes with a CD with some example exercises. When I’ve tried this it’s certainly been relaxing – I hadn’t tried any form of meditation before. It’s also interesting that thinking about the space around you is also something that’s used in the Alexander technique and (I’m told) various meditation techniques. Unfortunately, the book is written in a self-help style that makes me rather suspicious of it, particularly the wide-ranging claims for its health benefits sometimes based on single cases or unpublished data.

Moonwalking with Einstein – Joshua Foer

A book about memory champions and memory techniques of the “a journalist tries to become really good at something relatively obscure” genre (see also “Word Freak”, “Born To Run”, etc.) It’s a quick read, and if you’re interested in the subject of memory, or building expertise Anders Ericsson-style, worth a look.  In a strange coincidence, 6 months after I read this, my partner started work at a company run by one of the memory champions who appears in the book.

Why Have Kids? – Jessica Valenti

This probably isn’t what it sounds like from the title – it’s an excellent discussion of the absurd pressures on mothers (particularly in US culture) to meet impossible standards. It was written by Jessica Valenti after having her first child.

The Revolution will be Digitised – Heather Brooke

An interesting account of the Wikileaks saga from Heather Brooke, who is a fantastic campaigner on freedom of information issues. The most extraordinary sections deal with Julian Assange’s interactions with her, including his advances wherein he identifies himself with Jesus…

Born to Run – Christopher McDougall

An entertaining book about ultra-runners, barefoot-style running and the Tarahumara people, who run huge distances without apparently being susceptible to the injuries that plague runners elsewhere. It’s great in terms of adventure and storytelling, with lovely portraits of the runners the author meets. I wouldn’t read it if you’re looking for rigorous science and anthropology, but it’s excellent fun.

Rocket Surgery Made Easy – Steve Krug

I generally haven’t added work-related books to this list, but since I read this one straight through, it seemed as if it fitted better than then others which I tended to dip into and out-of more. This is by Steve Krug as a follow-up to “Don’t Make Me Think”, and aims to give you step-by-step guidance for running DIY usability testing sessions of websites.  I found the style somewhat irritating (as so often where technical books try for a light touch) but in practice it was very helpful for running some usability testing last year.  I’m hoping I’ll get to do that again next year.

The House of Silk – Anthony Horowitz

A new Sherlock Holmes novel by Anthony Horowitz. It’s a disturbing book, but brilliantly matches the style of the Holmes stories, I thought.

Plan of Attack – Bob Woodward

Bob Woodward’s book about the decisions that led to the invasion of Iraq in 2003 – I found this frankly terrifying.

The Sense of an Ending – Julian Barnes

The 2011 Booker prize winner – consummately well-written and an excellent read.

The Narrative of John Smith – Arthur Conan Doyle

The unpublished first novel of Arthur Conan Doyle. I found it tiresome, and not terribly interesting – I wouldn’t recommend it unless, I suppose, you’re a Conan Doyle completist.

Catherine of Aragon: Henry’s Spanish Queen – Giles Tremlett

A sympathetic biography of Catherine of Aragon – embarrassingly, I know virtually nothing of Tudor history, so this wasn’t just interesting, but almost entirely novel for me…

The Strange Case of Dr Jekyll and Mr Hyde – Robert Louis Stevenson

I’d forgotten how short this story is, in fact, but it’s still hugely enjoyable. It’s tense throughout, and I’d forgotten about the interesting character of the narrator, a quiet man who’s happy sitting in silence by the fire with his old friends.

Kidnapped – Robert Louis Stevenson

I don’t think that I’d ever read the original text of Kidnapped before, just a children’s edition and later a graphic novel version. Anyway, it’s still a great adventure and Balfour’s journey crosses lots of parts of Scotland that I know from past walking holidays.

The Ascent of Rum Doodle –  W. E. Bowman

An excellent comic novel from the 1950s about a group of mountaineers attempting to climb the fictional mountain “Rum Doodle” with the help of thousands of porters and a supply of Champagne for “medicinal purposes”. Very silly, and highly recommended.

Care of Wooden Floors – Will Wiles

This is the first novel by a friend I knew at college. I enjoyed this very much – I’ve been accused of giving away too much when discussing it before, so I’ll just say that it’s hilarious and agonising in equal parts, and definitely worth reading.

Wolf Hall – Hilary Mantel

Another Booker prize winner; my ignorance of history probably meant that I missed quite a lot of pleasure in associating the wonderfully drawn characters with their actual stories, but I enjoyed it nonetheless. The (much discussed) style of the writing, in particular the use of the pronoun “he”, initially irritated me a great deal, but I got used to it after a hundred pages or so. I really did think that Wolf Hall was good, but I don’t really see why so many people are quite so passionate about it.

The Afterparty – Leo Benedictus

Another novel written by someone that I used to know, and happily another great read. I found this story of a naive journalist caught up in events around a celebrity party completely gripping, and I enjoyed the meta-narrative too – it’s a smart and very well-written novel.

How To Be A Woman – Caitlin Moran

An often hilarious book – there are several sections that had both of us laughing out loud. The subject matter’s also great – I think it’s essentially aimed at younger people who would say that they’re in favour of equal rights for women, but bizarrely would also say that they’re not feminists.

The Blank Wall – Elisabeth Sanxay Holding

Elisabeth Sanxay Holding was an author that Raymond Chandler regarded as “the top suspense writer of them all”, but who is relatively unread. (You can get it from Persephone Books.) This is the story of a housewife during WWII who is trying to protect her family from events that are going out of control, a bit like Brief Encounter crossed with hardboiled detective fiction – I enjoyed it very much.

Quiet – Susan Cain

This is probably the most personally and professionally relevant book I read this year, as someone who’s introverted but works hard to not give that impression.  It’s about the ways in which society increasingly values the qualities we associate with extroversion rather than introversion, and why that might be a mistake. I found it encouraging and thought-provoking – I’d recommend it to anyone who’s quiet, introverted or sensitive (the book deals with several associated personality traits). There’s also a little material towards the end about how these traits can affect relationships which struck quite a few chords with me.

The most confusing git terminology

To add my usual disclaimer to the start of these blog posts, I should say that I love git; I think it’s a beautiful and elegant system, and it saves me huge amounts of time in my daily work. However, I think it’s a fair criticism of the system that its terminology is very confusing for newcomers, and in particular those who have come from using CVS or Subversion.

This is a personal list of some of my “favourite” points of confusion, which I’ve seen arise time and time again, both in real life and when answering questions on Stack Overflow. To be fair to all the excellent people who have contributed to git’s development, in most cases it’s clear that they are well aware that these terms can be problematic, and are trying to improve the situation subject to compatibility constraints. The problems that seem most bizarre are those that reuse CVS and Subversion terms for completely different concepts – I speculate a bit about that at the bottom.


If you’ve used Subversion or CVS, you’re probably used to “update” being a command that goes to the remote repository, and incorporates changes from the remote version into your local copy – this is (very broadly) analogous to “git pull”.  So, when you see the following error message when using git:

foo.c: needs update

You might imagine that this means you need to run “git pull”. However, that’s wrong.  In fact, what “needs update” means is approximately: “there are local modifications to this file, which you should probably commit or stash”.

“track” and “tracking”

The word “track” is used in git in three senses that I’m aware of. This ambiguity is particularly nasty, because the latter two collide at a point in learning the system where newcomers to git are likely to be baffled anyway. Fortunately, this seems to have been recognized by git’s developers (see below).

1. “track” as in “untracked files”

To say that a file is tracked in the repository appears to mean that it is either present in the index or exists in the commit pointed to by HEAD.  You see this usage most often in the output of “git status”, where it will list “untracked files”:

# On branch master
# Untracked files:
#   (use "git add <file>..." to include in what will be committed)
#    .classpath

This sense is relatively intuitive, I think – it was only after complaining for a while about the next two senses of “track” that I even remembered that there was also this one :)

2. “track” as in “remote-tracking branch”

As a bit of background, you can think of a remote-tracking branch as a local cache of the state of a branch in a remote repository.  The most commonly seen example is origin/master, or, to name that ref in full, refs/remotes/origin/master.  Such branches are usually updated by git fetch (and thus also potentially by git pull).  They are also updated by a successful push to the branch in the remote repository that they correspond to.   You can merge from them, examine their history, etc. but you can’t work directly on them.

The sense of “track” in the phrase “remote-tracking branch” is indicating that the remote-tracking branch is tracking the state of the branch in the remote repository the last time that remote-tracking branch was updated.  So, you might say that refs/remotes/origin/master is tracking the state of the branch master in origin.

The “tracking” here is defined by the refspec in the config variable remote.<remote-name>.fetch and the URL in the config variable remote.<remote-name>.url.

3. “track” as in “git branch –track foo origin/bar” and “Branch foo set up to track remote branch bar from origin”

Again, if you want to do some work on a branch from a remote repository, but want to keep your work separate from everything else in your repository, you’ll typically use a command like the following (or one of its many “Do What I Mean” equivalents):

git checkout --track -b foo origin/bar

… which will result in the following messages:

Branch foo set up to track remote branch bar from origin
Switched to a new branch 'foo'

The sense of “track” both in the command and the output is distinct from the previous sense – it means that config options have been set that associate your new local branch with another branch in the remote repository. The documentation sometimes refers to this relationship as making bar in origin “upstream” of foo. This “upstream” association is very useful, in fact: it enables nice features like being able to just type git pull while you’re on branch foo in order to fetch from origin and then merge from origin/bar. It’s also how you get helpful messages about the state of your branch relative to the remote-tracking branch, like “Your branch foo is 24 commits ahead of origin/bar and can be fast-forwarded”.

The tracking here is defined by config variables branch.<branch-name>.remote and branch.<branch-name>.merge.

“tracking” Summary

Fortunately, the third sense of “tracking” seems to be being carefully deprecated – for example, one of the possible options for push.default used to be tracking, but this is now deprecated in favour of the option name upstream. The commit message for 53c403116 says:

push.default: Rename ‘tracking’ to ‘upstream’

Users are sometimes confused with two different types of “tracking” behavior in Git: “remote-tracking” branches (e.g. refs/remotes/*/*) versus the merge/rebase relationship between a local branch and its @{upstream} (controlled by and config settings).

When the push.default is set to ‘tracking’, it specifies that a branch should be pushed to its @{upstream} branch. In other words, setting push.default to ‘tracking’ applies only to the latter of the above two types of “tracking” behavior.

In order to make this more understandable to the user, we rename the push.default == ‘tracking’ option to push.default == ‘upstream’.

push.default == ‘tracking’ is left as a deprecated synonym for ‘upstream’.


In CVS and Subversion, “commit” means to send your changes to the remote repository. In git the action of committing (with “git commit”) is entirely local; the closest equivalent of “cvs commit” is “git push”. In addition, the word “commit” in git has different verb and noun senses (although frankly I’ve never found this confusing myself). To quote from the helpful git glossary:


As a noun: A single point in the git history; the entire history of a project is represented as a set of interrelated commits. The word “commit” is often used by git in the same places other revision control systems use the words “revision” or “version”. Also used as a short hand for commit object.

As a verb: The action of storing a new snapshot of the project’s state in the git history, by creating a new commit representing the current state of the index and advancing HEAD to point at the new commit.

Strictly speaking that makes two different noun senses, but as a user I’ve rarely found that confusing.


In CVS and Subversion “checkout” creates a new local copy of the source code that is linked to that repository. The closest command in git is “git clone”. However, in git, “git checkout” is used for something completely distinct. In fact, it has two largely distinct modes of operation:

  1. To switch HEAD to point to a new branch or commit, in the usage git checkout <branch>. If <branch> is genuinely a local branch, this will switch to that branch (i.e. HEAD will point to the ref name) or if it otherwise resolves to a commit will detach HEAD and point it directly to the commit’s object name.
  2. To replace a file or multiple files in the working copy and the index with their content from a particular commit or the index. This is seen in the usages: git checkout -- (update from the index) and git checkout <tree-ish> -- (where <tree-ish> is typically a commit).

(git checkout is also frequently used with -b, to create a new branch, but that’s really a sub-case of usage 1.)

In my ideal world, these two modes of operation would have different verbs, and neither of them would be “checkout”.

“HEAD” and “head”

There are usually many “heads” (lower-case) in a git repository – the tip of each branch is a head. However, there is only one HEAD (upper-case) which is a symbolic ref which points to the current branch or commit.

“fetch” and “pull”

I wasn’t aware of this until Roy Badami pointed it out, but it seems that git and Mercurial have opposite meanings for “fetch” and “pull” – see the top two lines in this table of git / hg equivalences. I think it’s understandable that since git’s and Mercurial’s development were more or less concurrent, such unfortunate clashes in terminology might occur.

“push” and “pull”

“git pull” is not the opposite of “git push”; the closest there is to an opposite of “git push” is “git fetch”.

“hash”, “SHA1”, “SHA1sum”, “object name” and “object identifier”

These terms are often used synonymously to mean the 40 characters hexadecimal strings that uniquely identify objects in git. “object name” seems to be the most official, but the least used in general. Referring to an object name as a SHA1sum is potentially confusing, since the object name for a blob is not the same as the SHA1sum of the file.

“remote branch”

This term is only used occasionally in the git documentation, but it’s one that I would always try to avoid because it tends to be unclear whether you mean “a branch in a remote repository” or “a remote-tracking branch”. Whenever a git beginner uses this phrase, I think it’s worth clarifying this, since it can avoid later confusion.

“index”, “staging area” and “cache”

As nouns, these are all synonyms, which all exist for historical reasons. Personally, I like “staging area” the best since it seems to be the easiest concept to understand for git beginners, but the other two are used more commonly in the documentation.

When used as command options, --index and --cached have distinct and consistent meanings, as explained by Junio C. Hamano in this useful blog post.

Why are there so many of these points of confusion?

I would speculate that the most significant effect that contributed to these terminology confusions is that git was being actively used by an enthusiastic community from very early in its development, which means that early names for concepts have tended to persist for the sake of compatibility and consistency. That doesn’t necessarily account for the many conflicts with CVS / Subversion usage, however.

To be fair to git, thinking up verbs for particular commands in any software is tough, and there have been enough version control systems written that to completely avoid clashes would lead to some convoluted choices. However, it’s hard to see git’s use of CVS / Subversion terminology for completely different concepts as anything but perverse. Linus has made it very clear many times that he hated CVS, and joked that a design principle for git was WWCVSND (What Would CVS Never Do); I’m sympathetic to that, as I’m sure most are, especially after having switched to the DVCS mindset. However, could that attitude have extended to deliberately disregarding concerns about terminology that might make it actively harder for people to migrate to git from CVS / Subversion? I don’t know nearly enough about the early development of git to know. However, it wouldn’t have been tough to find better choices for commit, checkout and update in each of their various senses.

Extending the wireless range of a BT Home Hub 2

Update: While the device I discuss below initially seemed to work promisingly as a repeater, we’ve had lots of problems with it – it seems to work fine for a few hours, but then will stop working for no clear reason. At some point I’m going to try reflashing it as suggested in the link below, but I haven’t been in the same house as the router for any appreciable amount of time in the last 6 months. The post below is as I originally wrote it, but please be aware that it hasn’t worked nearly as well as I thought at the time of writing. Update 2:  having revisited the house recently, it seemed to be better having changed the default address of the TP Link device to be different from the BT Home Hub (duh) and outside the latter’s DHCP range, so I’d recommend checking that.

I’m posting this because we found it surprisingly hard to find convincing instructions for setting up a device to extend the range of a wireless router, and I suspect many other people might be in the same position.  In the end, the device we got was very cheap, easy to set up, and it’s a great improvement.

Please bear in mind that I’m just reporting what worked for us – I can’t guarantee the same will work in your situation or provide any technical support.  (I’m pretty ignorant about WiFi, it turns out.)  I imagine that this will work fine with other wireless routers and the BT Home Hub 3, but of course I can’t promise that.

The Problem

We found that one end of my parents’ house had very poor wireless LAN coverage, since for various reasons the BT Home Hub 2 has to be placed at the opposite end of the house and there are plenty of walls and other obstructions.  (For those outside the UK, the BT Home Hub is just an ADSL modem and wireless access point which BT give you when you order their broadband service – it’s quite a basic model and I assume the method described here would work with any wireless access point.)

We first tried a couple of commonly suggested solutions which didn’t involve buying new hardware:

  • Checking that  802.11n was turned on: this seems to be the default on this router, however, which had b, g and n turned on when I checked.
  • Changing the wireless channel to one with less traffic: the home hub actually has an option to pick a new channel automatically, presumably based on the other access points it can detect.  This did produce a slight improvement in range, in fact, but still not enough to cover the complete house.

The Solution

Some searching turned up this suggestion from Keith Beddoe from the BT community forums.  His second suggested option is to set up a particular TP Link wireless access point as a “Universal Repeater”, but suggests reflashing the access point with some files he supplies.  I’m always slightly wary of reflashing devices unless it’s absolutely necessary, and it’s not clear why it is in this case – the manual for the TL-WA701ND says that it can act both as a Repeater and a Universal Repeater.  (The distinction between those two for this device seems to be that the existing router must support WDS in order to use the Repeater option – since the BT Home Hub 2 doesn’t, we want the Universal Repeater option.)

The only steps I needed to take to set up the access point as a range extender were as follows:

  • Find a spot in the house that can still get a good signal from the original access point, but is near to the end of the house that had poor coverage.
  • Plug in the TP Link TL-WA701ND there.
  • Find an ethernet cable, and use that to plug a laptop into the TP Link device.
  • Configure the laptop to have a static IP address of
  • Visit and log in with the default username and password given in the “Quick Installation” guide.
  • Updated: the default IP address of the BT Home Hub we have is also, so it’s a good idea to change the IP address of the TP Link device to, say, – that’s outside the address range that will be allocated via DHCP by the BT Home Hub with its default settings, and helpfully this will also enable you to log into the TP Link device in normal usage.
  • Under the “Wireless” settings, select an operation mode of “Universal Repeater”.
  • Click “Connect” by the existing BT Home Hub.
  • Save those settings – it may want to reboot itself then, so let it do that if so.
  • Now go to “Wireless Security”.
  • Select WPA and enter the key for your existing BT Home Hub.  (If you have an earlier BT Home Hub, you might need to use WEP instead – it should say on the back of the hub where the key is shown.)
  • Save those settings, and I think it will want to reboot again.
  • Just to check that everything is working, now go to “Wireless Statistics”, and you should see that there’s at least one connection – that to the existing access point.
  • The bit that everyone apparently forgets (including me): change my laptop back to using DHCP instead of static IP.

The effect of this is that there are now two access points with the same SSID as the BT Home Hub listed in the output of “iwlist scan”.  The network manager on my laptop only lists it once, though, as does my smartphone.  If I walk with my laptop from one end of the house to the other, it seems to reconnect seamlessly when moving into an area that is only covered by the range extender, and our phones behave similarly.  So, that seems to be ideal and we get a good signal all over the house now.

The price of the TP Link TL-WA701ND is only £23, and this seems like a reasonable price to pay to get wireless LAN in the garden :)

Star Trek: The Next Generation

I’ve had a long-standing affection for Star Trek: The Next Generation, since it used to be shown on TV right after I got home from school and I would watch it almost every day.  I must have seen every episode several times when I was younger, but it was only recently that I went back to rewatch them on DVD, partly out of curiosity to see how they would stand up.  I think it’s fair to say that a lot of these episodes haven’t aged well (or were dreadful even at the time), but there are still plenty of gems there. As much for my own reference as anything else, the table below marks which episodes I still thought were good after a gap of many years.

In the table, I’ve also added links to two series of reviews.  The first are Zack Handlen’s excellent reviews of every ST:TNG episode (linked from the initials “ZH” below), and I’ve also added his grade for each episode.  I’ve found myself agreeing with his reviews more often than not, and in a some cases the reviews are significantly more enjoyable than the episodes themselves.  The second set of links are to reviews by Wil Wheaton (who played Wesley Crusher) of some of the episodes from Season 1 – it’s interesting to hear his take on these very variable episodes, particularly since the writers were not good to him at all.

A few other random observations:

  • It’s often been said, but there are plenty of episodes that are completely carried by Patrick Stewart – he’s just fantastic.
  • Predictably, perhaps, given that I enjoyed Star Trek: Deep Space Nine and Battlestar Galactica, it seems that lots of my favourite episodes were written by Ronald D. Moore.
  • I hated the holodeck episodes when I was younger, and I still think they have no place in the series. There are any number of problems with the holodeck conceit from a science fiction point of view, and it very rarely adds anything to the story-telling.
  • The pastiches of other genres are dreadful, in particular the insultingly awful “Dixon Hill” parodies of Raymond Chandler. As a general rule, if you think you can write something in the style of Raymond Chandler, you have no idea what makes his books good. (See also the ridiculous Robert Altman version of the Long Goodbye.)

These other links might also be of interest if you enjoyed ST:TNG:

Summary Table of Episodes

The source for this table is the spreadsheet here and the HTML was generated from that with this script.

The key to this colours in this table is:

  • One of my favourites
  • An episode I liked
  • An episode I’m still undecided about
  • A quite missable episode

Season 1

1×01, 1×02 Encounter at Farpoint WP: The new starship Enterprise begins her maiden voyage by uncovering the mysteries of an advanced space station. The crew’s mission is threatened by an omnipotent being named Q, who puts them (and specifically, Captain Picard) on trial for the crimes of all humanity. ZH (B-) WW
1×03 The Naked Now WP: A mysterious, communicable contaminant causes the crew to experience symptoms similar to alcohol intoxication. ZH (D-) WW
1×04 Code of Honor WP: Lt. Yar is abducted by the leader of a people who abide by a strict code of honor, which requires her participation in a fight to the death. ZH (C-) WW
1×05 The Last Outpost WP: An unknown force immobilizes the Enterprise during the Federation’s first encounter with a new alien threat—the Ferengi. ZH (C-) WW
1×06 Where No One Has Gone Before WP: Warp efficiency tests send the Enterprise traveling far beyond known space, where the crew’s imagination takes on real form. First appearance of Eric Menyuk as The Traveler. ZH (B-) WW
1×07 Lonely Among Us WP: An alien entity possesses Dr. Crusher, Worf, and Picard while the Enterprise is transporting delegates from two feuding planets. ZH (C-) WW
1×08 Justice WP: Wesley breaks an idyllic world’s trivial law by accidentally stepping on flowers and faces the death sentence. ZH (C) WW
1×09 The Battle WP: A Ferengi captain returns the abandoned Stargazer to its former captain, Jean-Luc Picard. Picard, who experiences severe headaches, begins to relive the “Battle of Maxia” in which he lost the ship. Guest star Frank Corsentino as DaiMon Bok. ZH (C+) WW
1×10 Hide and Q WP: Q returns to the Enterprise to tempt Commander Riker into joining the Q Continuum with the lure of Q’s powers. ZH (C-) WW
1×11 Haven WP: Lwaxana Troi visits her daughter, Counselor Troi, and prepares her for an arranged marriage. ZH (D) WW
1×12 The Big Goodbye WP: A computer malfunction traps Picard, Data, and Beverly in a Dixon Hill holodeck program set in early 20th-century Earth. ZH (B-) WW
1×13 Datalore WP: The Enterprise crew finds a disassembled android identical to Data at the site of the Omicron Theta colony—where Data was found—which was destroyed by a life form dubbed “the Crystalline Entity.” The reassembled android, Lore, brings the Crystalline Entity to the Enterprise. ZH (B-) WW
1×14 Angel One WP: The Enterprise visits a world dominated by women to rescue survivors of a downed freighter. ZH (F) WW
1×15 11001001 WP: Bynars upgrade the Enterprise’s computers in spacedock. Riker and Picard become distracted by a surprisingly realistic holodeck character. ZH (B) WW
1×16 Too Short a Season WP: The Enterprise transports a legendary geriatric admiral who must once again negotiate a hostage situation involving a man from decades earlier in his career. The admiral is mysteriously growing younger; by the time the Enterprise arrives he is a young man. ZH (C+) WW
1×17 When The Bough Breaks WP: A planet formerly existing only in legend uncloaks and requests help from the Enterprise. Planet’s inhabitants are sterile and want to adopt children from the Enterprise—by force, if necessary. ZH (B) WW
1×18 Home Soil WP: The crew of the Enterprise discovers a crystalline lifeform with murderous intelligence that has been killing the scientists on a terraforming project. ZH (B+) WW
1×19 Coming of Age WP: While Wesley takes a Starfleet Academy entrance exam, the senior staff of the Enterprise are placed under investigation by Starfleet. ZH (C) WW
1×20 Heart of Glory WP: Fugitive Klingons seeking battle attempt to hijack the Enterprise, and ask Worf to join them. ZH (B+) WW
1×21 The Arsenal of Freedom WP: Trapped on the surface of an abandoned planet, an away team becomes unwitting participants in the demonstration of an advanced weapons system. ZH (C+) WW
1×22 Symbiosis WP: Picard tries to mediate a trade dispute between two neighboring planets, one of which is the sole supplier of a drug to treat the other’s apparently fatal disease. ZH (B) WW
1×23 Skin of Evil WP: An evil, tar-like creature holds Troi hostage on an alien world. During the rescue mission, one of the Enterprise crew is killed. ZH (C-) WW
1×24 We’ll Always Have Paris WP: Picard meets an old flame, whose husband has been affected by a dimensional experiment accident. ZH (B-) WW
1×25 Conspiracy WP: The strange behavior of high-ranking officers—which earlier prompted the investigation of the crew (in “Coming of Age”)—leads Picard to uncover a conspiracy within Starfleet. ZH (B) WW
1×26 The Neutral Zone WP: A derelict satellite is found containing cryonically frozen humans from the 21st century as the Enterprise is sent to investigate the destruction of outposts near Romulan space. ZH (C-) WW

Season 2

2×01 The Child WP: Dr. Pulaski joins the Enterprise while La Forge prepares the Enterprise to transport dangerous plague specimens; Deanna spontaneously becomes pregnant and gives birth to a mysterious child. ZH (D+) WW
2×02 Where Silence Has Lease WP: The Enterprise becomes trapped in a “hole in space”, where the crew encounter strange spatial phenomena and crewless ships materialising in and out of existence. The crew suspect they are akin to lab rats. ZH (B) WW
2×03 Elementary, Dear Data WP: After Data easily solves an ordinary Sherlock Holmes holodeck mystery, Geordi asks the computer to make a Holmes villain capable of defeating Data. The resultant Professor Moriarty soon becomes far more powerful than expected. ZH (B+) WW
2×04 The Outrageous Okona WP: The Enterprise is caught up in the schemes of a flamboyant space rogue on the run, while Data explores humor with the help of a holodeck comedian (played by Joe Piscopo). ZH (C-) WW
2×05 Loud As A Whisper WP: The Enterprise hosts a deaf, telepathic ambassador who mediates difficult peace negotiations with the assistance of his trio of telepathic interpreters. ZH (B-) WW
2×06 The Schizoid Man WP: A brilliant scientist and former mentor of Data’s creator, Dr. Ira Graves, cheats death by uploading his memories and personality into the android Data. ZH (C-) WW
2×07 Unnatural Selection WP: The Enterprise receives a distress call from the USS Lantree where the crew find all hands dead from, apparently, old age. The cause of the accelerated aging must be found before scientists on a research colony suffer the same fate. ZH (C+) WW
2×08 A Matter Of Honor WP: Riker is assigned to a Klingon vessel via an officer exchange program between the Federation and the Klingon Empire. ZH (A-) WW
2×09 The Measure Of A Man WP: When Data refuses orders to be dismantled for research purposes, a hearing is convened to determine if he is a legal citizen or property of the Federation. Guest stars Amanda McBroom as JAG Philippa Louvois. ZH (A-) WW
2×10 The Dauphin WP: The Enterprise hosts a young world leader and her mysterious chaperone. Wesley soon falls in love with the young leader. Guest stars Paddi Edwards as Anya. ZH (B) WW
2×11 Contagion WP: A dangerous alien computer virus runs rampant through the Enterprise after causing the destruction of her sister ship, the USS Yamato. ZH (B+) WW
2×12 The Royale WP: Riker, Worf, and Data investigate a structure on an icy gas giant. Inside, they find a casino reconstructed from a bad Earth novel. ZH (B) WW
2×13 Time Squared WP: Picard encounters his future self, when the Enterprise becomes caught in a time loop. ZH (A-) WW
2×14 The Icarus Factor WP: Riker’s estranged father visits to brief him on the command he has been offered, and Worf’s friends discover he is about to miss an important Klingon rite of passage. ZH (C+) WW
2×15 Pen Pals WP: Data befriends a child from a doomed planet, breaking the Prime Directive. ZH (B-) WW
2×16 Q Who WP: Q flings the Enterprise 7,000 light years beyond Federation space and introduces the ship and its crew to the deadly Borg. ZH (A) WW
2×17 Samaritan Snare WP: A group of seemingly dimwitted aliens, the Pakleds, kidnap Geordi to make their ship go. ZH (B-) WW
2×18 Up The Long Ladder WP: Picard must find a way to bring two radically incompatible cultures together, lest both of them face extinction. ZH (C) WW
2×19 Manhunt WP: Deanna’s mother seeks a new husband, and she has her eyes on Picard. ZH (C) WW
2×20 The Emissary WP: Worf’s former lover comes aboard Enterprise to help deal with a Klingon sleeper ship. ZH (B+) WW
2×21 Peak Performance WP: The Enterprise and USS Hathaway face off in simulated combat maneuvers. Data fails to beat a humanoid at a game of Strategema and experiences feelings of self-doubt. ZH (B+) WW
2×22 Shades Of Gray WP: Riker is poisoned by an alien plant and, as he lies comatose, he relives various memories of his life serving the Enterprise. (Clip show as a result of the 1988 Writers Guild of America strike.) ZH (F) WW

Season 3

3×01 Evolution WP: Nanites escape Wesley’s lab and form a collective intelligence, threatening an astrophysicist’s only chance at performing a stellar experiment. Guest star Ken Jenkins as Dr. Paul Stubbs. ZH (B) WW
3×02 The Ensigns of Command WP: Data must persuade a stubborn colony to evacuate their homeland under threat of a powerful and mysterious race. ZH (B+) WW
3×03 The Survivors WP: The Enterprise investigates the last two survivors of an annihilated world, as the entire surface has been transformed to dust except their one little garden and house. Guest stars John Anderson and Anne Haney as Kevin and Rishon Uxbridge. ZH (A) WW
3×04 Who Watches The Watchers WP: Deanna and Riker must rectify the damage done when two primitives from Mintaka III catch glimpse of a Federation observation team and eventually conclude that Captain Picard is a god. ZH (A) WW
3×05 The Bonding WP: A mysterious entity seeks to comfort a boy who has lost his mother in an accident on its planet. Guest star Gabriel Damon as Jeremy Aster. ZH (B+) WW
3×06 Booby Trap WP: The Enterprise falls victim to an ancient booby trap set to snare starships; while in an effort to find an escape, Geordi finds himself falling for the holodeck’s representation of a famous Federation engineer. Guest star Susan Gibney as Dr. Leah Brahms. ZH (B) WW
3×07 The Enemy WP: Geordi is trapped on a harsh planet with a hostile Romulan, and the two must work together to survive. ZH (A-) WW
3×08 The Price WP: Troi falls in love with a charismatic negotiator who vies for rights to a wormhole. But several different groups are after the wormhole as it may be the only known stable wormhole in existence. Guest star Dan Shor as Dr. Arridor. ZH (B-) WW
3×09 The Vengeance Factor WP: Riker exposes an assassin in the midst of critical peace talks. ZH (B) WW
3×10 The Defector WP: Determined to avert a war, a Romulan officer defects to warn Picard of his Empire’s invasion plans. ZH (A) WW
3×11 The Hunted WP: A genetically modified soldier reveals the social problems of a world hoping to join the Federation. Guest star James Cromwell as Prime Minister Nayrok. ZH (A) WW
3×12 The High Ground WP: Dr. Crusher is kidnapped by terrorists who need medical assistance as the technology employed in their attacks is detrimental to their own health. ZH (B-) WW
3×13 Déjà Q WP: The Q Continuum strips Q of his powers, and dumps him aboard the Enterprise. ZH (B+) WW
3×14 A Matter of Perspective WP: Riker is accused of murder, and the holodeck is used to reconstruct the events from different perspectives. ZH (B-) WW
3×15 Yesterday’s Enterprise WP: The Enterprise-C arrives from the past, causing a shift in reality—and the return of the deceased Tasha Yar. ZH (A) WW
3×16 The Offspring WP: Data creates a young gynoid, which he considers his daughter, “Lal”. But a Starfleet admiral arrives demanding she be removed from the Enterprise. Guest star Hallie Todd as Lal. ZH (A-) WW
3×17 Sins of the Father WP: Worf goes on trial to prove his father’s innocence after the Klingon High Council declares that Worf’s father is a traitor, and worked with the Romulans all along. ZH (A) WW
3×18 Allegiance WP: Aliens kidnap Picard and replace him with a duplicate, who sends the Enterprise to a pulsar. Meanwhile, the real Picard and three other captives try to escape from their prison. ZH (B) WW
3×19 Captain’s Holiday WP: Picard is convinced to take some much needed shore leave, but gets wrapped up in a woman’s treasure hunt. Guest star Max Grodenchik as Sovak. ZH (B) WW
3×20 Tin Man WP: A gifted telepath whom Deanna Troi once treated as a patient comes aboard to establish first contact with an unknown vessel near an unstable star before the Romulans do. ZH (B+) WW
3×21 Hollow Pursuits WP: Lt. Barclay’s use of the holodeck as an escape interferes with his duties. Meanwhile, the Enterprise suffers from mysterious and random malfunctions. ZH (B) WW
3×22 The Most Toys WP: An obsessed collector is determined to add Data to his private collection of unique items. Guest star Saul Rubinek as Kivas Fajo. ZH (A-) WW
3×23 Sarek WP: The Enterprise hosts Ambassador Sarek, but his deteriorating mental health causes unforeseen problems. ZH (A) WW
3×24 Ménage à Troi WP: The Ferengi kidnap Deanna, her mother and Riker. ZH (B) WW
3×25 Transfigurations WP: The Enterprise rescues a humanoid with amnesia and incredible healing powers. ZH (B-) WW
3×26 The Best of Both Worlds WP: Picard is kidnapped by the Borg, who begin their invasion of Federation space. ZH (A) WW

Season 4

4×01 The Best of Both Worlds, Part II WP: Picard is rescued from the Borg as the Enterprise races to save Earth. A great number of Starfleet ships are destroyed by the lone Borg ship, although an away team finally rescues Picard. Data interfaces with the half-Borg Picard and finds a way to shut down the Borg ship. Guest star Elizabeth Dennehy as Starfleet Commander Shelby. ZH (A) WW
4×02 Family WP: Picard visits his family in France and Worf’s human parents come aboard the Enterprise. ZH (A) WW
4×03 Brothers WP: Data is summoned by his creator Noonien Soong who is still alive, and they are joined by Lore. ZH (B+) WW
4×04 Suddenly Human WP: Picard must help a human boy, raised by aliens, to decide his fate. ZH (A-) WW
4×05 Remember Me WP: After an apparent failure of a warp-field experiment, people begin to disappear from the Enterprise with only Dr. Crusher remembering that they ever existed. ZH (B) WW
4×06 Legacy WP: Tasha Yar’s sister Ishara seeks to restore order on their conflict-ridden colony world. ZH (B) WW
4×07 Reunion WP: Worf’s ex-girlfriend returns, and along with Picard, the two mediate a Klingon power dispute and Worf discovers more family. ZH (B+) WW
4×08 Future Imperfect WP: Riker finds himself sixteen years in the future, his memory of the interim erased by a dormant virus. ZH (A-) WW
4×09 Final Mission WP: Wesley sets off on his final mission with the Enterprise accompanied by Picard, but they become stranded on a desert planet. ZH (B+) WW
4×10 The Loss WP: An unknown force captures the Enterprise and causes Deanna to lose her empathic powers. ZH (C+) WW
4×11 Data’s Day WP: Data gets dancing lessons from Dr. Crusher in preparation of Chief O’Brien’s wedding as the Enterprise brings Ambassador T’Pel to the Romulans for negotiations. ZH (B-) WW
4×12 The Wounded WP: A rogue Starfleet Captain jeopardizes the Cardassian peace treaty. ZH (A-) WW
4×13 Devil’s Due WP: A powerful mythic figure from a millennium ago returns to enslave a planet in accordance with a contract. However Picard is convinced she is an opportunistic charlatan. Guest star Marta DuBois as Ardra. ZH (B+) WW
4×14 Clues WP: The crew, with the exception of Data, is rendered unconscious for 30 seconds after going through a localized wormhole. However, various clues suggest they were unconscious for an entire day. ZH (B+) WW
4×15 First Contact WP: Riker is hospitalized during a botched pre-first contact mission. Xenophobia results in increasing hostility toward his presence. ZH (B+) WW
4×16 Galaxy’s Child WP: The Enterprise accidentally kills a space creature, and the crew rush to save its unborn offspring. Meanwhile, Geordi meets the engineer he fell in love with and finds to his shock, she’s nothing like the woman he encountered on the holodeck. Guest star Susan Gibney as Dr. Leah Brahms. ZH (C+) WW
4×17 Night Terrors WP: The Enterprise is trapped in a rift. The crew succumbs to REM sleep deprivation, while Deanna has a recurring nightmare. ZH (B) WW
4×18 Identity Crisis WP: Geordi transforms into an alien creature with strong instinct to return to its planet of origin. ZH (B+) WW
4×19 The Nth Degree WP: After an encounter with an alien probe Barclay experiences great leaps in confidence and intelligence. ZH (B) WW
4×20 Qpid WP: Q returns to test Picard’s love for an old flame. ZH (B-) WW
4×21 The Drumhead WP: A witchhunt ensues for suspected Romulan spies aboard the Enterprise. Guest star Jean Simmons as Rear Admiral Norah Satie. ZH (A) WW
4×22 Half a Life WP: Lwaxana Troi finally finds love, but discovers her man must undergo a ritualistic suicide. Guest star David Ogden Stiers as Timicin. ZH (B+) WW
4×23 The Host WP: Dr. Crusher falls in love with Odan, only to discover that Odan is a symbiote, which is implanted into Riker after his original host dies. Odan continues peace negotiations using Riker as a temporary host. ZH (B+) WW
4×24 The Mind’s Eye WP: The Romulans brainwash Geordi to carry out a covert mission. ZH (A) WW
4×25 In Theory WP: Data participates in a romantic relationship with a fellow crew member. ZH (B) WW
4×26 Redemption WP: Worf leaves the Enterprise to fight on behalf of Gowron in a Klingon civil war. ZH (A-) WW

Season 5

5×01 Redemption II WP: A fleet of 23 Federation ships blockades Romulan support to the Duras family, resulting in Gowron’s installation as Chancellor. ZH (B) WW
5×02 Darmok WP: Picard must learn to communicate with an alien captain who speaks in metaphors before a dangerous beast kills them both. Guest star Paul Winfield as Dathan. ZH (A) WW
5×03 Ensign Ro WP: After an attack on a Federation outpost, Picard is sent to locate a Bajoran terrorist, with the help of Ensign Ro Laren. ZH (A-) WW
5×04 Silicon Avatar WP: The crew, with the help of a scientist whose son lived on Data’s home world, attempt to communicate with the Crystalline Entity. ZH (B+) WW
5×05 Disaster WP: The Enterprise is without power, trapping Picard in a turbolift with three children and trapping others in various locations. Command of the bridge falls to Counselor Troi, who feels ill-prepared. ZH (B+) WW
5×06 The Game WP: Wesley visits the Enterprise, but finds the crew addicted to a mind-altering computer game. Guest star Ashley Judd as Robin Lefler. ZH (C) WW
5×07 Unification I WP: Spock is reported to have defected to the Romulans. Picard and Data travel to Romulus on a cloaked Klingon vessel to investigate. ZH (B) WW
5×08 Unification II WP: Spock attempts to unify the Vulcans and Romulans in peace, but falls into a Romulan trap. ZH (B) WW
5×09 A Matter of Time WP: A historian from the 26th century visits the Enterprise, while they help a planet prevent a nuclear winter. Guest star Matt Frewer as Berlinghoff Rasmussen. ZH (B) WW
5×10 New Ground WP: Worf tries to be a father to his son, Alexander, while the Enterprise helps to test a new propulsion technology. ZH (B-) WW
5×11 Hero Worship WP: Data saves the life of an orphaned boy, who begins to emulate him. ZH (B+) WW
5×12 Violations WP: An alien traveling aboard the Enterprise telepathically molests Troi and invades the minds of Beverly Crusher and William Riker. ZH (C+) WW
5×13 The Masterpiece Society WP: The Enterprise helps a far-flung eugenic human colony avoid destruction, but upsets its delicate balance by ending 200 years of isolation. ZH (B-) WW
5×14 Conundrum WP: The crew’s memory is erased, and they discover they are being manipulated into being the key part of a war. ZH (B+) WW
5×15 Power Play WP: Troi, O’Brien, and Data are possessed by entities who want control of the ship. ZH (B) WW
5×16 Ethics WP: Worf becomes paralyzed—and suicidal—and Dr. Crusher consults a risk-taking researcher to save his life. ZH (B-) WW
5×17 The Outcast WP: Riker falls in love with an androgynous person after rescuing some others trapped in “null space.” ZH (A) WW
5×18 Cause and Effect WP: The Enterprise becomes stuck in a causality loop, but the crew retain some memory of previous instances. Guest star Kelsey Grammer as Morgan Bateson. ZH (A) WW
5×19 The First Duty WP: Wesley is questioned over a Starfleet Academy flight-training accident. Guest star Ray Walston as Boothby. ZH (A) WW
5×20 Cost of Living WP: Deanna’s mother, Lwaxana, arrives to marry a man she has never met. Worf has difficulty rearing Alexander, which is exacerbated when Lwaxana takes the boy under her wing. ZH (C) WW
5×21 The Perfect Mate WP: Picard forces himself to resist the charms of a female empathic metamorph, who is sent to marry an alien leader as a peace offering. Guest star Famke Janssen as Kamala. ZH (B+) WW
5×22 Imaginary Friend WP: A child’s imaginary playmate takes on real form and threatens the well-being of the Enterprise. ZH (B-) WW
5×23 I Borg WP: The Enterprise rescues a Borg survivor, and Picard plans to use him as a weapon against his nemesis by exposing him to a computer virus. ZH (A-) WW
5×24 The Next Phase WP: A transporter accident traps Geordi and Ensign Ro out of phase; while the others plan their funeral they must find a way to reverse the process and save the Enterprise from destruction. ZH (A-) WW
5×25 The Inner Light WP: A space probe creates a telepathic tether and causes Picard to experience, in twenty-five minutes, a lifetime as a married man on a world that was destroyed a millennium ago. ZH (A+) WW
5×26 Time’s Arrow WP: A 500-year-old artifact is uncovered on Earth: Data’s severed head. The Enterprise investigates alien involvement in Earth’s past and Data fulfills his destiny. ZH (B-) WW

Season 6

6×01 Time’s Arrow, Part II WP: The Enterprise crew follow Data to San Francisco of the 1890s. The crew deals with Samuel Clemens (and run into Jack London), while trying to find a way to prevent aliens from interfering with 19th-century Earth. Guest star: Jerry Hardin as Samuel Clemens. ZH (D+) WW
6×02 Realm of Fear WP: Barclay must overcome his fear of the transporter to solve a mystery. ZH (B-) WW
6×03 Man of the People WP: A psychic ambassador uses Deanna’s mind to influence the outcome of his mission. ZH (C) WW
6×04 Relics WP: The Enterprise investigates a vessel that crashed on the surface of a Dyson sphere 75 years ago. An undegraded pattern is found in the transporter buffer, that of Mr. Scott. Feeling out of place and obsolete, Scotty agrees to return to his vessel with Geordi to help restore the logs, and they become the only hope when the Enterprise is accidentally pulled inside the sphere. ZH (A) WW
6×05 Schisms WP: Several members of the crew are abducted and experimented on while they sleep. ZH (B) WW
6×06 True Q WP: Q reveals a secret about a young woman from Kansas who is visiting the Enterprise. Guest star: Olivia D’Abo as Amanda Rogers. ZH (B+) WW
6×07 Rascals WP: A transporter malfunction turns Picard, Keiko, Ro and Guinan into children, who become the ship’s only hope when they are left aboard while the adult crew are forced to perform dangerous labor by Ferengi pirates. ZH (B-) WW
6×08 A Fistful of Datas WP: Data’s mind is connected to the ship’s computer, which creates unforeseen effects on the holodeck. ZH (B+) WW
6×09 The Quality of Life WP: Data observes self-guided “tools” used at a mining station display signs of sentience, and fights for their preservation, even risking Captain Picard’s life on the ground that it is unacceptable to kill one sentient being to save another. ZH (B+) WW
6×10 Chain of Command, Part I WP: Captain Jellico is assigned command of the Enterprise, while Picard is sent on a covert mission into Cardassian territory. Guest star: Ronny Cox as Edward Jellico. ZH (B+) WW
6×11 Chain of Command, Part II WP: Picard, having been captured, is tortured by a sadistic Cardassian interrogator (played by David Warner). ZH (A) WW
6×12 Ship in a Bottle WP: Barclay accidentally awakens Prof. Moriarty on the holodeck, who uses the powers at his disposal to coerce the crew into finding a way to allow him to leave the holodeck. Guest star: Daniel Davis as Professor Moriarty. ZH (A-) WW
6×13 Aquiel WP: Geordi falls for an alien Starfleet officer who is suspected of murder. Guest star Renee Jones as Aquiel Uhnari. ZH (C) WW
6×14 Face of the Enemy WP: Deanna is involuntarily recruited to assist in the transport of Romulan defectors across the border. ZH (B+) WW
6×15 Tapestry WP: An accident kills Picard, and he finds an afterlife with Q analyzing his past choices. ZH (A) WW
6×16 Birthright, Part I WP: Worf is told on Deep Space Nine his father is alive, and being held prisoner by the Romulans. Meanwhile an engineering experiment accidentally results in Data’s first dream. Guest star: James Cromwell as Jaglom Shrek. ZH (B+) WW
6×16 Birthright, Part II WP: Worf, now a prisoner, tries to teach the Klingon refugees the ways of the warrior. ZH (B) WW
6×18 Starship Mine WP: Thieves attempt to steal trilithium from Enterprise during a Baryon sweep at the Remmler Array, and Picard alone must thwart them. ZH (B) WW
6×19 Lessons WP: Picard becomes involved with a woman who is serving on the Enterprise, but he must send her into a dangerous mission. ZH (B+) WW
6×20 The Chase WP: Picard tries to solve an ancient genetic mystery uncovered by his archaeological mentor, and faces stiff competition. Guest star: Norman Lloyd as Richard Galen. ZH (B+) WW
6×21 Frame of Mind WP: Riker finds himself prisoner in an alien mental institution, which resembles scenes from Beverly’s play. ZH (A) WW
6×22 Suspicions WP: Dr. Crusher risks her career to solve the murder of Ferengi scientist Dr. Reyga and vindicate his research. ZH (B-) WW
6×23 Rightful Heir WP: Worf experiences a crisis of faith, and travels to a Klingon holy site where the mythic figure Kahless returns to lead the Klingon people. ZH (A) WW
6×24 Second Chances WP: Riker encounters a duplicate of himself created by a transporter malfunction stranded on a planet. “Thomas” vies for Deanna’s affections. ZH (B+) WW
6×25 Timescape WP: The Enterprise is caught in temporal stasis, and on the brink of destruction by a Romulan ship. ZH (B) WW
6×26 Descent WP: The crew encounter a group of Borg acting individually, and Data briefly experiences emotions. ZH (B) WW

Season 7

7×01 Descent, Part II WP: The Borg are being led by Lore. Data falls under his control by being fed negative emotions. ZH (B-) WW
7×02 Liaisons WP: Worf and Troi reluctantly play host to two Lyraan ambassadors, while Picard crashes in a shuttle with another Lyraan. He is rescued by a human female who exhibits strange behavior. ZH (B-) WW
7×03 Interface WP: Geordi tries to rescue his mother’s starship via a remotely controlled probe. ZH (C+) WW
7×04 Gambit, Part I WP: The Enterprise crew investigate the apparent murder of Captain Picard during an archaeological trip. Riker is kidnapped by mercenaries and finds Picard working as part of their crew. ZH (B) WW
7×05 Gambit, Part II WP: Picard and Riker help mercenaries collect archaeological artifacts to prevent an ancient Vulcan weapon falling into the wrong hands. ZH (B-) WW
7×06 Phantasms WP: Data experiences strange dreams, while the Enterprise has issues with its new Warp-core. But all is not as it seems. ZH (B+) WW
7×07 Dark Page WP: A psychic breakdown puts Lwaxana Troi in a coma, and Deanna works to save her life. Features a young Kirsten Dunst, playing the little girl Hedril. ZH (B-) WW
7×08 Attached WP: Reclusive aliens imprison Picard and Dr. Crusher on charges of espionage, and experimental implants linking their minds telepathically cause them to face their latent feelings for each other. ZH (B+) WW
7×09 Force of Nature WP: A pair of scientists show that warp drives are harming the fabric of space. ZH (B) WW
7×10 Inheritance WP: Data encounters a woman claiming to be his “mother”. ZH (B-) WW
7×11 Parallels WP: Worf finds himself randomly shifting between alternate realities. ZH (A-) WW
7×12 The Pegasus WP: Riker’s former Captain boards the Enterprise to retrieve the USS Pegasus. Picard investigates the circumstances of its loss and finds that there has been a cover-up. Features Terry O’Quinn as Admiral Eric Pressman. ZH (A-) WW
7×13 Homeward WP: Worf’s human foster brother violates the Prime Directive to save a doomed primitive race. ZH (C+) WW
7×14 Sub Rosa WP: Dr. Crusher attends her grandmother’s funeral, and takes on an unusual family tradition. ZH (D) WW
7×15 Lower Decks WP: Junior officers buck for promotion as one of them is assigned the dangerous task of helping a Cardassian spy. ZH (A) WW
7×16 Thine Own Self WP: Data loses his memory after retrieving radioactive fragments on a planet’s surface and endangers the humanoid settlement he encounters, while Deanna studies to become a bridge officer. ZH (B+) WW
7×17 Masks WP: The Enterprise finds an ancient library that recreates its civilization by taking possession of Data and transforming the ship. ZH (C) WW
7×18 Eye of the Beholder WP: Deanna investigates the suicide of a crewman and uncovers a murder that took place during construction of the Enterprise. ZH (C-) WW
7×19 Genesis WP: A routine medical treatment inadvertently creates a virus that begins to de-evolve the Enterprise crew while Picard and Data are on an away mission. This is the first and only episode to be directed by Gates McFadden, who plays Dr. Crusher. ZH (C-) WW
7×20 Journey’s End WP: Wesley considers his future, as the Enterprise is ordered to remove Native American colonists from a planet that is about to fall under Cardassian jurisdiction. Guest star Richard Poe as Gul Evek. Final appearance of Eric Menyuk as The Traveler. ZH (C) WW
7×21 Firstborn WP: Worf attempts to convince his son Alexander to embrace his warrior heritage. Guest star James Sloyan as K’mtar. ZH (C+) WW
7×22 Bloodlines WP: DaiMon Bok returns to exact revenge on Picard, by trying to kill the son Picard never knew he had. Guest star Lee Arenberg as DaiMon Bok. ZH (C+) WW
7×23 Emergence WP: The Enterprise becomes an emergent intelligence. ZH (B-) WW
7×24 Preemptive Strike WP: Ensign Ro graduates from advanced tactical training, and is sent by Picard to lure Maquis terrorists into a trap. Guest star Richard Poe as Gul Evek. ZH (A) WW
7×25 All Good Things… WP: Picard finds himself alternating between three time periods thanks to Q, with a spacetime distortion that threatens to destroy reality growing larger in the past, and smaller in the future. ZH (A) WW

No, “cassette tape” hasn’t been removed from the Concise Oxford English Dictionary

I’ve seen many people link to a news story claiming that “cassette tape” has been removed from the Concise OED — this story appears to be complete nonsense,  and I’m genuinely intrigued as to how it got started.  Of course, none of the news stories that repeat this claim cite any source.  This is a minor example of a serious problem that Francis has blogged about.

TIME magazine is perhaps the highest profile publication to  make the claim, saying that “cassette tape” is being removed from the Oxford English Dictionary and referencing The Huffington Post as its only source.  Firstly, the TIME story is obviously nonsense taken literally, since the OED is a historical dictionary and words generally aren’t removed from its 20 volumes at all.  Other stories clarify that this is supposedly a scandal with the new (12th) edition of the Concise Oxford English Dictionary, which was released in August 2011 to a flurry of stories about the various entertaining new words that have been added to it.  The news stories about those words seem to be largely based on two blog posts (here and here) from Oxford University Press that discuss in general terms what has defined the COED over its 100 year history.  Neither of those posts mention anything about “cassette tape” being removed.  (I haven’t found any more official press release about this new edition online.)

Anyway, suspecting that this story was nonsense, I did the obvious thing and looked in the new 12th edition of the COED in a local bookshop.  Of course, cassette tape is still here:

So, my real question is: “Who is it that just invents this stuff?”

Official Kindle subscription to The Guardian

It seems that finally The Guardian have an official Kindle version, that you can subscribe to here:

The Guardian and The Observer [Kindle Edition]

A subscription costs a very reasonable £9.99 per month, and like all the official Kindle periodicals, it appears on your device automatically if wireless is enabled.  It looks as if they’ve done a very nice job – I’d encourage anyone who was regularly using my unofficial Guardian for Kindle project to subscribe to the official version.  As well as the convenience of automatic delivery, you get more complete content (since a number of articles aren’t available for free via the Guardian’s API) and bigger photos for each story.  In addition, with newspapers in general struggling these days, it’s good to be able to support them by paying for this service.

Where does this leave my unofficial version?  My plan is to leave in place the system that generates the web-hosted version each day, but prominently link to the official version on that page.  The version generated from the Guardian’s API might be occasionally useful for people, and I think the project is of interest in its own right – figuring out how to generate Kindle books and then periodicals was hard work, and it’s still a satisfying project to have got working so well in the end.  Even masthead images now work, thanks to a tip from Marco Arment!  (Again I should thank Dominic Evans for figuring out how to switch from generating books to periodicals, and his other contributions.)

I’ve summarized what I’ve learned so far about generating periodicals using Kindlegen in this Stack Overflow answer, and I’ll update that if I find out anything new.

Making an encrypted partition on a USB drive

On Ubuntu or Debian, it’s really simple to create an encrypted partition on a newly-purchased USB mass storage device.  In my case, I had bought a 1TB hard drive which had very mixed reviews, some people saying their drives had failed very early.  I wanted to be able to return the drive under warranty if it broke without worrying about personal data.

It turns out that if you want to reformat a partition on an external USB drive so that it’s encrypted, this is just a matter of doing the following:

sudo luksformat -t ext4 /dev/partitiondevice

…. where /dev/partitiondevice is the device for the drive partition you want to overwrite.  Obviously, this will destroy everything that was previously on that partition.

I like to use a proper filesystem for USB mass storage devices, but if you leave out the -t ext4 then the default is to use VFAT.

When you next plug in that drive, you’ll be prompted to enter the password that you picked when creating the partition – if you type that correctly, the drive will be mounted and usable.  (If you mistype it, you’re not given another chance to enter the password, so you’ll need to go to the command line and do: gvfs-mount -d /dev/partitiondevice to try again.)

One small thing is that the mount point in /media will be based on a UUID by default, but if you set the ext4 partition label, it’ll be mounted under that name in /media/ instead.  To do this, starting from when your disk is mounted, you can run mount without parameters to find the unencrypted device name and then unmount it and change the label:

$ umount /dev/mapper/udisks-luks-uuid-b7bbb2c8-etc
$ e2label /dev/mapper/udisks-luks-uuid-b7bbb2c8-etc topsekrit

If you unplug and plug in the disk again, it should be mounted on /media/topsekrit