Switching mail reader from VM to Gnus

This page describes the steps involved in switching from using VM as a mail reader to Gnus. I've set this up with:

Gnus v5.10.7 and GNU Emacs 21.3.1
Gnus v5.10.6 and GNU Emacs 21.3.1

... but I suspect it will work with later versions of them as well. I haven't tried to get the setup to work with XEmacs, since it doesn't seem to handle UTF-8 nearly so well as GNU Emacs.

For various reasons, I found this made my life much better, but this may well be far from the case for you. In particular, the following things might not suit you:

I like reading my email like a newsgroup, with my sent messages mixed in with the replies and threaded properly. This works OK, but with some provisos mentioned below. You might not like that style, though, in which case you'll need to adjust the steps below.
Gnus doesn't have recursive virtual folders like VM, although you can get a similar effect using shortcuts in the summary window like: / a (limit to author), / s (limit to subject), etc. See / C-h for others.
I don't want to use Gnus for reading NNTP newsgroups at all, so I've made nnml my primary select method. If you want to read news with Gnus too, you probably don't want that. I do need to read a couple of newsgroups too, but for them I just use gnus-group-browse-foreign-server (B in the group buffer.)

On the other hand:

I can now read and compose UTF-8 mails properly. (VM really can't cope with UTF-8 encoded mail properly, as far as I could tell, and it doesn't seem to be actively being maintained. (The comments on Google Groups lead me to believe that this is still broken.)
All my old mail is indexed and searchable using nnir.el and swish++, which is fast and efficient (if sometimes rather scary, when an unexpected email from when I was much younger turns up in the search results :))
Viewing large mail folders (with over 100 000 messages) is much quicker with Gnus's nnml backend than large VM folders. I gather that doing that efficiently with VM virtual folders is possible, but I never worked out how.

The comprehensive Gnus manual is very useful, obviously. I've barely mentioned anything about actually using Gnus here, since it would just duplicate the manual. However, it is taking me a long time to get used to different keyboard shortcuts, and the very different style of working with mail.

You need a pretty recent version of Gnus for IMAP over SSL to work properly; I found Gnus 5.10.6 worked fine in this regard. If you need, to download that, compile it, and add it to your load path:

(setq load-path (cons (expand-file-name "~/elisp/gnus/lisp") load-path))

Then make sure that you're requiring 'gnus-load as well (require 'gnus-load) (both those line should be added to your .emacs

Next, you need to create a simple ~/.gnus file. That's set up to work for me on DICE computers in inf.ed.ac.uk - you should go through that line by line to make sure it makes sense for you. For example, change the email addresses, think about whether you want all your incoming and outgoing mail in one big "inbox" folder, etc. At home, where I get more mail, I've split them into folders called mail-1994, ..., mail-2005 and have an "other" catch-all folder for mail from people who use stupid Date: lines in their email.]

(If you try to use that file without having bbdb (the Insidious Big Brother database) installed then you'll get an error complaining about bbdb-insinuate-gnus. This may be in a separate package on your distribution, or you can download it from the site linked to earlier in this paragraph.)

By default Gnus wants to put mail in ~/Mail, so make sure that's on a partition with lots of space, symlinking to another directory if you need to.

You won't be subscribed to the "inbox" group the first time you start up, so once you've logged into the imap server, and it's fetched all your mail, you'll just get a blank Gnus Group page. To find the inbox group, do j inbox then u over the group to subscribe. Hitting RET will let you browse it.

First, import your old mail. This is described in the Gnus manual under Incorporating old mail, and I suggest you follow the instructions there.

So, following those suggestions, I respooled my old VM INBOX, sent-mail and older folders using that method. (Strangely, one of them had problems, and I had to strip out the first couple of messages with VM folder information in them before it worked.) Then you should unsubscribe from the nndoc folder groups you've just browsed.

Now if you view all the messages in inbox with C-u RET, they should all appear threaded and in the right order. However, if you only view 200, they'll be the last 200 that you spooled into there, rather than the last 200 chronologically, since that goes by the order in the mail folder. If that bothers you, you can enter the group viewing all the messages and then do M P b (to process-mark all messages) and B m inbox RET (to move all those messages back into the same group) which (rather hackily) has the effect of reordering the folder.

nnir and swish++

It's been a while since I set up my initial index, but I think it was done as described in /usr/share/doc/swish++/examples/email_indexing/Email-Indexing-Mini-Howto.txt.gz. One strange thing is that nnir.el calls search++ with a ~/Mail/swish++.conf configuration file parameter and complains if it's not there, or if it's empty. If you put a # on the first line, it's happy. Anyway, I built the index initially with:

index++ -v3 --index-file=/home/mark/Mail/swish++.index -e 'mail:*' -E 'Incomin*' -E '*~' -E '.overview' -E '.marks' /home/mark/Mail/

And added the following line to my crontab:


40 5 * * *      /usr/bin/index++ -v0 --incremental --index-file=/home/mark/Mail/swish++.index -e 'mail:*' -E 'Incomin*' -E '*~' -E '.overview' -E '.marks' /home/mark/Mail/ && mv /home/mark/Mail/swish++.index.new /home/mark/Mail/swish++.index

i.e. update the index every morning at 5:40am.

Now G G in the Group buffer searches all my mail *ever*. :)

Incoming* files?

If you find your Emacs crashes before spooling fetched mail, for some reason, the mail might be left in a file called something like /Mail/IncomingXXXXX. If that happens, you can browse the file an an nndoc folder with C-U G f (select the Incoming file) (select "mbox"); process-mark all messages with M P a; use B r to respool all the messages and delete the Incoming file by hand.

Finally...

If you're happy with all that, add (error "You're forgetting to use Gnus instead.") to the top of your .vm file :-)

Things That Are Still Broken

When respooling mail for one reason or another, I've ended up with many, many duplicate articles in my nnml directories. Ideally I'd like to delete these, but there doesn't seem to be a simple way to do this. The normal suggestion seems to be to process-mark all the messages in a folder and the respool them all with B r.

Some other suggestions, such as writing a script that finds duplicate messages and deletes both the file and the corresponding entry in the .overview file, didn't work for me - I wrote a similar script but ended up with the duplicate articles still in the group, and lots of "article is missing" error when I look at them. gnus-regenerate-nov-databases doesn't seem to fix that at all, either.

I suppose I need to write a function like this one. However, gnus-suppress-duplicates doesn't seem to work for me at all. For instance, I would have hoped that this would work:

C-u RET RET
M P b (mark all duplicates)
M-& (next command applies to marked messages)
M-u (mark as unread)
q (to go back to the group buffer)
M-x set-variable RET gnus-suppress-duplicates RET t RET
C-u RET RET (to enter the group)

As far as I understand it, duplicates *should* be marked, but aren't after entering a group of unread articles with gnus-suppress-duplicates set to t (The plan would have then been to mark all read messages, and delete them with M-& B backspace.)