Gmail Backups with fetchmail on OS X

Posted on August 14, 2008
Filed under Featured, Mac, Scripts

I use GMail as my personal email provider, and as much as I like the simple and snappy UI, the conversation views and the filtering and search possibilities, I’ve grown more and more worried about having all of my (important) mails stored on someone else’s servers. Now, out of all of the big IT companies in the world I’d say I trust Google a heck of a lot more than anyone else, but this doesn’t mean that I shouldn’t take into consideration the possibility of something going wrong on their end and as a result some (or Bob forbid, all) of my mails disappearing into bit heaven.

The good news is that Google provides a nice, standard POP3 interface for downloading emails from their service, and all of the software required for downloading messages via POP is already installed in Mac OS X Leopard by default. Below I’ll go through all of the steps it took me to set up periodical and automated GMail backups on my Macbook.

The following were my sources for most of the information presented here:

Configure GMail for POP access

  1. Go to GMail’s “Forwarding and POP/IMAP” settings
  2. Set the “POP access” setting to “Enable POP for all mail (even mail that’s already been downloaded)”
  3. Make sure that the “When messages are accessed with POP” setting says “keep Gmail’s copy in the inbox”

Configure fetchmail to Download the Messages

  1. Create ~/.fetchmailrc with the following contents (replacing GMAIL_USERNAME, GMAIL_PASSWORD and LOCAL_USERNAME with their real values in your case):
  2. poll pop.gmail.com with proto POP3 and options no dns
    user 'GMAIL_USERNAME@gmail.com' there with password 'GMAIL_PASSWORD' is 'LOCAL_USERNAME' here and wants mda "/usr/bin/procmail -d %T"  options ssl keep sslcertck sslcertpath "/Users/LOCAL_USERNAME/.ssl/certs"
    

    This configuration basically tells fetchmail to:

    • Connect to the GMail POP server using the POP3 protocol,
    • Map your GMail username to your localhost username,
    • Deliver the downloaded mail into your local system mailbox (mail spool file),
    • Connect using an encrypted SSL connection,
    • Strictly check the SSL certificates of the server it connects to against local trusted certificates,
    • Search for the local certificates in ~/.ssl/certs
  3. Set the proper rights for ~/.fetchmailrc:
  4.     chmod 710 ~/.fetchmailrc
  5. Get the pop.gmail.com server’s SSL certificate from the POP port (note that this should be done using a secure, trusted internet connection — otherwise it kind of defeats the purpose):
  6.     openssl s_client -connect pop.gmail.com:995 -showcerts

    From the output of this command, copy the part that looks like this:

    -----BEGIN CERTIFICATE-----
    (a big bunch of characters here)
    -----END CERTIFICATE-----

    And save it into a file called ~/.ssl/certs/gmailpop.pem.

  7. Download the certificate authority (Equifax) root certificate (for example from GeoTrust’s site here, where it says “Equifax Secure Certificate Authority (Base-64 encoded X.509)”), rename it to “equifax.pem” and move it to ~/.ssl/certs/equifax.pem (and just like the previous step, this should be done over a trusted internet connection as well).
  8. Hash the certificates in this directory by running:
  9.     c_rehash ~/.ssl/certs/

Optional: Specify the Location for Downloaded Messages

We’ve configured fetchmail to send messages to procmail, which will handle the storing of them in a mail spool file. By default this will be in /var/mail/LOCAL_USERNAME, but if you want to store it somewhere else, you can specify the location in the ~/.procmailrc configuration file. Below is an example (replace LOCAL_USERNAME with your username):

MAILDIR="$HOME/.mailspool"
DEFAULT="$HOME/.mailspool/LOCAL_USERNAME"

I prefer to exclude my mail spool file from Time Machine backups (because it’s a large file that changes very often which makes it take up a lot of disk space from my backup volume) so I have it located in ~/.mailspool/ like in the above example. This is because in OS X /var is actually a symbolic link to /private/var and Time Machine has a bug where it’s impossible to exclude anything from under that path from backups (for example, if I choose to exclude /private/var/mail in the GUI, it’ll replace this selection with /var/mail automatically, and even though /var/mail is then excluded, it’ll still back up /private/var/mail).

Download Messages with fetchmail

You can now download the mail by running this command (the -v argument is for verbose output):

fetchmail -v

Unless you only have a few mails in your Gmail box, this command will initially have to be run several times in a row in order to get all of the messages since it only downloads a few hundred messages at a time. The mail spool file containing all of the downloaded messages will then be found in /var/mail/LOCAL_USERNAME (or in whichever location you’ve specified in ~/.procmailrc).

It’s a good idea to set up a launchd job for running this command between regular intervals so that you wouldn’t have to remember to manually do it yourself. Lingon is an ideal GUI app for this purpose. My GMail backup launchd job doesn’t call fetchmail directly, though: it calls this helper Python script that I’ve written. The script does the following:

Feel free to use it (remember to set the values in the “settings” section before running it, though).

Read Downloaded Messages with Thunderbird

In Mozilla Thunderbird, go to Tools > Account Settings… > Local Folders to get to the Local directory path. Go into this directory and create a symbolic link there that points to the mail spool file:

cd PATH_COPIED_FROM_THUNDERBIRD
ln -s /var/mail/LOCAL_USERNAME ./GMailBackups

After this you’ll be able to find your backed up GMail messages in a folder called “GMailBackups” under “Local Folders” in Thunderbird.

 

Comments

20 Responses to “Gmail Backups with fetchmail on OS X”

Show/hide comments & reply form: