Gmail Backups with fetchmail on OS X
Posted on August 14, 2008
Filed under Mac, Scripts
I use GMail as my personal email provider, and as much as I like the simple and snappy UI, the conversation views and the filtering and search possibilities, I’ve grown more and more worried about having all of my (important) mails stored on someone else’s servers. Now, out of all of the big IT companies in the world I’d say I trust Google a heck of a lot more than anyone else, but this doesn’t mean that I shouldn’t take into consideration the possibility of something going wrong on their end and as a result some (or Bob forbid, all) of my mails disappearing into bit heaven.
The good news is that Google provides a nice, standard POP3 interface for downloading emails from their service, and all of the software required for downloading messages via POP is already installed in Mac OS X Leopard by default. Below I’ll go through all of the steps it took me to set up periodical and automated GMail backups on my Macbook.
The following were my sources for most of the information presented here:
- LifeHacker: Geek to Live: Back up Gmail with fetchmail
- Logic Bound: Backup Gmail Data
- bronski.net: fetchmail & SSL
- Matthias Andree and Paul Elliott on [fetchmail-users]: What is a “local issuer certificate”?
Configure GMail for POP access
- Go to GMail’s “Forwarding and POP/IMAP” settings
- Set the “POP access” setting to “Enable POP for all mail (even mail that’s already been downloaded)”
- Make sure that the “When messages are accessed with POP” setting says “keep Gmail’s copy in the inbox”
Configure fetchmail to Download the Messages
- Create ~/.fetchmailrc with the following contents (replacing GMAIL_USERNAME, GMAIL_PASSWORD and LOCAL_USERNAME with their real values in your case):
- Connect to the GMail POP server using the POP3 protocol,
- Map your GMail username to your localhost username,
- Deliver the downloaded mail into your local system mailbox (mail spool file),
- Connect using an encrypted SSL connection,
- Strictly check the SSL certificates of the server it connects to against local trusted certificates,
- Search for the local certificates in ~/.ssl/certs
- Set the proper rights for ~/.fetchmailrc:
- Get the pop.gmail.com server’s SSL certificate from the POP port (note that this should be done using a secure, trusted internet connection — otherwise it kind of defeats the purpose):
- Download the certificate authority (Equifax) root certificate (for example from GeoTrust’s site here, where it says “Equifax Secure Certificate Authority (Base-64 encoded X.509)”), rename it to “equifax.pem” and move it to ~/.ssl/certs/equifax.pem (and just like the previous step, this should be done over a trusted internet connection as well).
- Hash the certificates in this directory by running:
poll pop.gmail.com with proto POP3 and options no dns user 'GMAIL_USERNAME@gmail.com' there with password 'GMAIL_PASSWORD' is 'LOCAL_USERNAME' here and wants mda "/usr/bin/procmail -d %T" options ssl keep sslcertck sslcertpath "/Users/LOCAL_USERNAME/.ssl/certs"
This configuration basically tells fetchmail to:
From the output of this command, copy the part that looks like this:
(a big bunch of characters here)
-----END CERTIFICATE-----
And save it into a file called ~/.ssl/certs/gmailpop.pem.
Download Messages with fetchmail
You can now download the mail by running this command (the -v argument is for verbose output):
Unless you only have a few mails in your Gmail box, this command will initially have to be run several times in a row in order to get all of the messages since it only downloads 200-400 messages at a time. The mail spool file containing all of the downloaded messages will then be found in /var/mail/LOCAL_USERNAME.
It’s a good idea to set up a launchd job for running this command between regular intervals so that you wouldn’t have to remember to manually do it yourself. Lingon is an ideal GUI app for this purpose. My GMail backup launchd job doesn’t call fetchmail directly, though: it calls this helper Python script that I’ve written. The script does the following:
- Runs fetchmail,
- Displays Growl messages about the result (success or failure, number of messages downloaded if successful, etc.),
- Optionally saves fetchmail’s output into a log file when done.
Feel free to use it (remember to set the values in the “settings” section before running it, though).
Read Downloaded Messages with Thunderbird
In Mozilla Thunderbird, go to Tools > Account Settings… > Local Folders to get to the Local directory path. Go into this directory and create a symbolic link there that points to the mail spool file (/var/mail/LOCAL_USERNAME):
ln -s /var/mail/LOCAL_USERNAME ./GMailBackups
After this you’ll be able to find your backed up GMail messages in a folder called “GMailBackups” under “Local Folders” in Thunderbird.
Comments
14 Responses to “Gmail Backups with fetchmail on OS X”
From looking at the DBmail “big picture” diagram (this one: http://www.dbmail.org/dokuwiki/doku.php?id=bigpicture) and the man page of procmail (the MDA used in the directions on this page,) it seems that procmail receives the email from fetchmail via stdin. The corresponding part in the DBMail design (a MDA that receives messages via stdin) seems to be dbmail-smtp (in the diagram it says local mail delivery via a UNIX pipe goes there.)
So maybe you could replace this part in .fetchmailrc:
with something like this:
If dbmail-smtp wants the local username of the user to whom’s mailbox to deliver the emails to as an argument, it seems you can use that %T modifier for that.
$ which c_rehash
/opt/local/bin/c_rehash
$ c_rehash ~/.ssl/certs/
Doing /Users/robbins/.ssl/certs/
WARNING: equifax.pem does not contain a certificate or CRL: skipping
WARNING: gmailpop.pem does not contain a certificate or CRL: skipping
I made sure that the directories down from my homedir were all 775 and I know that I cut and pasted the certs as instructed.
I can think of a couple things you might want to check:
1) if you’re using OS X, try using the standard c_rehash in /usr/bin/.
2) try the -verbose argument to try and get more info about the failure
also, i’ve tried using the -v when using fetchmail and all i’ve got is SMTP transaction error while fetching. Can you help me out?
Have you made sure that you selected “Enable POP for all mail (even mail that’s already been downloaded)” in GMail’s settings, instead of “Enable POP for mail that arrives from now on”?
The reason I didn’t do it that way is because I wanted to automate this process and have it run in the background instead of having to remember to open Thunderbird and refresh it at regular intervals. But of course if you use Thunderbird as your main mail client anyway, this is a good tip if it indeed accomplishes the same thing.
“…
fetchmail: Server certificate verification error: unable to get local issuer certificate
fetchmail: Certificate/fingerprint verification was somehow skipped!
fetchmail: SSL connection failed.
…”
Liegt das an Gmail oder an Snow Leopard oder an mir???
I have the same symptom as Christian Bender. When I run the openssl command, I get a certificate chain with two certificates in it. I’ve tried using each one separately as my gmailpop.pem, and both of them get hashed alright, but fetchmail complains.
Which of the certificates in the chain are we supposed to use? Or are we supposed to combine them somehow?
Thanks for your help with this. I tried getting backup-gmail to work, but it’s closed-source and not designed for Mac OS X. Your solution seems superior.
Wanted to also comment and say that this method works flawlessly with 10.6 (I didn’t think 10.6 would break anything, but you never know).