Gmail Backups with fetchmail on OS X

August 14, 2008 Filed under Featured, Mac, Scripts

I use GMail as my personal email provider, and as much as I like the simple and snappy UI, the conversation views and the filtering and search possibilities, I've grown more and more worried about having all of my (important) mails stored on someone else's servers. Now, out of all of the big IT companies in the world I'd say I trust Google a heck of a lot more than anyone else, but this doesn't mean that I shouldn't take into consideration the possibility of something going wrong on their end and as a result some (or Bob forbid, all) of my mails disappearing into bit heaven.

The good news is that Google provides a nice, standard POP3 interface for downloading emails from their service, and all of the software required for downloading messages via POP is already installed in Mac OS X Leopard by default. Below I'll go through all of the steps it took me to set up periodical and automated GMail backups on my Macbook.

The following were my sources for most of the information presented here:

Configure GMail for POP access

Go to GMail's "Forwarding and POP/IMAP" settings
Set the "POP access" setting to "Enable POP for all mail (even mail that's already been downloaded)"
Make sure that the "When messages are accessed with POP" setting says "keep Gmail's copy in the inbox"

Configure fetchmail to Download the Messages

Create ~/.fetchmailrc with the following contents (replacing GMAIL_USERNAME, GMAIL_PASSWORD and LOCAL_USERNAME with their real values in your case):

poll pop.gmail.com with proto POP3 and options no dns
user 'GMAIL_USERNAME@gmail.com' there with password 'GMAIL_PASSWORD' is 'LOCAL_USERNAME' here and wants mda "/usr/bin/procmail -d %T"  options ssl keep sslcertck sslcertpath "/Users/LOCAL_USERNAME/.ssl/certs"

Connect to the GMail POP server using the POP3 protocol,
Map your GMail username to your localhost username,
Deliver the downloaded mail into your local system mailbox (mail spool file),
Connect using an encrypted SSL connection,
Strictly check the SSL certificates of the server it connects to against local trusted certificates,
Search for the local certificates in ~/.ssl/certs

Set the proper rights for ~/.fetchmailrc:


	chmod 710 ~/.fetchmailrc

Get the pop.gmail.com server's SSL certificate from the POP port (note that this should be done using a secure, trusted internet connection -- otherwise it kind of defeats the purpose):


	openssl s_client -connect pop.gmail.com:995 -showcerts

-----BEGIN CERTIFICATE-----
(a big bunch of characters here)
-----END CERTIFICATE-----

~/.ssl/certs/gmailpop.pem

Download the certificate authority (Equifax) root certificate (for example from GeoTrust's site here, where it says "Equifax Secure Certificate Authority (Base-64 encoded X.509)"), rename it to "equifax.pem" and move it to ~/.ssl/certs/equifax.pem (and just like the previous step, this should be done over a trusted internet connection as well).
Hash the certificates in this directory by running:


	c_rehash ~/.ssl/certs/

Optional: Specify the Location for Downloaded Messages

We've configured fetchmail to send messages to procmail, which will handle the storing of them in a mail spool file. By default this will be in /var/mail/LOCAL_USERNAME, but if you want to store it somewhere else, you can specify the location in the ~/.procmailrc configuration file. Below is an example (replace LOCAL_USERNAME with your username):


MAILDIR="$HOME/.mailspool"
DEFAULT="$HOME/.mailspool/LOCAL_USERNAME"

I prefer to exclude my mail spool file from Time Machine backups (because it's a large file that changes very often which makes it take up a lot of disk space from my backup volume) so I have it located in ~/.mailspool/ like in the above example. This is because in OS X /var is actually a symbolic link to /private/var and Time Machine has a bug where it's impossible to exclude anything from under that path from backups (for example, if I choose to exclude /private/var/mail in the GUI, it'll replace this selection with /var/mail automatically, and even though /var/mail is then excluded, it'll still back up /private/var/mail).

Download Messages with fetchmail

You can now download the mail by running this command (the -v argument is for verbose output): fetchmail -v Unless you only have a few mails in your Gmail box, this command will initially have to be run several times in a row in order to get all of the messages since it only downloads a few hundred messages at a time. The mail spool file containing all of the downloaded messages will then be found in /var/mail/LOCAL_USERNAME (or in whichever location you've specified in ~/.procmailrc).

It's a good idea to set up a launchd job for running this command between regular intervals so that you wouldn't have to remember to manually do it yourself. Lingon is an ideal GUI app for this purpose. My GMail backup launchd job doesn't call fetchmail directly, though: it calls this helper Python script that I've written. The script does the following:

Runs fetchmail,
Displays Growl messages about the result (success or failure, number of messages downloaded if successful, etc.),
Optionally saves fetchmail's output into a log file when done.

Feel free to use it (remember to set the values in the "settings" section before running it, though).

Read Downloaded Messages with Thunderbird

In Mozilla Thunderbird, go to Tools > Account Settings... > Local Folders to get to the Local directory path. Go into this directory and create a symbolic link there that points to the mail spool file:

cd PATH_COPIED_FROM_THUNDERBIRD
ln -s /var/mail/LOCAL_USERNAME ./GMailBackups

After this you'll be able to find your backed up GMail messages in a folder called "GMailBackups" under "Local Folders" in Thunderbird.

20 Comments

Steven Garner February 25, 2009 at 8:22 AM

Wonderful instructions … many thanks! Always good to have a backup. You know what would be cool though? Backing up to a database (MySQL, SQLite, etc.) rather than a mail client. Then one has all the power of a database for searching etc. A good open source tool for this is http://www.dbmail.org. But it would be great if someone could give instructions similar to yours on how to move all of a gmail account into a database.

Ali Rantakari February 25, 2009 at 3:14 PM

Hi Steven,

From looking at the DBmail “big picture” diagram (this one: http://www.dbmail.org/dokuwiki/doku.php?id=bigpicture) and the man page of procmail (the MDA used in the directions on this page,) it seems that procmail receives the email from fetchmail via stdin. The corresponding part in the DBMail design (a MDA that receives messages via stdin) seems to be dbmail-smtp (in the diagram it says local mail delivery via a UNIX pipe goes there.)

So maybe you could replace this part in .fetchmailrc:

and wants mda "/usr/bin/procmail -d %T"

with something like this:

and wants mda "/path-to/dbmail-smtp <arguments>"

If dbmail-smtp wants the local username of the user to whom’s mailbox to deliver the emails to as an argument, it seems you can use that %T modifier for that.

Lao Ming Liu February 26, 2009 at 4:15 AM

I tried this but ran into difficulty when I did the c_rehash.

$ which c_rehash /opt/local/bin/c_rehash

$ c_rehash ~/.ssl/certs/ Doing /Users/robbins/.ssl/certs/ WARNING: equifax.pem does not contain a certificate or CRL: skipping WARNING: gmailpop.pem does not contain a certificate or CRL: skipping

I made sure that the directories down from my homedir were all 775 and I know that I cut and pasted the certs as instructed.

Ali Rantakari February 26, 2009 at 2:44 PM

Hi Liu,

I can think of a couple things you might want to check:

if you’re using OS X, try using the standard c_rehash in /usr/bin/.
try the -verbose argument to try and get more info about the failure

antónio August 8, 2009 at 7:11 PM

I’m having the same problem as Liu, but i’m a recent convert to osx and i don’t know how to use the c_rehash in /usr/bin/

also, i’ve tried using the -v when using fetchmail and all i’ve got is SMTP transaction error while fetching. Can you help me out?

antónio August 9, 2009 at 4:31 AM

I’m sorry to have commented without proper testing. Turns out you have to be really careful when saving the pem files, as linebreaks and spacing are more than enough to result in a non-valid certificate. Currently i’m on a 3g connection with capped bandwith, but as soon as i get home i’ll do further testing. As for now, it only downloads the last 54 messages I have.

Ali Rantakari August 9, 2009 at 11:29 AM

Hi António,

Have you made sure that you selected “Enable POP for all mail (even mail that’s already been downloaded)” in GMail’s settings, instead of “Enable POP for mail that arrives from now on”?

antónio August 11, 2009 at 9:29 AM

Yes, I had that enabled and on the setting it stated that all the mail since ‘06 would be downloaded. It seemed to work on a windows machine I’m about to turn off in a few days. Turns out I had to reset the same setting so it would download everything. I got it to work with thunderbird as well (though some messages are appearing with duplicates), but I’m still a newbie as far as launchd and launchctl are concerned. I’ve got the .plist file according to Apple’s and I’ve loaded the job, but it hasn’t updated yet.

Anuj Gakhar September 9, 2009 at 6:10 PM

Why not just map your ThunderBird to use IMAP to get email from Gmail and make it available for offline access. I have done that and as long as I have ThunderBird open, I have a exact mirror copy of my gmail account in Thunderbird, available offline.

Ali Rantakari September 9, 2009 at 8:19 PM

Hi Anuj,

The reason I didn’t do it that way is because I wanted to automate this process and have it run in the background instead of having to remember to open Thunderbird and refresh it at regular intervals. But of course if you use Thunderbird as your main mail client anyway, this is a good tip if it indeed accomplishes the same thing.

Christian Bender September 12, 2009 at 10:07 PM

Ich denke ich habe eigentlich alles wie hier beschrieben gemacht. Aber es klappt leider nicht. “… fetchmail: Server certificate verification error: unable to get local issuer certificate fetchmail: Certificate/fingerprint verification was somehow skipped! fetchmail: SSL connection failed. …” Liegt das an Gmail oder an Snow Leopard oder an mir???

liam November 8, 2009 at 4:33 PM

Great post!! Thanks so much!

Gary November 20, 2009 at 8:17 AM

Hi, I have the same symptom as Christian Bender. When I run the openssl command, I get a certificate chain with two certificates in it. I’ve tried using each one separately as my gmailpop.pem, and both of them get hashed alright, but fetchmail complains.

Which of the certificates in the chain are we supposed to use? Or are we supposed to combine them somehow?

Thanks for your help with this. I tried getting backup-gmail to work, but it’s closed-source and not designed for Mac OS X. Your solution seems superior.

Parnell Springmeyer February 2, 2010 at 5:53 AM

Excellent article! It is thorough, well written, and helpful.

Wanted to also comment and say that this method works flawlessly with 10.6 (I didn’t think 10.6 would break anything, but you never know).

Jim April 15, 2010 at 3:31 AM

You may need to edit c_rehash or create a link to make it work. On my new osx box it referenced /usr/bin/perl5 which doesnt exist, /usr/bin/perl linked to the newest version (>5) so I just enabled root and removed the ‘5’ from c_rehash and things worked great.

Thanks for the way to automate gmail backups.

Ali Rantakari June 16, 2010 at 12:20 AM

I’ve updated the post with some info about specifying an alternate location for the mail spool file.

enkerli.wordpress.com/ September 27, 2010 at 9:24 AM

Same issue as Christian and Gary. Eventually used Gmail-Backup instead, but this does look like a better solution, in a few respects. One thing I find kind of surprising, at this point, is that Google doesn’t offer an elegant solution for backups, with data portability in mind. At any rate, some cues as to solve this SSL problem would be appreciated. Just in case it matters: I’m using a non-admin user account, it seems the username shouldn’t include “@gmail.com,” I got the file with two certificates by piping the command’s output to a file (I then deleted lines and saved in the right location), and my Gmail account has been allowing POP for quite a while.

Bas Ben Zineb July 27, 2011 at 2:35 PM

Big Tnx! After being discouraged by some negative posts about the fetchmail route this really did it for me. Very elegant

@Jim, this was the final step I needed. Do I put the ‘5’ back again?

Mark July 28, 2011 at 11:41 AM

Thanks for this, it got me most of the way to a full backup (3.5G ish done, about 1G to go).

I had to embed my ‘fetchmail’ script into a loop for repeated downloading, with a five minute delay between loops to get there, but….

However, it’s now not working. I’m getting lots of ’not flushed’ messages - and so nothing else is downloading at all.

Any clues?

Mutt, OpenGPG, and GMAIL - ToValhalla May 12, 2013 at 6:28 PM

[…] [7] Rantakari, Ali. “Gmail Backups with fetchmail on OS X.” Viewed on 040911 from http://hasseg.org/blog/post/161/gmail-backups-with-fetchmail-on-os-x/ […]