eMxyzptlk

My hacker blog

Migrating Emails From Yahoo SMB to Google Apps

My friend asked me to help her migrate her emails from Yahoo SMB to Google Apps. Google Apps does offer importing emails from Yahoo (given the email is identified as hosted by Yahoo!), but the MX records has already been changed and the only option left was to import everything using POP which of course works mostly for the INBOX, and there are AFAIK absolutely no guarantees that all of the folders will be synced. I have not tried that myself.

There are some services or applications from the Google Apps Market Place that offer importing emails, however they were asking for a bit more permissions that I feel comfortable giving them, besides, due to the nature of the emails she has I felt that it’s better to keep third-party involvement to the bare minimum.

I decided to go with a different approach, call it manual approach, using iSync. iSync (the command line is actually called mbsync) is a tool that will help you keep a local Maildir and a remote IMAP server in sync. It is up to you to decide which direction the sync should go and what folders are you looking to synchronize.

To start, we need to write a few configuration files for mbsync, the first one is for pulling down the entire archive from the old provider (in this case Yahoo!)

To keep yourself sane, create a new folder dedicated for the migration. I called mine ~/migration and I kept everything I need for the migration in this folder.

Create ~/migration/mbsync-from-yahoo with the following content, change only the user and the password for now.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
IMAPAccount yahoo
Host imap.mail.yahoo.com
User user@example.com
Pass secret_password
UseIMAPS yes
CertificateFile /etc/ssl/certs/ca-certificates.crt

IMAPStore remote
Account yahoo

MaildirStore local
Path /home/user/migration/mail/

Channel yahoo
Master :remote:
Slave :local:
Patterns * !"Bulk Mail"
Create Slave
Expunge None
Sync Pull
SyncState *

Now execute mbsync --config ~/migration/mbsync-from-yahoo --list and analyze the list of folders. For each folder that you do not wish to sync, add it within double quotes and preceded by a bang (i.e !"Some folder") to the end of the line that starts with Patterns. The patterns define the pattern of the folders to sync, the ! prefix is a negation. For more information on that, please check man 1 mbsync.

Once done and have verified the list of folders, we need to start importing everything from Yahoo!. We normally only need to invoke mbsync --config ~/migration/mbsync-from-yahoo --all but Yahoo has a rate limit and it will eventually close the connection. Mbsync will stop the process and exit with an error. To help automate the process even more, let’s catch the errors by checking the exit code and restart the process if it’s greater than 0

1
while true; do mbsync --config ~/migration/mbsync-from-yahoo --all; if [[ $? -eq 0 ]]; then break; fi; done

It’s time to go for a walk, coffee or whatever your pleasure is, it will take some time, it took me close to 8 hours to download 65K emails.

… much later …

To be able to synchronize (or push) the emails to Google Apps servers, we need to go through few simple modifications to the Maildir that we just created, but before going through the changes we need to create a backup of the Maildir. Do so with tar czf mail.tar.gz mail. Now let’s modify the maildir:

  • Remove all of the files left by mbsync find ~/migration/mail -type f -name '.??*' -exec rm {} ';'
  • Create a new bash script ~/migration/remove_ids.sh with the following content and execute it
1
2
3
4
5
6
7
8
9
10
11
12
13
14
#!/bin/bash
# This script removes the ID from the email file.

set -ex

cd ~/migration/mail
for folder in *; do
  echo "${folder}"
  cd "${folder}"
  for file in `find . -type f -name '*U=*'`; do
    mv "${file}" "`echo $file | sed -e 's:,U=[0-9]*::g'`"
  done
  cd ..
done
  • (Optional) If you’d like to put everything under one label (using sub-labels) which would help you find all of the important emails later:
1
2
3
cd ~/migration/mail
mkdir -p Yahoo/{cur,new,tmp}
for i in *; do mv "${i}" "Yahoo/.${i}"; done

Do not worry about the error message “Cannot move Yahoo to a subdirectory of itself”, that is expected.

Finally, it’s time to send everything to Google Apps, now that we have everything set. Create the file ~/migration/mbsync-to-google-apps with the following contents and don’t forget to adjust the user-name and password field.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
IMAPAccount googleapps
Host imap.gmail.com
User user@example.com
Pass secret_password
UseIMAPS yes
CertificateFile /etc/ssl/certs/ca-certificates.crt

IMAPStore remote
Account googleapps

MaildirStore local
Path /home/user/migration/mail/

Channel googleapps
Master :remote:
Slave :local:
Patterns *
Create Master
Expunge None
Sync Push
SyncState *

Now run the mbsync with the command:

1
while true; do mbsync --config ~/migration/mbsync-to-google-apps --all; if [[ $? -eq 0 ]]; then break; fi; done

Even though Google Apps will not rate-limit you, it’s best to catch errors and restart without any manual intervention.

Blog Re-opened

After a few years in the dark, my blog has just been re-opened. It has been so long since my last blog entry that my life has took a turn to a very different yet exciting new path.

I am currently working at Google, as an SRE and most my daily basis work involves system troubleshooting. On the other hand, I’ve fallen in love with Go and it’s where I spend most of my free time.

I’ll be blogging a lot about systems and system administration, but I will also share everything that I know and will know about Go, so stay tuned!