Zimbra Server Migration and Zimbra Account Transfer – The Perfect Method

1. Introduction

Zimbra project doesn’t have a cross migration or proper account transfer documentation. All they tell is to do copy the folder /opt/zimbra to your new servers. But if any of those files infected with a rootkit or other malicious scripts , then your new server also will be compromised. So never sync or copy the entire directory of your zimbra installation. Zimbra also tell you to upgrade your production server to the latest version before migration. But improper upgrade may result in entire data lose. With this procedure you can do:

  • Migrate zimbra from one Operating System To another.
  • Migrate zimbra account between any hardware and Operating systems configurations.
  • No interruption on production server like software upgrade or service disable.
  • Migrate zimbra from old version to a new version server
  • Zimbra cross migrations without copying entire directories.

2. Requirement

You need an old server with zmibra account and a new fresh server with the Os you wish. Dont’ create or make any custom configuration or setting in you new server. Please make sure to set your new servers hostname same as the old one.

  • Old server
    • Need ssh root login
    • Need zimbra admin logins
    • Enough HDD space to store backups
  • New server
    • Must be installed with latest stable zimbra
    • Need ssh root logins
    • Need zimbra admin logins
    • Enough HDD space to store backups

3. Presetup

You need to setup an ssh key from the new server’s root account to the old server’s root account.
Reduce the TTL of MX records of your domain to 500 seconds . So that you can easily switch the domain’s IP after migration. Please remember to schedule the migration task on non peek hours.

Create a directory in both new and old server into which we store all required files and data for doing the migration

[root@zimbra ~]# mkdir /backups/zmigrate
[root@zimbra ~]# chown zimbra.zimbra /backups/zmigrate
[root@zimbra ~]# su - zimbra

All operation in your Zimbra server must be performed as Zimbra user itself, otherwise you will get permission and ownership issues in your zimbra server

4. Backup all data from Old server

We are going to copy all data from old server without interrupting the services.

4.1 Find all domains

You need to find all the domains from your old server. We will store the domain list in a file called domains.txt. You need to back all the domains list as follows,

zimbra@zimbra:~$ cd /backups/zmigrate
zimbra@zimbra:/backups/zmigrate$ zmprov gad > domains.txt
zimbra@zimbra:/backups/zmigrate$ cat domains.txt

Now remove all domains and subdomains related with the main hostname of your server foo.com , from this list (domains.txt) because it was already created in your new server. So there is no need to create a new domain with the same name.

4.2 Find all admin accounts

Most of these servers will have only one admin. But some servers have multiple admins. So it will be good to find all admin accounts. We will store the admins list in admins.txt

zimbra@zimbra:/backups/zmigrate$ zmprov gaaa > admins.txt
zimbra@zimbra:/backups/zmigrate$ cat admins.txt

4.3 Find all email accounts

Next step is to find all the email accounts hosted in your old server. Get a list of your email accounts and save in the file emails.txt . So from this file we can see how many accounts that need to migrate.

zimbra@zimbra:/backups/zmigrate$ zmprov -l gaa >emails.txt
zimbra@zimbra:/backups/zmigrate$ cat emails.txt

Please remove all the email accounts from the file /backups/zmigrate/emails.txt with a starting words like spam, virus, ham, galsync . There is no need to restore these accounts. Even if you still need to restore , you can do it. I don’t like spam and virus emails.

4.4 Get all distribution lists

You need to get all the distributions list and store it in a file called distributinlist.txt.

zimbra@zimbra:~$ zmprov gadl > /backups/zmigrate/distributinlist.txt
zimbra@zimbra:~$ cat /backups/zmigrate/distributinlist.txt

4.5 Get all members in distribution lists

In this step we are going to collect all members in each of these distributions. We will create a folder called distributinlist_members and create a file under this folder named distributinlist.txt , then store all the distributions members.

zimbra@zimbra:~$ mkdir /backups/zmigrate/distributinlist_members
zimbra@zimbra:~$ for i in `cat /backups/zmigrate/distributinlist.txt`; do zmprov gdlm $i > /backups/zmigrate/distributinlist_members/$i.txt ;echo "$i"; done

4.6 Find all email account’s passwords

Now need to find the encrypted password of all of your old email accounts and store it under a folder named userpass/ as follows:

zimbra@zimbra:/backups/zmigrate$ mkdir userpass
zimbra@zimbra:/backups/zmigrate$ for i in `cat emails.txt`; do zmprov  -l ga $i userPassword | grep userPassword: | awk '{ print $2}' > userpass/$i.shadow; done

4.7 Backup all user names , Display names and Given Names

Zimbra will accept a Names and Disaplay names in email accounts during account creation. So we need to restore those data too. We will create a directory called userdata/ which contains these details of each of those email accounts

zimbra@zimbra:/backups/zmigrate$ mkdir userdata
zimbra@zimbra:/backups/zmigrate$ for i in `cat emails.txt`; do zmprov ga $i  | grep -i Name: > userdata/$i.txt ; done

4.8 Now backup all email account

This will take some time to take backup of all email accounts. So you can run this command behind “screen”. A tgz file will be created with each emails name. We will use this files to transfer email accounts.

zimbra@zimbra:/backups/zmigrate$ for email in `cat /backups/zmigrate/emails.txt`; do for i in `cat ../emails.txt `; do zmmailbox -z -m $i getRestURL '/?fmt=tgz' > $i.tgz ;  echo $email ; done

This tgz files contains

  • Mail
  • Contacts
  • Calendars
  • Briefcase
  • Tasks
  • Searches
  • Tags
  • Folders

All subfolders are included, except Junk and Trash. There is no way to include these in the big dump, but they can be exported separately:

4.9 Now backup alias

Some times your server may have email aliases for certain accounts. So you need to copy those aliases too. We will create a sub folder called alias/ for storing the backup of Alias.

zimbra@zimbra:/backups/zmigrate$ mkdir -p alias/
zimbra@zimbra:/backups/zmigrate$ for i in `cat emails.txt`; do zmprov ga  $i | grep zimbraMailAlias |awk '{print $2}' > alias/$i.txt ;echo $i ;done

Some of your email accounts don’t have alias. So the above created files may be an empty file. Remove those empty files as follows,’\

zimbra@zimbra:/backups/zmigrate$ find alias/ -type f -empty | xargs -n1 rm -v 

4.10 Rsync folder to new server

Now we have all the required data to do the migration process. As a summery :

  • /backups/zmigrate – Have all the backups stored
  • /backups/zmigrate/domains.txt – Contains the domains names
  • /backups/zmigrate/emails.txt – Contains the list of email accounts
  • /backups/zmigrate/distributinlist.txt – Contains the distribution lists
  • /backups/zmigrate/distributinlist_members – Contains the members in each of your distributions
  • /backups/zmigrate/userpass – Contains the encrypted password of your email accounts
  • /backups/zmigrate/userdata – containts the email accounts user informations
  • /backups/zmigrate/alias – Contains all the aliases of your email accounts

Also the parent folder /backups/zmigrate contains a lot of zip file which are the data inside emails.

Now rsync the files as follows,

root@newserver # rsync -avp -e 'ssh -p 22' root@old-server-ip:/backups/zmigrate /backups/

5. Restore in new server

So after finishing the rsync process , we need to restore this in your new server as follows:

All this operations must be carried out as zimbra sudo user itself. Don’t use root account to store the backups

[root@zimbra ~]# su - zimbra

5.1 Restore all domains

Now create all the domains that we have from the file /backups/zmigrate/domains.txt

[zimbra@zimbra zmigrate]$ for i in `cat /backups/zmigrate/domains.txt `; do  zmprov cd $i zimbraAuthMech zimbra ;echo $i ;done

You can also verify the domains created from the zimbra admin panel too

5.2 Create email accounts and set the old password

We need to create the email accounts for storing the mails. We also need to set the old passwords too. We already collected the account info and passwords.

To Create email accounts and restore passwords . Please use the following script to create it

#Scrit  for creating the email accounts createacct.sh
for i in `cat $USERS`
givenName=$(grep givenName: $USERDDATA/$i.txt | cut -d ":" -f2)
displayName=$(grep displayName: $USERDDATA/$i.txt | cut -d ":" -f2)
shadowpass=$(cat $USERPASS/$i.shadow)
zmprov ca $i CHANGEme cn "$givenName" displayName "$displayName" givenName "$givenName" 
zmprov ma $i userPassword "$shadowpass"

5.3 Restore email accounts

Now we are going to restore the emails from the Zip file. This process may take some hours. So it will be good to run behind “screen” command.

[zimbra@zimbra zmigrate]$ for i in `cat /backups/zmigrate/emails.txt`; do zmmailbox -z -m $i postRestURL "/?fmt=tgz&resolve=skip" /backups/zmigrate/$i.tgz ;  ; echo "$i -- finished "; done
gm@fun.com -- finished 
forest@fun.com -- finished 

5.4 Now recreate the distribution lists

It is time to recreate all the distribution lists as follows.

[zimbra@zimbra zmigrate]$ for i in `cat distributinlist.txt`; do zmprov cdl $i ; echo "$i -- done " ; done
budgetrtodomainusers@fun.com -- done 

5.5 Restore the distribution lists

After creating the distribution lists we need to add all the members inside the distribution lists. We have the distribution lists in the folder distributinlist_members/ and the list is in distributionlist.txt file. Please use the following small script to restore the distribution lists.

[zimbra@zimbra zmigrate]$ cat restoredist.sh 
# add all memebers to each of these distribution lists
for i in `cat distributinlist.txt`
	for j in `grep -v '#' distributinlist_members/$i.txt |grep '@'` 
	zmprov adlm $i $j
	echo " $j member has been added to list $i"


5.6 Restore Alias accounts

Please use the following script to restore alias. This will add all the aliases in your email accounts.

for i in `cat /backups/zmigrate/emails.txt`
	if [ -f "alias/$i.txt" ]; then
	for j in `grep '@' /backups/zmigrate/alias/$i.txt`
	zmprov aaa $i $j
	echo "$i HAS ALIAS $j --- Restored"

6. Conclusion

So now we migrated all our email accounts. It is time for DNS change. You need to shut down the old zimbra services and change the DNS. After that send some test emails and make sure everything is working fine. Next step is to secure your zimbra server. You need to install ssl certificates and firewall in your new zimbra server. Now you have a new server with new packages and files with the same old email accounts and its data.

7. References


Zimbra Server Migration

[SOLVED]CRITICAL! LDAP primary MDB database is 90% full in Zimbra

Zimbra logo

You may get this type errors as email in the zimbra admin email account. This was related with the zimbra 8.x changes in ldap db configuration. You may probably see this issue during a zimbra migration or restore task
First check the following values in your zimbra

#su - zimbra
#zmlocalconfig -s ldap_db_maxsize
ldap_db_maxsize = 85899345920
#zmlocalconfig -s ldap_accesslog_maxsize
ldap_accesslog_maxsize = 85899345920

You can see the db size and accesslog size is same that is 80 GB , Now please setup the size as follows,

# zmlocalconfig -e ldap_db_maxsize=85899345920

# zmlocalconfig -e ldap_accesslog_maxsize=10737418240

Give some time for applying the changes

# sleep 90

Now stop slapd

# ldap stop

Go to database directory

# cd /opt/zimbra/data/ldap/mdb/db

Take a backup of ldap database

# /opt/zimbra/libexec/zmslapcat /opt/zimbra/data/ldap/mdb/db

Rename the original db to .old

# mv data.mdb data.mdb.old

Recreate database with new size and restore from backup

# /opt/zimbra/openldap/sbin/slapadd -q -b "" -F /opt/zimbra/data/ldap/config -l /opt/zimbra/data/ldap/mdb/db/ldap.bak

Now start slapd

# ldap start

Tar the original if you need

# gzip data.mdb.old

The above will fix your issue

How to configure Zimbra + CSF – The Best Zimbra Firewall Configuration

CSF is one of the best opensource firewalls that using in most of the hosting servers like cPanel and Directadmin . Also it is one of the best firewall for installing Zimbra Mail server . This documentation will help you to configure the CSF firewall in a Zimbra Standalone installation server.

Before starting the installation , you may need to read the documentation available on http://wiki.zimbra.com/wiki/Ports , this will help you to get a quick understanding of ports that required to open in a Zimbra server.

Install CSF :

You can download CSF from http://configserver.com/cp/csf.html and install it . After that open the CSF configuration and enable the following ports,

TCP_IN = "22,25,53,80,110,143,443,465,587,993,995,7071"
TCP_OUT = "22,25,53,80,110,113,443,465,587,993,995,7071"

Now you need to open the file /etc/csf/csf.pignore and add the following zimbra packages paths.


This will help to white list these binaries in CSF

Now you can start the CSF as follows and test it.

# /etc/init.d/csf start

You may need to test the mail server and its functionalities . After that you can disable the testing mode in csf.conf and reload CSF. You can also perform other generic CSF tweaks after that.

Syslint Technologies provide all sort of Zimbra Technical Support and management services.

How to Configure Zimbra With External Email Address

You can add your external email accounts to your zimbra email  server from your end user  interface . But sometimes the testing from Zimbr -> Preferences -> Accounts  will fail with an   error like as follows,

2013-05-05 12:29:59,533 WARN  [qtp1347725365-191:https://x.x.x.xservice/soap/TestDataSourceRequest] [name=mail@yourdomain.com;mid=3;ip=;ua=ZimbraWebClient - FF17 (Linux)/8.0.3_GA_5664;] datasource - Test failed: DataSource{id=TestId, type=pop3, enabled=false, name=Test, host=yourdomain.com, port=110, connectionType=cleartext, username=mail@yourdomain.com, folderId=-1}
com.zimbra.common.service.ServiceException: system failure: Unable to connect to POP3 server: DataSource{id=TestId, type=pop3, enabled=false, name=Test, host=mail.yourdomain.com, port=110, connectionType=cleartext, username=mail@yourdomainc.om, folderId=-1}
at com.zimbra.common.service.ServiceException.FAILURE(ServiceException.java:258)
at com.zimbra.cs.datasource.Pop3Sync.connect(Pop3Sync.java:157)
at com.zimbra.cs.datasource.Pop3Sync.test(Pop3Sync.java:110)
at com.zimbra.cs.datasource.DataSourceManager.test(DataSourceManager.java:213)
at com.zimbra.cs.service.mail.TestDataSource.handle(TestDataSource.java:131)
at com.zimbra.soap.SoapEngine.dispatchRequest(SoapEngine.java:500)
at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.java:363)
at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.java:236)
at com.zimbra.soap.SoapServlet.doWork(SoapServlet.java:290)
at com.zimbra.soap.SoapServlet.doPost(SoapServlet.java:206)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at com.zimbra.cs.servlet.ZimbraServlet.service(ZimbraServlet.java:208)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:565)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1361)
at com.zimbra.cs.servlet.SetHeaderFilter.doFilter(SetHeaderFilter.java:57)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1332)
at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:77)
at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:181)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1332)
at com.zimbra.cs.servlet.ZimbraQoSFilter.doFilter(ZimbraQoSFilter.java:114)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1332)
at org.eclipse.jetty.servlets.DoSFilter.doFilterChain(DoSFilter.java:464)
at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:327)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1332)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:477)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:227)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1031)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:406)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:965)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:250)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:149)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111)
at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:312)
at org.eclipse.jetty.server.handler.DebugHandler.handle(DebugHandler.java:77)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111)
at org.eclipse.jetty.server.Server.handle(Server.java:349)
at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:452)
at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:894)
at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:948)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:857)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:77)
at org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:191)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:606)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:46)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:603)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:538)
at java.lang.Thread.run(Thread.java:722)

It solutions is as follows

1) Login to your zimbra sever over ssh and  do the following ,

# su - zimbra
# zmlocalconfig | grep javamail_pop3_enable_starttls

If you see the result of the above command as  ”  javamail_pop3_enable_starttls = true ” , you need to change it to false


# zmlocalconfig -e javamail_pop3_enable_starttls=false
# zmmailboxdctl restart

Now try to login to your zimbra and add the  email account  and test it. If  you still see an issue with ssl , please do the following too,

# zmlocalconfig | grep certs
  data_source_trust_self_signed_certs = false
  mailboxd_truststore = /opt/zimbra/java/jre/lib/security/cacerts
  ssl_allow_accept_untrusted_certs = true
  ssl_allow_mismatched_certs = true
  ssl_allow_untrusted_certs = false
# zmlocalconfig -e ssl_allow_untrusted_certs=true
# zmlocalconfig -e data_source_trust_self_signed_certs=true
# zmmailboxdctl restart

This will fix your issue with external email accounts in Zimbra.