Recommended way to pause postfix local delivery while taking snapshot for backup

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Recommended way to pause postfix local delivery while taking snapshot for backup

Dominic Raferd
Is there a best/recommended way to pause postfix local deliveries so that I can take an LVM snapshot of the local mails for backup purposes? The pause only has to be momentary, while the snapshot is taken, but the files need to be in a consistent state. If anyone also knows the way to pause Dovecot imap/pop3 similarly (as this could also be accessing the same files), that would be helpful too.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Recommended way to pause postfix local delivery while taking snapshot for backup

Robert Schetterer-2
Am 09.04.2017 um 10:52 schrieb Dominic Raferd:
> Is there a best/recommended way to pause postfix local deliveries so
> that I can take an LVM snapshot of the local mails for backup purposes?
> The pause only has to be momentary, while the snapshot is taken, but the
> files need to be in a consistent state. If anyone also knows the way to
> pause Dovecot imap/pop3 similarly (as this could also be accessing the
> same files), that would be helpful too.

why not simply write copies of all mails via bcc as archive and/or
use dsync or rsync to write backups with dovecot ?


Best Regards
MfG Robert Schetterer

--
[*] sys4 AG

http://sys4.de, +49 (89) 30 90 46 64
Schleißheimer Straße 26/MG, 80333 München

Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263
Vorstand: Patrick Ben Koetter, Marc Schiffbauer
Aufsichtsratsvorsitzender: Florian Kirstein
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Recommended way to pause postfix local delivery while taking snapshot for backup

Philip Paeps
In reply to this post by Dominic Raferd
On 2017-04-09 10:52:58 (+0200), Dominic Raferd <[hidden email]>
wrote:

> Is there a best/recommended way to pause postfix local deliveries so
> that I
> can take an LVM snapshot of the local mails for backup purposes? The
> pause
> only has to be momentary, while the snapshot is taken, but the files
> need
> to be in a consistent state. If anyone also knows the way to pause
> Dovecot
> imap/pop3 similarly (as this could also be accessing the same files),
> that
> would be helpful too.

You can kill two birds with one stone if you deliver mails using LMTP
instead of letting Postfix local(8) deliver directly to mailboxes.

As I understand it, Dovecot's LMTP implementation will ensure the mail
store is always in a consistent state: either a message will be stored
and indexed or it won't.

If you want to ensure no deliveries are attempted while you're taking a
snapshot, running LMTP over a TCP socket and blocking new connections
with the firewall would do it.  Postfix will queue messages for later
delivery when it can't connect to the LMTP server.

Philip

--
Philip Paeps
Senior Reality Engineer
Ministry of Information
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Recommended way to pause postfix local delivery while taking snapshot for backup

Wietse Venema
In reply to this post by Dominic Raferd
Dominic Raferd:
> Is there a best/recommended way to pause postfix local deliveries so that I
> can take an LVM snapshot of the local mails for backup purposes? The pause
> only has to be momentary, while the snapshot is taken, but the files need
> to be in a consistent state. If anyone also knows the way to pause Dovecot
> imap/pop3 similarly (as this could also be accessing the same files), that
> would be helpful too.

# postconf defer_transports=xxx
# postfix reload

For a suitable value of xxx (name in master.cf).

This tells the queue manager to stop generating delivery requests.
It does not suspend a delivery request that is already in progress;
by design, Postfix has no single program that knows 'everything'.

As documented, defer_transports also does not block forced delivery
commands such as "postfix flush" and "sendmail -q".

        Wietse
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Recommended way to pause postfix local delivery while taking snapshot for backup

Viktor Dukhovni
In reply to this post by Dominic Raferd

> On Apr 9, 2017, at 4:52 AM, Dominic Raferd <[hidden email]> wrote:
>
> Is there a best/recommended way to pause postfix local deliveries so that
> I can take an LVM snapshot of the local mails for backup purposes?

The "best" way is to use a file system that supports snapshots, such as
"zfs".  Otherwise, what Wietse said about "defer_transports".  Covers
the Postfix part.

> The pause only has to be momentary, while the snapshot is taken, but the
> files need to be in a consistent state. If anyone also knows the way to
> pause Dovecot imap/pop3 similarly (as this could also be accessing the
> same files), that would be helpful too.

You'd have to stop Dovecot, terminating all active connections, or use
a file-system that supports snapshots.

With maildir, you can probably not worry too much about consistency,
and just do periodic rsync without stopping Postfix or Dovecot.

--
        Viktor.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Recommended way to pause postfix local delivery while taking snapshot for backup

Dominic Raferd
Thanks to all for the replies. This is an existing somewhat elderly setup and I don't want to make major changes (file system, LMTP etc) at this stage. For postfix, Wietse's suggestion should work nicely. I guess I should build in a delay of a second or two after issuing the command to ensure that any local deliveries in progress are completed before the snapshot is taken - this is not a very big/busy server. For dovecot well I will just stop it. So something like this:

postconf defer_transports=local
postfix reload
sleep 2s
/etc/init.d/dovecot stop # (this is on Debian 6.0, no systemd)
# take lvm snapshot here, then:
postconf defer_transports=
postfix reload
/etc/init.d/dovecot start
# backup from snapshot can now proceed

On 9 April 2017 at 15:29, Viktor Dukhovni <[hidden email]> wrote:

> On Apr 9, 2017, at 4:52 AM, Dominic Raferd <[hidden email]> wrote:
>
> Is there a best/recommended way to pause postfix local deliveries so that
> I can take an LVM snapshot of the local mails for backup purposes?

The "best" way is to use a file system that supports snapshots, such as
"zfs".  Otherwise, what Wietse said about "defer_transports".  Covers
the Postfix part.

> The pause only has to be momentary, while the snapshot is taken, but the
> files need to be in a consistent state. If anyone also knows the way to
> pause Dovecot imap/pop3 similarly (as this could also be accessing the
> same files), that would be helpful too.

You'd have to stop Dovecot, terminating all active connections, or use
a file-system that supports snapshots.

With maildir, you can probably not worry too much about consistency,
and just do periodic rsync without stopping Postfix or Dovecot.

--
        Viktor.


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Recommended way to pause postfix local delivery while taking snapshot for backup

Peter Ajamian
In reply to this post by Dominic Raferd
On 09/04/17 20:52, Dominic Raferd wrote:
> Is there a best/recommended way to pause postfix local deliveries so
> that I can take an LVM snapshot of the local mails for backup purposes?
> The pause only has to be momentary, while the snapshot is taken, but the
> files need to be in a consistent state. If anyone also knows the way to
> pause Dovecot imap/pop3 similarly (as this could also be accessing the
> same files), that would be helpful too.

I'm fairly sure you can just snapshot and any mail that has at least
been queued will be safe, either in the queue or delivered.  There is a
small chance that the snapshot will be taken in between the point where
the message is delivered and when it is removed from the queue in which
case you can end up with a duplicate of that message when you restore as
postfix will see the message in the queue and deliver it again.

This is all minor in comparison to the mail that would be lost after the
snapshot is taken and when the system dies before you restore (after the
system dies you're actually safe because sending MXes will defer their
mail to you).  No amount of pausing during the snapshot will recover
mail that you accepted after the snapshot.

Even if you were to loose any mail that was in postfix during the
snapshot (which won't happen) it's still negligible compared to what
you'll loose after the snapshot.  So at the end of the day I really
wouldn't worry so much about this.  Take your snapshot, and take your
backup and understand that if you ever get to the point where everything
blows up you *will* loose some mail, there is no real way around that.


Peter
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Recommended way to pause postfix local delivery while taking snapshot for backup

lists@lazygranch.com
My 2 cents. 

I had a Digital Ocean snapshot that turned out to be unbootable. (I have no explanation.) My snapshot procedure since then is I shut down all the services, power down, take the snapshot and restore from that snapshot. 

Me paranoid? Yep. I also keep two snapshots, both of which have been proven to be bootable. 

You would think an image of a droplet that was running should be bootable, but real life can be full of surprises. 
 
Regarding rsync, at least you won't lose user data and configuration files. But restoration from rsync still sounds like work to me.

Digital Ocean had one situation where a backup was bad. 

https://murze.be/2016/02/today-digitalocean-lost-our-entire-server/

‎I'm picking on Digital Ocean, but Vultr and Linode have similar incidents. 

If my flow was to do powered (operating) images...well I wouldn't. I'd go for rsync. I would have a script scheme to rebuild (reconfigure) the OS from a stock droplet.

 





Loading...