Quantcast

load balanced emails servers pair

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

load balanced emails servers pair

Patrick Chemla
Hi,

I have a running Fedora 24 emails server using postfix 3.1.3, with courier.

I wonder how to build a pair of MTAs to secure emails at all time,
having 2 servers receiving the emails, and users could connect to either
server to get emails, maybe on a load balanced way.

Problems are with synchronization when receiving emails from outside, or
emails read, emails moved,....

Does someone have a good guide, howto, doc to achieve this?

Thanks for help.

Patrick


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: load balanced emails servers pair

Wietse Venema
Patrick Chemla:

> Hi,
>
> I have a running Fedora 24 emails server using postfix 3.1.3, with courier.
>
> I wonder how to build a pair of MTAs to secure emails at all time,
> having 2 servers receiving the emails, and users could connect to either
> server to get emails, maybe on a load balanced way.
>
> Problems are with synchronization when receiving emails from outside, or
> emails read, emails moved,....

You need a redundant message store. In pre-cloud times, people
would use a NAS filer with redundant disks, store email as maildir
files (one per message) and MDAs would mount that store via NFS.
Perhaps that model still works for you.

> Does someone have a good guide, howto, doc to achieve this?
>
> Thanks for help.
>
> Patrick
>
>
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: load balanced emails servers pair

Patrick Chemla
Hi Wietse,

Of course I thought about such NAS solution, but I wanted to check if
there is a way with 2 separate disks, with a kind of that could be aware
of emails files changes.

Actually, the mail server run onto a VM, on a big server. I have another
big server with same emails VM, and I just rsync --delete --update from
the first one to the second. So I have a full image copy every 5
minutes, but only one real MTA.

I will check the NAS option, if there is no other way.

Thanks
Patrick

Le 24/01/2017 à 13:45, Wietse Venema a écrit :

> Patrick Chemla:
>> Hi,
>>
>> I have a running Fedora 24 emails server using postfix 3.1.3, with courier.
>>
>> I wonder how to build a pair of MTAs to secure emails at all time,
>> having 2 servers receiving the emails, and users could connect to either
>> server to get emails, maybe on a load balanced way.
>>
>> Problems are with synchronization when receiving emails from outside, or
>> emails read, emails moved,....
> You need a redundant message store. In pre-cloud times, people
> would use a NAS filer with redundant disks, store email as maildir
> files (one per message) and MDAs would mount that store via NFS.
> Perhaps that model still works for you.
>
>> Does someone have a good guide, howto, doc to achieve this?
>>
>> Thanks for help.
>>
>> Patrick
>>
>>
>>

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: load balanced emails servers pair

Patrick Domack
This would not be a good thing to do, as deleted email will magically  
reappear.

Using unison to sync it worked for me, over 10years ago. But these  
days, just use dsync part of dovecot, and your life will be happy.


Quoting Patrick Chemla <[hidden email]>:

> Hi Wietse,
>
> Of course I thought about such NAS solution, but I wanted to check  
> if there is a way with 2 separate disks, with a kind of that could  
> be aware of emails files changes.
>
> Actually, the mail server run onto a VM, on a big server. I have  
> another big server with same emails VM, and I just rsync --delete  
> --update from the first one to the second. So I have a full image  
> copy every 5 minutes, but only one real MTA.
>
> I will check the NAS option, if there is no other way.
>
> Thanks
> Patrick
>
> Le 24/01/2017 à 13:45, Wietse Venema a écrit :
>> Patrick Chemla:
>>> Hi,
>>>
>>> I have a running Fedora 24 emails server using postfix 3.1.3, with courier.
>>>
>>> I wonder how to build a pair of MTAs to secure emails at all time,
>>> having 2 servers receiving the emails, and users could connect to either
>>> server to get emails, maybe on a load balanced way.
>>>
>>> Problems are with synchronization when receiving emails from outside, or
>>> emails read, emails moved,....
>> You need a redundant message store. In pre-cloud times, people
>> would use a NAS filer with redundant disks, store email as maildir
>> files (one per message) and MDAs would mount that store via NFS.
>> Perhaps that model still works for you.
>>
>>> Does someone have a good guide, howto, doc to achieve this?
>>>
>>> Thanks for help.
>>>
>>> Patrick
>>>
>>>
>>>



Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: load balanced emails servers pair

Eero Volotinen-2
how about mounting ceph or glusterfs disk to
message store?

eero

25.1.2017 5.18 ap. "Patrick Domack" <[hidden email]> kirjoitti:
This would not be a good thing to do, as deleted email will magically reappear.

Using unison to sync it worked for me, over 10years ago. But these days, just use dsync part of dovecot, and your life will be happy.


Quoting Patrick Chemla <[hidden email]>:

Hi Wietse,

Of course I thought about such NAS solution, but I wanted to check if there is a way with 2 separate disks, with a kind of that could be aware of emails files changes.

Actually, the mail server run onto a VM, on a big server. I have another big server with same emails VM, and I just rsync --delete --update from the first one to the second. So I have a full image copy every 5 minutes, but only one real MTA.

I will check the NAS option, if there is no other way.

Thanks
Patrick

Le 24/01/2017 à 13:45, Wietse Venema a écrit :
Patrick Chemla:
Hi,

I have a running Fedora 24 emails server using postfix 3.1.3, with courier.

I wonder how to build a pair of MTAs to secure emails at all time,
having 2 servers receiving the emails, and users could connect to either
server to get emails, maybe on a load balanced way.

Problems are with synchronization when receiving emails from outside, or
emails read, emails moved,....
You need a redundant message store. In pre-cloud times, people
would use a NAS filer with redundant disks, store email as maildir
files (one per message) and MDAs would mount that store via NFS.
Perhaps that model still works for you.

Does someone have a good guide, howto, doc to achieve this?

Thanks for help.

Patrick






Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: load balanced emails servers pair

Patrick Domack
All options, assuming your imap/pop/lmtp are compatable and friendly using it.

I know dovecot you should only access a mailstore from one host at a  
time, don't just randomly balance things, or it can corrupt the index  
files.

Quoting Eero Volotinen <[hidden email]>:

> how about mounting ceph or glusterfs disk to
> message store?
>
> eero
>
> 25.1.2017 5.18 ap. "Patrick Domack" <[hidden email]> kirjoitti:
>
>> This would not be a good thing to do, as deleted email will magically
>> reappear.
>>
>> Using unison to sync it worked for me, over 10years ago. But these days,
>> just use dsync part of dovecot, and your life will be happy.
>>
>>
>> Quoting Patrick Chemla <[hidden email]>:
>>
>> Hi Wietse,
>>>
>>> Of course I thought about such NAS solution, but I wanted to check if
>>> there is a way with 2 separate disks, with a kind of that could be aware of
>>> emails files changes.
>>>
>>> Actually, the mail server run onto a VM, on a big server. I have another
>>> big server with same emails VM, and I just rsync --delete --update from the
>>> first one to the second. So I have a full image copy every 5 minutes, but
>>> only one real MTA.
>>>
>>> I will check the NAS option, if there is no other way.
>>>
>>> Thanks
>>> Patrick
>>>
>>> Le 24/01/2017 à 13:45, Wietse Venema a écrit :
>>>
>>>> Patrick Chemla:
>>>>
>>>>> Hi,
>>>>>
>>>>> I have a running Fedora 24 emails server using postfix 3.1.3, with
>>>>> courier.
>>>>>
>>>>> I wonder how to build a pair of MTAs to secure emails at all time,
>>>>> having 2 servers receiving the emails, and users could connect to either
>>>>> server to get emails, maybe on a load balanced way.
>>>>>
>>>>> Problems are with synchronization when receiving emails from outside, or
>>>>> emails read, emails moved,....
>>>>>
>>>> You need a redundant message store. In pre-cloud times, people
>>>> would use a NAS filer with redundant disks, store email as maildir
>>>> files (one per message) and MDAs would mount that store via NFS.
>>>> Perhaps that model still works for you.
>>>>
>>>> Does someone have a good guide, howto, doc to achieve this?
>>>>>
>>>>> Thanks for help.
>>>>>
>>>>> Patrick
>>>>>
>>>>>
>>>>>
>>>>>
>>
>>
>>



Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: load balanced emails servers pair

rightkicktech.gmail.com
A shared storage with glusterfs seems a nice approach.
In this way, it doesn't matter which server receives the mail, as long as the MDAs of each server write on the shared storage.

Alex

On January 25, 2017 6:08:59 PM EET, Patrick Domack <[hidden email]> wrote:
All options, assuming your imap/pop/lmtp are compatable and friendly using it.

I know dovecot you should only access a mailstore from one host at a
time, don't just randomly balance things, or it can corrupt the index
files.

Quoting Eero Volotinen <[hidden email]>:

how about mounting ceph or glusterfs disk to
message store?

eero

25.1.2017 5.18 ap. "Patrick Domack" <[hidden email]> kirjoitti:

This would not be a good thing to do, as deleted email will magically
reappear.

Using unison to sync it worked for me, over 10years ago. But these days,
just use dsync part of dovecot, and your life will be happy.


Quoting Patrick Chemla <[hidden email]>:

Hi Wietse,

Of course I thought about such NAS solution, but I wanted to check if
there is a way with 2 separate disks, with a kind of that could be aware of
emails files changes.

Actually, the mail server run onto a VM, on a big server. I have another
big server with same emails VM, and I just rsync --delete --update from the
first one to the second. So I have a full image copy every 5 minutes, but
only one real MTA.

I will check the NAS option, if there is no other way.

Thanks
Patrick

Le 24/01/2017 à 13:45, Wietse Venema a écrit :

Patrick Chemla:

Hi,

I have a running Fedora 24 emails server using postfix 3.1.3, with
courier.

I wonder how to build a pair of MTAs to secure emails at all time,
having 2 servers receiving the emails, and users could connect to either
server to get emails, maybe on a load balanced way.

Problems are with synchronization when receiving emails from outside, or
emails read, emails moved,....

You need a redundant message store. In pre-cloud times, people
would use a NAS filer with redundant disks, store email as maildir
files (one per message) and MDAs would mount that store via NFS.
Perhaps that model still works for you.

Does someone have a good guide, howto, doc to achieve this?

Thanks for help.

Patrick










--
Sent from my Android device with K-9 Mail. Please excuse my brevity.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: load balanced emails servers pair

Wietse Venema
rightkicktech.gmail.com:
> A shared storage with glusterfs seems a nice approach.
> In this way, it doesn't matter which server receives the mail, as long as the MDAs of each server write on the shared storage.

Just a note of caution: Postfix performance is dominated by small-file
I/O (many small short-lived files). If you want to compare file systems,
I suggest running the smallfile benchmark.

        Wietse
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

No worthy mechs found

Martin Brampton
I have several servers, all configured to send mail by relaying through
my main mail server on port 587. SFAIK they are all configured the same,
and use TLS with plain text authentication. They all have identical
sasl_passwd and sasl_passwd.db files. Security options are:
smtp_sasl_security_options = noanonymous

All but one of them works. The one that doesn't logs the "No worthy
mechs found" message and SASL authentication failed message. The main
mail server (to which it is trying to connect) logs that the problem
server has established a TLS 1.2 connection, but then disconnects and
does nothing more.

What should I look for to fix this?


Fuller configuration details:
root@named:/etc/postfix# postconf -n
alias_database = hash:/etc/aliases
alias_maps = hash:/etc/aliases
append_dot_mydomain = no
biff = no
config_directory = /etc/postfix
inet_interfaces = loopback-only
inet_protocols = ipv4
mailbox_size_limit = 0
mydestination =
myhostname = named.black-sheep-research.com
mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128
myorigin = $mydomain
readme_directory = no
recipient_delimiter = +
relayhost = [mx.yorkshiremail.com]:587
smtp_sasl_auth_enable = yes
smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
smtp_sasl_security_options = noanonymous
smtp_tls_CAfile = /etc/ssl/certs/ca-certificates.crt
smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache
smtp_use_tls = yes
smtpd_banner = $myhostname ESMTP $mail_name (Raspbian)
smtpd_tls_cert_file = /etc/ssl/certs/ssl-cert-snakeoil.pem
smtpd_tls_key_file = /etc/ssl/private/ssl-cert-snakeoil.key
smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache
smtpd_use_tls = yes


smime.p7s (5K) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: No worthy mechs found

Viktor Dukhovni
On Sun, Jan 29, 2017 at 06:35:46PM +0000, Martin Brampton wrote:

> I have several servers, all configured to send mail by relaying through my
> main mail server on port 587. SFAIK they are all configured the same, and
> use TLS with plain text authentication. They all have identical sasl_passwd
> and sasl_passwd.db files.

Do they all have the requisite SASL plugin modules?

> All but one of them works. The one that doesn't logs the "No worthy mechs
> found" message and SASL authentication failed message.

Most likely the SASL "plain" plugin is not installed.

--
        Viktor.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: No worthy mechs found

Martin Brampton
Thanks for the pointer. It seems that libsasl2-modules was missing.
Since I followed the same procedure in all cases, I guess it is the
result of a different image from a different VPS provider.


On 29/01/2017 19:17, Viktor Dukhovni wrote:

> On Sun, Jan 29, 2017 at 06:35:46PM +0000, Martin Brampton wrote:
>
>> I have several servers, all configured to send mail by relaying through my
>> main mail server on port 587. SFAIK they are all configured the same, and
>> use TLS with plain text authentication. They all have identical sasl_passwd
>> and sasl_passwd.db files.
>
> Do they all have the requisite SASL plugin modules?
>
>> All but one of them works. The one that doesn't logs the "No worthy mechs
>> found" message and SASL authentication failed message.
>
> Most likely the SASL "plain" plugin is not installed.
>


smime.p7s (5K) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: load balanced emails servers pair

Patrick Chemla
In reply to this post by rightkicktech.gmail.com
Thanks all for your answers.

I have at last setup the NAS, and mails are received there.

So I will set the second server and second MTA, and both will receive emails.

Next step is to give users access to both servers to retreive emails.

As a load-balancer could help easily for http/https access, how to deal with IMAP ports? How to load-balance IMAP ports?

Thanks
Patrick

Le 29/01/2017 à 14:29, rightkicktech.gmail.com a écrit :
A shared storage with glusterfs seems a nice approach.
In this way, it doesn't matter which server receives the mail, as long as the MDAs of each server write on the shared storage.

Alex

On January 25, 2017 6:08:59 PM EET, Patrick Domack [hidden email] wrote:
All options, assuming your imap/pop/lmtp are compatable and friendly using it.

I know dovecot you should only access a mailstore from one host at a  
time, don't just randomly balance things, or it can corrupt the index  
files.

Quoting Eero Volotinen [hidden email]:

how about mounting ceph or glusterfs disk to message store? eero 25.1.2017 5.18 ap. "Patrick Domack" [hidden email] kirjoitti:
This would not be a good thing to do, as deleted email will magically reappear. Using unison to sync it worked for me, over 10years ago. But these days, just use dsync part of dovecot, and your life will be happy. Quoting Patrick Chemla [hidden email]: Hi Wietse,
Of course I thought about such NAS solution, but I wanted to check if there is a way with 2 separate disks, with a kind of that could be aware of emails files changes. Actually, the mail server run onto a VM, on a big server. I have another big server with same emails VM, and I just rsync --delete --update from the first one to the second. So I have a full image copy every 5 minutes, but only one real MTA. I will check the NAS option, if there is no other way. Thanks Patrick Le 24/01/2017 à 13:45, Wietse Venema a écrit :
Patrick Chemla:
Hi, I have a running Fedora 24 emails server using postfix 3.1.3, with courier. I wonder how to build a pair of MTAs to secure emails at all time, having 2 servers receiving the emails, and users could connect to either server to get emails, maybe on a load balanced way. Problems are with synchronization when receiving emails from outside, or emails read, emails moved,....
You need a redundant message store. In pre-cloud times, people would use a NAS filer with redundant disks, store email as maildir files (one per message) and MDAs would mount that store via NFS. Perhaps that model still works for you. Does someone have a good guide, howto, doc to achieve this?
Thanks for help. Patrick

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: load balanced emails servers pair

rightkicktech.gmail.com
One approach could be to setup a load balancer (lvs, haproxy) in front of the servers to share the IMAP load. Make also the sessions persistent to avoid issues with authentication.

Alex

On February 9, 2017 11:57:09 AM GMT+02:00, Patrick Chemla <[hidden email]> wrote:
Thanks all for your answers.

I have at last setup the NAS, and mails are received there.

So I will set the second server and second MTA, and both will receive emails.

Next step is to give users access to both servers to retreive emails.

As a load-balancer could help easily for http/https access, how to deal with IMAP ports? How to load-balance IMAP ports?

Thanks
Patrick

Le 29/01/2017 à 14:29, rightkicktech.gmail.com a écrit :
A shared storage with glusterfs seems a nice approach.
In this way, it doesn't matter which server receives the mail, as long as the MDAs of each server write on the shared storage.

Alex

On January 25, 2017 6:08:59 PM EET, Patrick Domack [hidden email] wrote:
All options, assuming your imap/pop/lmtp are compatable and friendly using it.

I know dovecot you should only access a mailstore from one host at a  
time, don't just randomly balance things, or it can corrupt the index  
files.

Quoting Eero Volotinen [hidden email]:

how about mounting ceph or glusterfs disk to message store? eero 25.1.2017 5.18 ap. "Patrick Domack" [hidden email] kirjoitti:
This would not be a good thing to do, as deleted email will magically reappear. Using unison to sync it worked for me, over 10years ago. But these days, just use dsync part of dovecot, and your life will be happy. Quoting Patrick Chemla [hidden email]: Hi Wietse,
Of course I thought about such NAS solution, but I wanted to check if there is a way with 2 separate disks, with a kind of that could be aware of emails files changes. Actually, the mail server run onto a VM, on a big server. I have another big server with same emails VM, and I just rsync --delete --update from the first one to the second. So I have a full image copy every 5 minutes, but only one real MTA. I will check the NAS option, if there is no other way. Thanks Patrick Le 24/01/2017 à 13:45, Wietse Venema a écrit :
Patrick Chemla:
Hi, I have a running Fedora 24 emails server using postfix 3.1.3, with courier. I wonder how to build a pair of MTAs to secure emails at all time, having 2 servers receiving the emails, and users could connect to either server to get emails, maybe on a load balanced way. Problems are with synchronization when receiving emails from outside, or emails read, emails moved,....
You need a redundant message store. In pre-cloud times, people would use a NAS filer with redundant disks, store email as maildir files (one per message) and MDAs would mount that store via NFS. Perhaps that model still works for you. Does someone have a good guide, howto, doc to achieve this?
Thanks for help. Patrick

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.



--
Sent from my Android device with K-9 Mail. Please excuse my brevity.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: load balanced emails servers pair

rightkicktech.gmail.com
Another approach is to load balance with DNS. You setup a domain that resolves to the set of IP addresses of your mail servers. You might even imploy priority in the DNS records for weighted load sharing. You have the persistence of the session as a bonus. Not so efficient always but way more simple.

Alex

On February 10, 2017 1:07:45 AM GMT+02:00, Alex <[hidden email]> wrote:
One approach could be to setup a load balancer (lvs, haproxy) in front of the servers to share the IMAP load. Make also the sessions persistent to avoid issues with authentication.

Alex

On February 9, 2017 11:57:09 AM GMT+02:00, Patrick Chemla <[hidden email]> wrote:
Thanks all for your answers.

I have at last setup the NAS, and mails are received there.

So I will set the second server and second MTA, and both will receive emails.

Next step is to give users access to both servers to retreive emails.

As a load-balancer could help easily for http/https access, how to deal with IMAP ports? How to load-balance IMAP ports?

Thanks
Patrick

Le 29/01/2017 à 14:29, rightkicktech.gmail.com a écrit :
A shared storage with glusterfs seems a nice approach.
In this way, it doesn't matter which server receives the mail, as long as the MDAs of each server write on the shared storage.

Alex

On January 25, 2017 6:08:59 PM EET, Patrick Domack [hidden email] wrote:
All options, assuming your imap/pop/lmtp are compatable and friendly using it.

I know dovecot you should only access a mailstore from one host at a  
time, don't just randomly balance things, or it can corrupt the index  
files.

Quoting Eero Volotinen [hidden email]:

how about mounting ceph or glusterfs disk to message store? eero 25.1.2017 5.18 ap. "Patrick Domack" [hidden email] kirjoitti:
This would not be a good thing to do, as deleted email will magically reappear. Using unison to sync it worked for me, over 10years ago. But these days, just use dsync part of dovecot, and your life will be happy. Quoting Patrick Chemla [hidden email]: Hi Wietse,
Of course I thought about such NAS solution, but I wanted to check if there is a way with 2 separate disks, with a kind of that could be aware of emails files changes. Actually, the mail server run onto a VM, on a big server. I have another big server with same emails VM, and I just rsync --delete --update from the first one to the second. So I have a full image copy every 5 minutes, but only one real MTA. I will check the NAS option, if there is no other way. Thanks Patrick Le 24/01/2017 à 13:45, Wietse Venema a écrit :
Patrick Chemla:
Hi, I have a running Fedora 24 emails server using postfix 3.1.3, with courier. I wonder how to build a pair of MTAs to secure emails at all time, having 2 servers receiving the emails, and users could connect to either server to get emails, maybe on a load balanced way. Problems are with synchronization when receiving emails from outside, or emails read, emails moved,....
You need a redundant message store. In pre-cloud times, people would use a NAS filer with redundant disks, store email as maildir files (one per message) and MDAs would mount that store via NFS. Perhaps that model still works for you. Does someone have a good guide, howto, doc to achieve this?
Thanks for help. Patrick

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.




--
Sent from my Android device with K-9 Mail. Please excuse my brevity.
Loading...