gmgr behaviour in virtual servers

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

gmgr behaviour in virtual servers

pete pot


This may not be exactly a Postfix question, but related...
I have noticed that when there are a lot of emails in the deferred queue and i use 'postqueue -f' to send them all, vmware virtual servers

are incredible much faster than physical servers!!
Virtual servers can take 3 or 4 seconds to empty the queue, while the physical one more than 1 minute...
-Both servers use solid state drives, have plenty of free ram available and exactly the same physical HW.
-Postfix in both servers have the same config.

Has anyone else noticed this?

Why does this happen? My best guess is that it may be a disk cache issue but this is just a guess...

can anyone guess any circunstance (maybe in qmgr) that can explain this???


Pete.
Reply | Threaded
Open this post in threaded view
|

Re: gmgr behaviour in virtual servers

Wietse Venema
pete pot:
>
>
> This may not be exactly a Postfix question, but related...
> I have noticed that when there are a lot of emails in the deferred queue and i use 'postqueue -f' to send them all, vmware virtual servers
>
> are incredible much faster than physical servers!!

You say both servers run off SSD and have plenty memory. Remaining
differences:

- Your VMM cheats, by disabling SYNCHRONOUS UPDATES of the virtual
disk. This means you lose mail when the host OS crashes. Postfix
requires that fsync returns after the data is stored. With caching
in host memory, fsync returns while the data sits in host memory.

- Your OS in the VM has atime updates disabled, and the physicsal
host not.

Or a combination of the above.

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: gmgr behaviour in virtual servers

pete pot
In reply to this post by pete pot
>> This may not be exactly a Postfix question, but related...
>> I have noticed that when there are a lot of emails in the deferred queue and i use 'postqueue -f' to send them all, vmware virtual servers
>>
>> are incredible much faster than physical servers!!
>
>
>You say both servers run off SSD and have plenty memory. Remaining
>differences:
>
>- Your VMM cheats, by disabling SYNCHRONOUS UPDATES of the virtual
>disk. This means you lose mail when the host OS crashes. Postfix
>requires that fsync returns after the data is stored. With caching
>in host memory, fsync returns while the data sits in host memory.
>
>- Your OS in the VM has atime updates disabled, and the physicsal
>host not.
>
>Or a combination of the above.

- Makes total sense that fsync falsely returns before all data are written to real drive in a VM... i did not know Postfix waits for fsync, thanks Wietse!
- Regarding atime... is qmgr 'atime' dependant? or maybe you suggest it due to the time the OS takes to update the 'atime'?

What i have noticed, Wietse, is that inmediatelly after the "postqueue -f" the log shows tons of qmgr messages like this:
       
        postfix/qmgr[65371]: ED9FD4209E4: from=<>, size=22211, nrcpt=1 (queue active)

And then..
- Immediately many successful postfix/smtp lines...  (in the VM)
- After a long delay many successful postfix/smtp lines...  (in the Physical server)

May this delay be due to the qmgr files movement from deferred to active folders? that would be coherent with the fsync issue... since it only affects writes to disk, not reads.
       

Thanks Wietse!

Pete.
Reply | Threaded
Open this post in threaded view
|

Re: gmgr behaviour in virtual servers

Viktor Dukhovni
In reply to this post by Wietse Venema


> On Jan 23, 2018, at 8:29 AM, Wietse Venema <[hidden email]> wrote:
>
> - Your VMM cheats, by disabling SYNCHRONOUS UPDATES of the virtual
> disk. This means you lose mail when the host OS crashes. Postfix
> requires that fsync returns after the data is stored. With caching
> in host memory, fsync returns while the data sits in host memory.
>
> - Your OS in the VM has atime updates disabled, and the physicsal
> host not.
>
> Or a combination of the above.

There could also be performance impacting differences in the syslog
configuration (a bit less likely perhaps, but not impossible).

--
        Viktor.

Reply | Threaded
Open this post in threaded view
|

Re: gmgr behaviour in virtual servers

Wietse Venema
In reply to this post by pete pot
pete pot:
> - Makes total sense that fsync falsely returns before all data are
> written to real drive in a VM... i did not know Postfix waits for
> fsync, thanks Wietse!

This is a VM configuration error. The writes should be synchronous, so that
mail will not be lost after fsync() returns.

> - Regarding atime... is qmgr 'atime' dependant? or maybe you suggest
> it due to the time the OS takes to update the 'atime'?

Postfix is mtime dependent, not atime dependent.

> What i have noticed, Wietse, is that inmediatelly after the
> "postqueue -f" the log shows tons of qmgr messages like this:
>
> postfix/qmgr[65371]: ED9FD4209E4: from=<>, size=22211, nrcpt=1 (queue active)

The queue manager reads queue files, *BUT!* it also starts deliveries
immediately in-between file reads.

> And then..
> - Immediately many successful postfix/smtp lines...  (in the VM)
> - After a long delay many successful postfix/smtp lines...  (in the Physical server)

SMTP requires DNS lookups. If your DNS configuration differs between
VM and non,-VM that could expain the SMTP latency difference.

        Wietse.