Quantcast

cleanup writes incredible slowly to disk (very slow mail delivery)

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

cleanup writes incredible slowly to disk (very slow mail delivery)

Michael Meier
hi all

I've noticed that for sending big mails (10MB) it can take up to 30 min
until they are sent. So I investigated a bit into the problem. I
deactivated amavis and dkim, so they do not seem to be part of the problem.
What happens is, that as soon as I write a bigger e-mail, the system
load goes up from 0 to around 2.5 and when the mail is send, goes back
to 0 again, but there is almost no cpu usage at all. So it should have
something to do with the I/O system I guess.
iotop tells me that cleanup is writing at around incredible 60 kb/s to
the disk.
All other processes work perfectly and don't get slowed down while the
mail is send, I can even copy to the same partition at normal speed. It
is just cleanup which seems to be so slow. There is also still enough
memory left. Besides the speed, email delivery works perfectly, there
are also no log messages indicating anything.
I don't think it has to do with the network card since I also tried out

smtp-source -t crap@localhost -l 102400000 127.0.0.1

and it shows the exact same behaviour.
Anybody has any idea what the problem might be?
I'm using postfix 2.11.3-1 on a debian jessie (stable) with kernel
4.9.0-0-686-pae.

thanks in advance
michael
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: cleanup writes incredible slowly to disk (very slow mail delivery)

Ralph Corderoy
Hi michael,

> What happens is, that as soon as I write a bigger e-mail, the system
> load goes up from 0 to around 2.5 and when the mail is send, goes back
> to 0 again, but there is almost no cpu usage at all. So it should have
> something to do with the I/O system I guess.  iotop tells me that
> cleanup is writing at around incredible 60 kb/s to the disk.

Try running `dstat -tcdngym -D sda,sdb', changing the list of disks to
match yours, leaving it to bed in so you know what's normal, and then
doing smtp-source to trigger the problem.  You might see high `wai',
`wait' percentages showing processes are stalled waiting for I/O.

> I can even copy to the same partition at normal speed.

What type of filesystems are being used?

--
Cheers, Ralph.
https://plus.google.com/+RalphCorderoy
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: cleanup writes incredible slowly to disk (very slow mail delivery)

Michael Meier
>> What happens is, that as soon as I write a bigger e-mail, the system
>> load goes up from 0 to around 2.5 and when the mail is send, goes back
>> to 0 again, but there is almost no cpu usage at all. So it should have
>> something to do with the I/O system I guess.  iotop tells me that
>> cleanup is writing at around incredible 60 kb/s to the disk.
>
> Try running `dstat -tcdngym -D sda,sdb', changing the list of disks to
> match yours, leaving it to bed in so you know what's normal, and then
> doing smtp-source to trigger the problem.  You might see high `wai',
> `wait' percentages showing processes are stalled waiting for I/O.
>
wait goes directly up from 0 to 50 and stays there


>> I can even copy to the same partition at normal speed.
>
> What type of filesystems are being used?
>
It's still ext3

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: cleanup writes incredible slowly to disk (very slow mail delivery)

Wietse Venema
In reply to this post by Michael Meier
Michael Meier:

> hi all
>
> I've noticed that for sending big mails (10MB) it can take up to 30 min
> until they are sent. So I investigated a bit into the problem. I
> deactivated amavis and dkim, so they do not seem to be part of the problem.
> What happens is, that as soon as I write a bigger e-mail, the system
> load goes up from 0 to around 2.5 and when the mail is send, goes back
> to 0 again, but there is almost no cpu usage at all. So it should have
> something to do with the I/O system I guess.
> iotop tells me that cleanup is writing at around incredible 60 kb/s to
> the disk.
> All other processes work perfectly and don't get slowed down while the
> mail is send, I can even copy to the same partition at normal speed. It
> is just cleanup which seems to be so slow. There is also still enough
> memory left. Besides the speed, email delivery works perfectly, there
> are also no log messages indicating anything.
> I don't think it has to do with the network card since I also tried out
>
> smtp-source -t crap@localhost -l 102400000 127.0.0.1

Completes in a 1-3 seconds on a laptop (ssd) and old workstation
(rust). Obviously the problem is outside of Postfix.

Maybe the outpout from 'lsattr /var/spool/postfix' will reveal
special configuration (sync writes, or something else).

        Wietse

> and it shows the exact same behaviour.
> Anybody has any idea what the problem might be?
> I'm using postfix 2.11.3-1 on a debian jessie (stable) with kernel
> 4.9.0-0-686-pae.
>
> thanks in advance
> michael
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: cleanup writes incredible slowly to disk (very slow mail delivery)

Michael Meier
>> I've noticed that for sending big mails (10MB) it can take up to 30
>> min
>> until they are sent. So I investigated a bit into the problem. I
>> deactivated amavis and dkim, so they do not seem to be part of the
>> problem.
>> What happens is, that as soon as I write a bigger e-mail, the system
>> load goes up from 0 to around 2.5 and when the mail is send, goes back
>> to 0 again, but there is almost no cpu usage at all. So it should have
>> something to do with the I/O system I guess.
>> iotop tells me that cleanup is writing at around incredible 60 kb/s to
>> the disk.
>> All other processes work perfectly and don't get slowed down while the
>> mail is send, I can even copy to the same partition at normal speed.
>> It
>> is just cleanup which seems to be so slow. There is also still enough
>> memory left. Besides the speed, email delivery works perfectly, there
>> are also no log messages indicating anything.
>> I don't think it has to do with the network card since I also tried
>> out
>>
>> smtp-source -t crap@localhost -l 102400000 127.0.0.1
>
> Completes in a 1-3 seconds on a laptop (ssd) and old workstation
> (rust). Obviously the problem is outside of Postfix.
>
> Maybe the outpout from 'lsattr /var/spool/postfix' will reveal
> special configuration (sync writes, or something else).

Decades of using Linux and I had no idea you could set sync writes on
file/directory level!
That was exactly the problem. Would be interesting to know how that ever
got set.
Thanks a lot. I really couldn't imagine in any way what the problem
could be...
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: cleanup writes incredible slowly to disk (very slow mail delivery)

Wietse Venema
Michael Meier:

> >> smtp-source -t crap@localhost -l 102400000 127.0.0.1
> >
> > Completes in a 1-3 seconds on a laptop (ssd) and old workstation
> > (rust). Obviously the problem is outside of Postfix.
> >
> > Maybe the outpout from 'lsattr /var/spool/postfix' will reveal
> > special configuration (sync writes, or something else).
>
> Decades of using Linux and I had no idea you could set sync writes on
> file/directory level!
> That was exactly the problem. Would be interesting to know how that ever
> got set.
> Thanks a lot. I really couldn't imagine in any way what the problem
> could be...

That setting may date from the time that Linux rename() could remove
a file from its old directory before adding it to its new directory,
resulting in 'lost' files after a system crash (presumably, directory
updates got reordered to reduce disk seek activity).

That behavior violated the SMTP standard that that says "[an MTA]
MUST NOT lose the message for frivolous reasons, such as because
the host later crashes or because of a predictable resource shortage."

If you're paranoid, setting the 'D' flag (synchronous  directory
updates) may still be an option. This is rather platform-specific
and I make no promises about Linux file systems bevhavior.

        Wietse
Loading...