pickup performance

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

pickup performance

Raymond Sellars

Hi


Looking for some hints on a performance problem i have with postfix and looping mail through a content filter and it slowly feedback back out via the maildrop.


It would seem that pickup process is trickle feeding the maildrop back into the active queue. I conscious that pickup is single thread but for comparison on reboot of the match postfix will seemingly take the maildrop items real fast and smtp send them out. But if I batch send the items in via postfix input queue (from an external smtp server) it goes slow.


Scenario:

Stress test were I send 30-60k of messages to a single domain (not the best test but its suits my requirements).

Jame SMTP server -> sends to postfix (25 connections) -> postfix setup with a content filter that is a python script (see below) -> reinjects using send mail -> pickup trickles messages into active -> the relay (smtp) to another mail server but very slowly, i.e 1 a second or there about.

Again. If I reboot the box (restarting postfix doesn't seem to do it) the relay sending goes out at 20 connections and really fast.

Snippet from the python script:

# pass message to python script which calls webservice
/usr/bin/python /etc/postfix/custom_scripts/audit/mime.py /etc/postfix/custom_scripts/audit/messages/rawmessage_$$.msg "$@"
# || { echo Python script did not execute; exit $EX_UNAVAILABLE; }

# send message back to queue - strip the postfix instance name
/usr/sbin/sendmail -G -i -f "$1" "${@:3}" </etc/postfix/custom_scripts/audit/messages/rawmessage_$$.msg

I don't believe the above is an issue. Its the fact I'm sendmail it to maildrop and pickup is being slow.

CPU utilisation is moderate to low when pickup is being slow. IO wait is averaging 20% so i don't think its a deliberate bottle neck. maildrop directory entries are in the order of 20-30k.

default_destination_concurrency_limit = 50 - i've tried bumping this. Works on the fresh restart but seems to have no bearing ones the postfix has been running, i.e. concurrency doesn't ramp up.

in_flow_delay = 0, didn't seem to have any influence either.

My thoughts:
- on a reboot (but why not restart) pickup is detecting the maildrop items and then pushing them fast into the active/incoming queue.
- when running normally and after delivery via sendmail pickup is not detecting the items and only picking up at a rate limit - why?

Any ideas or suggestions would be appreciated.

Thanks

Raymond