postqueue: warning: unix_trigger: write to public/qmgr: Broken pipe

classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

postqueue: warning: unix_trigger: write to public/qmgr: Broken pipe

Marek Kozlowski-2
:-)

postfix + postsrsd + clamav + spamassassin + dovecot

Everything seems to work OK. No changes done recently (used to work for
a long long time). No error messages in logs. Some mails are delivered
correctly and immediately. Outgoing mail - OK. The problems are:

1. About 20-50 mails shown by a `mailq'
2. Some mails are delivered veeeeeery sloooowly (I cannot find
differences between "bad" and "good" mails)
3. # postqueue -vf
postqueue: name_mask: ipv4
postqueue: inet_addr_local: configured 2 IPv4 addresses
postqueue: warning: unix_trigger: write to public/qmgr: Broken pipe

Unfortunately I can't find explanation for this error. Any help welcome!
I'm guessing something wrong with some queue. But more precisely..? Any
part of configuration on request - but if I don;t understand the
situation I can see a very limited sense in sending all settings for all
services...

Best regards,
Marek

--
Dr Eng. Marek Kozłowski
Senior Lecturer
Unix and Network Administrator
Warsaw University of Technology
Faculty of Mathematics and Information Sciences
ul. Koszykowa 75,
00-662 Warszawa
POLAND
tel.: +48 601 827 225



smime.p7s (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: postqueue: warning: unix_trigger: write to public/qmgr: Broken pipe

Wietse Venema
A problem report without looking at the logs?

http://www.postfix.org/DEBUG_README.html#logging

DO NOT turn on debug logging unless asked to do so.

        Wietser
Reply | Threaded
Open this post in threaded view
|

Re: postqueue: warning: unix_trigger: write to public/qmgr: Broken pipe

Marek Kozlowski-2
W dniu 2018-11-14 16:07, Wietse Venema napisał(a):
> A problem report without looking at the logs?
>
> http://www.postfix.org/DEBUG_README.html#logging
>
> DO NOT turn on debug logging unless asked to do so.
>
> Wietser

The ONLY entries matching include:

1. [warning,mail]postfix/postqueue[13925]: warning: unix_trigger: write
to public/qmgr: Broken pipe
2. [warning,mail]postfix/master[9522]: warning: unix_trigger_event: read
timeout for service public/pickup
3. [warning,mail]postfix/cleanup[9530]: warning: read TCP map reply from
127.0.0.1:10002: unexpected EOF (Connection timed out)

So they don't give much more information than messages in my previous
post I'm afraid.

Best regards,
Marek

BTW: mails ARE delivered locally but with a substantial delay.
Reply | Threaded
Open this post in threaded view
|

Re: postqueue: warning: unix_trigger: write to public/qmgr: Broken pipe

Viktor Dukhovni


> On Nov 14, 2018, at 11:04 AM, Marek Kozlowski <[hidden email]> wrote:
>
> The ONLY entries matching include:
>
> 1. [warning,mail]postfix/postqueue[13925]: warning: unix_trigger: write to public/qmgr: Broken pipe
> 2. [warning,mail]postfix/master[9522]: warning: unix_trigger_event: read timeout for service public/pickup
> 3. [warning,mail]postfix/cleanup[9530]: warning: read TCP map reply from 127.0.0.1:10002: unexpected EOF (Connection timed out)
>
> So they don't give much more information than messages in my previous post I'm afraid.

If restarting Postfix (which re-creates all the internal unix-domain sockets)
is not enough, then reboot your system.  Something is wrong with unix-domain
IPC.  The "unix_trigger" function writes a single byte to a unix-domain socket
or fifo.  If that's failing, your O/S is not functioning correctly.

--
        Viktor.

Reply | Threaded
Open this post in threaded view
|

Re: postqueue: warning: unix_trigger: write to public/qmgr: Broken pipe

Marek Kozlowski-2
W dniu 2018-11-14 17:33, Viktor Dukhovni napisał(a):

>> On Nov 14, 2018, at 11:04 AM, Marek Kozlowski
>> <[hidden email]> wrote:
>>
>> The ONLY entries matching include:
>>
>> 1. [warning,mail]postfix/postqueue[13925]: warning: unix_trigger:
>> write to public/qmgr: Broken pipe
>> 2. [warning,mail]postfix/master[9522]: warning: unix_trigger_event:
>> read timeout for service public/pickup
>> 3. [warning,mail]postfix/cleanup[9530]: warning: read TCP map reply
>> from 127.0.0.1:10002: unexpected EOF (Connection timed out)
>>
>> So they don't give much more information than messages in my previous
>> post I'm afraid.
>
> If restarting Postfix (which re-creates all the internal unix-domain
> sockets)
> is not enough, then reboot your system.  Something is wrong with
> unix-domain
> IPC.  The "unix_trigger" function writes a single byte to a unix-domain
> socket
> or fifo.  If that's failing, your O/S is not functioning correctly.

I've restarted postfix as well as the machine. It didn't help :-(
Maybe there is some cached runtime information for postfix or some
broken database?

Best regards,
Marek
Reply | Threaded
Open this post in threaded view
|

Re: postqueue: warning: unix_trigger: write to public/qmgr: Broken pipe

Viktor Dukhovni
> On Nov 14, 2018, at 12:00 PM, Marek Kozlowski <[hidden email]> wrote:
>
> Maybe there is some cached runtime information for postfix or some broken database?

No.  The communications sockets don't any such "memory".  If they're dropping packets,
your kernel is fubared, and you need a more reliable O/S.  Unless the packet loss is
due to some sort of security software (SELinux, AppArmor?)...

Postfix unix-domain sockets have worked reliably for 21 years on multiple Unix-like
operating systems (Solaris, HP-UX, AIX, FreeBSD, NetBSD, OpenBSD, Linux, MacOS/X).
If your system is having trouble delivering 1-byte messages over a unix-domain socket,
something is quite wrong with the system rather than Postfix.

The only thing to check is that you have nothing peculiar in master.cf.  All
the entries should have type (2nd column) either "inet" or "unix", you should
not be using "fifo".

--
        Viktor.

Reply | Threaded
Open this post in threaded view
|

Re: postqueue: warning: unix_trigger: write to public/qmgr: Broken pipe

Wietse Venema
In reply to this post by Marek Kozlowski-2
> 2. [warning,mail]postfix/master[9522]: warning: unix_trigger_event: read
> timeout for service public/pickup

What does the master.cf entry for the pickup service look like? It
should say 'unix' not 'fifo'. Postfix stopped using the 'fifo' years
ago.

        Wietse

20120924

        ...

        Cleanup: don't emulate UNIX-domain sockets over FIFOs on
        Solaris systems less than 10 years old. This allows us to
        globally s/fifo/unix/ in master.cf.  Files: makedefs,
        util/sys_defs.h.

        Laptop-friendliness: avoid disk spin-up on idle systems by
        s/fifo/unix/ in master.cf.  Files: conf/master.cf.

Reply | Threaded
Open this post in threaded view
|

Re: postqueue: warning: unix_trigger: write to public/qmgr: Broken pipe

Marek Kozlowski-2
W dniu 2018-11-14 19:29, Wietse Venema napisał(a):
>> 2. [warning,mail]postfix/master[9522]: warning: unix_trigger_event:
>> read
>> timeout for service public/pickup
>
> What does the master.cf entry for the pickup service look like? It
> should say 'unix' not 'fifo'. Postfix stopped using the 'fifo' years
> ago.

smtp      inet  n       -       n       -       -       smtpd
   -o content_filter=spamassassin
   -o receive_override_options=no_address_mappings
smtpd     pass  -       -       n       -       -       smtpd
submission inet n       -       n       -       -       smtpd
   -o syslog_name=postfix/submission
   -o smtpd_tls_security_level=encrypt
   -o smtpd_sasl_auth_enable=yes
   -o smtpd_reject_unlisted_recipient=no
   -o smtpd_relay_restrictions=permit_sasl_authenticated,reject
   -o milter_macro_daemon_name=ORIGINATING
smtps     inet  n       -       n       -       -       smtpd
   -o syslog_name=postfix/smtps
   -o smtpd_tls_wrappermode=yes
   -o smtpd_sasl_auth_enable=yes
   -o smtpd_reject_unlisted_recipient=no
   -o smtpd_relay_restrictions=permit_sasl_authenticated,reject
   -o milter_macro_daemon_name=ORIGINATING
pickup    unix  n       -       n       60      1       pickup
cleanup   unix  n       -       n       -       0       cleanup
qmgr      unix  n       -       n       300     1       qmgr
tlsmgr    unix  -       -       n       1000?   1       tlsmgr
rewrite   unix  -       -       n       -       -       trivial-rewrite
bounce    unix  -       -       n       -       0       bounce
defer     unix  -       -       n       -       0       bounce
trace     unix  -       -       n       -       0       bounce
verify    unix  -       -       n       -       1       verify
flush     unix  n       -       n       1000?   0       flush
proxymap  unix  -       -       n       -       -       proxymap
proxywrite unix -       -       n       -       1       proxymap
smtp      unix  -       -       n       -       -       smtp
relay     unix  -       -       n       -       -       smtp
showq     unix  n       -       n       -       -       showq
error     unix  -       -       n       -       -       error
retry     unix  -       -       n       -       -       error
discard   unix  -       -       n       -       -       discard
local     unix  -       n       n       -       -       local
virtual   unix  -       n       n       -       -       virtual
lmtp      unix  -       -       n       -       -       lmtp
anvil     unix  -       -       n       -       1       anvil
scache    unix  -       -       n       -       1       scache
spamassassin unix -     n       n       -       -       pipe
   user=nobody argv=/usr/bin/vendor_perl/spamc -f -e /usr/sbin/sendmail
-oi -f ${sender} ${recipient}

Best regards,
/m
Reply | Threaded
Open this post in threaded view
|

Re: postqueue: warning: unix_trigger: write to public/qmgr: Broken pipe

Marek Kozlowski-2
In reply to this post by Wietse Venema
:-)

A new type of warning in logs found:

[warning,mail]postfix/master[458]: warning: master_wakeup_timer_event:
service pickup(public/pickup): Resource temporarily un
available

May I ask for some explanation?

Best regards,
Marek
Reply | Threaded
Open this post in threaded view
|

Re: postqueue: warning: unix_trigger: write to public/qmgr: Broken pipe

Wietse Venema
In reply to this post by Marek Kozlowski-2
Marek Kozlowski:
> W dniu 2018-11-14 19:29, Wietse Venema napisa?(a):
> >> 2. [warning,mail]postfix/master[9522]: warning: unix_trigger_event:
> >> read
> >> timeout for service public/pickup
> >
> > What does the master.cf entry for the pickup service look like? It
> > should say 'unix' not 'fifo'. Postfix stopped using the 'fifo' years
> > ago.
...
> pickup    unix  n       -       n       60      1       pickup

Your kernel's UNIX-domain stack is messed up, or some 'security'
system is interfering with proper operation.

https://dilbert.com/strip/1995-06-24

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: postqueue: warning: unix_trigger: write to public/qmgr: Broken pipe

Marek Kozlowski-2
:-)

On 11/14/18 8:40 PM, Wietse Venema wrote:

> Marek Kozlowski:
>> W dniu 2018-11-14 19:29, Wietse Venema napisa?(a):
>>>> 2. [warning,mail]postfix/master[9522]: warning: unix_trigger_event:
>>>> read
>>>> timeout for service public/pickup
>>>
>>> What does the master.cf entry for the pickup service look like? It
>>> should say 'unix' not 'fifo'. Postfix stopped using the 'fifo' years
>>> ago.
> ...
>> pickup    unix  n       -       n       60      1       pickup
>
> Your kernel's UNIX-domain stack is messed up, or some 'security'
> system is interfering with proper operation.
>
> https://dilbert.com/strip/1995-06-24
Well. I've disconnected spamskurwysyn (I mean: spamassassin) - in master.cf:

smtp      inet  n       -       n       -       -       smtpd
#  -o content_filter=spamassassin
   -o receive_override_options=no_address_mappings

And now it works perfect (except: with no spam protection). I guess that
spamassassin problems are not for this forum...?

Best regards,
Marek



smime.p7s (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: postqueue: warning: unix_trigger: write to public/qmgr: Broken pipe

Wietse Venema
Marek Kozlowski:
> Well. I've disconnected spamskurwysyn (I mean: spamassassin) - in master.cf:
>
> smtp      inet  n       -       n       -       -       smtpd
> #  -o content_filter=spamassassin
>    -o receive_override_options=no_address_mappings
>
> And now it works perfect (except: with no spam protection). I guess that
> spamassassin problems are not for this forum...?

If your system can't handle the additional load of running
spamassassin, then you need a better computer system.

The Dilbert strip was right.

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: postqueue: warning: unix_trigger: write to public/qmgr: Broken pipe

Marek Kozlowski-2
In reply to this post by Marek Kozlowski-2


On 11/14/18 10:01 PM, Marek Kozlowski wrote:

> :-)
>
> On 11/14/18 8:40 PM, Wietse Venema wrote:
>> Marek Kozlowski:
>>> W dniu 2018-11-14 19:29, Wietse Venema napisa?(a):
>>>>> 2. [warning,mail]postfix/master[9522]: warning: unix_trigger_event:
>>>>> read
>>>>> timeout for service public/pickup
>>>>
>>>> What does the master.cf entry for the pickup service look like? It
>>>> should say 'unix' not 'fifo'. Postfix stopped using the 'fifo' years
>>>> ago.
>> ...
>>> pickup    unix  n       -       n       60      1       pickup
>>
>> Your kernel's UNIX-domain stack is messed up, or some 'security'
>> system is interfering with proper operation.
>>
>> https://dilbert.com/strip/1995-06-24
>
> Well. I've disconnected spamskurwysyn (I mean: spamassassin) - in
> master.cf:
>
> smtp      inet  n       -       n       -       -       smtpd
> #  -o content_filter=spamassassin
>    -o receive_override_options=no_address_mappings
I mean:
smtp      inet  n       -       n       -       -       smtpd
#  -o content_filter=spamassassin
#  -o receive_override_options=no_address_mappings

;-)


smime.p7s (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: postqueue: warning: unix_trigger: write to public/qmgr: Broken pipe

Viktor Dukhovni
> On Nov 14, 2018, at 5:14 PM, Marek Kozlowski <[hidden email]> wrote:
>
>>> Your kernel's UNIX-domain stack is messed up, or some 'security'
>>> system is interfering with proper operation.
>>>
>>> https://dilbert.com/strip/1995-06-24
>>
>> Well. I've disconnected spamskurwysyn (I mean: spamassassin) - in master.cf:
>> smtp      inet  n       -       n       -       -       smtpd
>> #  -o content_filter=spamassassin
>>   -o receive_override_options=no_address_mappings
>
> I mean:
> smtp      inet  n       -       n       -       -       smtpd
> #  -o content_filter=spamassassin
> #  -o receive_override_options=no_address_mappings

This makes no difference, the Dilbert verdict remains unchanged.

--
        Viktor.

Reply | Threaded
Open this post in threaded view
|

Re: postqueue: warning: unix_trigger: write to public/qmgr: Broken pipe

Marek Kozlowski-2
:-)

On 11/14/18 11:25 PM, Viktor Dukhovni wrote:

>> On Nov 14, 2018, at 5:14 PM, Marek Kozlowski <[hidden email]> wrote:
>>
>>>> Your kernel's UNIX-domain stack is messed up, or some 'security'
>>>> system is interfering with proper operation.
>>>>
>>>> https://dilbert.com/strip/1995-06-24
>>>
>>> Well. I've disconnected spamskurwysyn (I mean: spamassassin) - in master.cf:
>>> smtp      inet  n       -       n       -       -       smtpd
>>> #  -o content_filter=spamassassin
>>>    -o receive_override_options=no_address_mappings
>>
>> I mean:
>> smtp      inet  n       -       n       -       -       smtpd
>> #  -o content_filter=spamassassin
>> #  -o receive_override_options=no_address_mappings
>
> This makes no difference, the Dilbert verdict remains unchanged.
Well, yes and no.
I've investigated the problem a little bit and it seems that the heart
of the problem is no unix_trigger sent from sendmail to qmgr due to a
broken pipe. As a result qmgr waits for a timer. OK, seems like
something wrong with the system. Unfortunately hardware/system upgrade
won't solve the problem. Why? The hardware is more than strong enough.
The system is up-to-date.

I've been running postfix for a long time in a systemd Linux under
virtual machine (qemu+kvm). I'm starting to suppose it is not a hardware
itself; it's something related to systemd or qemu. I'm wondering if
there are some users running postfix with no problems under the most
recent systemd (which in fact IS a kind of magic)? using postfix on a
guest over qemu? using postfix on a virtualized systemd system?

Best regards,
Marek

--
Dr Eng. Marek Kozłowski
Senior Lecturer
Unix and Network Administrator
Warsaw University of Technology
Faculty of Mathematics and Information Sciences
ul. Koszykowa 75,
00-662 Warszawa
POLAND
tel.: +48 601 827 225


smime.p7s (4K) Download Attachment