Segfault

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Segfault

Gabriel Marais
Hi Guys

One of my SMTP servers became unavailable yesterday. I'm not sure if it
is directly related to Postfix but I am seeing the following out of my
syslog log (especially interested in the kernel: segfaults):-

Apr 23 19:00:15 smtp-02 postfix/master[18906]: warning: process
/usr/lib/postfix/pickup pid 21010 exit status 1
Apr 23 19:00:15 smtp-02 postfix/master[18906]: warning:
/usr/lib/postfix/pickup: bad command startup -- throttling
Apr 23 19:00:56 smtp-02 kernel: [15661.808548] showq[21030]: segfault at
2 ip 00007ff20615d251 sp 00007ffd8b03df70 error 4 in
NKa5HF1nTzKs.so.x86_64[7ff20614d000+13000]
Apr 23 19:00:56 smtp-02 postfix/master[18906]: warning: process
/usr/lib/postfix/showq pid 21030 killed by signal 11
Apr 23 19:00:56 smtp-02 postfix/master[18906]: warning:
/usr/lib/postfix/showq: bad command startup -- throttling
Apr 23 19:00:57 smtp-02 kernel: [15662.933957] proxymap[21032]: segfault
at 2 ip 00007fc43472f251 sp 00007ffd7e1701d0 error 4 in
NKa5HF1nTzKs.so.x86_64[7fc43471f000+13000]
Apr 23 19:00:57 smtp-02 postfix/master[18906]: warning: process
/usr/lib/postfix/proxymap pid 21032 killed by signal 11
Apr 23 19:00:57 smtp-02 postfix/master[18906]: warning:
/usr/lib/postfix/proxymap: bad command startup -- throttling

I have another server setup, with the same version of
Ubuntu 14.04.x
Postfix 2.11.0

  ldd /usr/lib/postfix/smtpd
         linux-vdso.so.1 =>  (0x00007ffccc9e9000)
         libpostfix-master.so.1 => /usr/lib/libpostfix-master.so.1
(0x00007fb9bf758000)
         libpostfix-tls.so.1 => /usr/lib/libpostfix-tls.so.1
(0x00007fb9bf53f000)
         libpostfix-dns.so.1 => /usr/lib/libpostfix-dns.so.1
(0x00007fb9bf338000)
         libpostfix-global.so.1 => /usr/lib/libpostfix-global.so.1
(0x00007fb9bf0f9000)
         libpostfix-util.so.1 => /usr/lib/libpostfix-util.so.1
(0x00007fb9beebb000)
         libsasl2.so.2 => /usr/lib/x86_64-linux-gnu/libsasl2.so.2
(0x00007fb9bec9f000)
         libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
(0x00007fb9bea81000)
         libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fb9be6b9000)
         libssl.so.1.0.0 => /lib/x86_64-linux-gnu/libssl.so.1.0.0
(0x00007fb9be459000)
         libcrypto.so.1.0.0 => /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
(0x00007fb9be07d000)
         libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2
(0x00007fb9bde62000)
         libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fb9bdc5d000)
         libdb-5.3.so => /usr/lib/x86_64-linux-gnu/libdb-5.3.so
(0x00007fb9bd8bb000)
         libnsl.so.1 => /lib/x86_64-linux-gnu/libnsl.so.1
(0x00007fb9bd6a1000)
         /lib64/ld-linux-x86-64.so.2 (0x000055c33ffbb000)


Does anyone have an idea what would cause this behaviour?
--




Regards

Gabriel Marais

Office: +27 861 466 546 x 7001
Mobile: +27 83 663 2222
Mail: [hidden email]

Unit 11, Ground Floor, Berkley Office Park
Cnr Bauhinia & Witch Hazel Str,
Highveld, Centurion, South-Africa
0157

PO Box 15846, Lyttelton, South Africa, 0140
Reply | Threaded
Open this post in threaded view
|

Re: Segfault

Wietse Venema
> Does anyone have an idea what would cause this behaviour?

Look for the mesage in the maildrop queue.

postqueue -p
postcat -q the-queue-id

pickup daemons should not crash.

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: Segfault

Viktor Dukhovni
In reply to this post by Gabriel Marais

> On Apr 24, 2017, at 3:09 AM, Gabriel Marais <[hidden email]> wrote:
>
> Apr 23 19:00:15 smtp-02 postfix/master[18906]: warning: process /usr/lib/postfix/pickup pid 21010 exit status 1
> Apr 23 19:00:15 smtp-02 postfix/master[18906]: warning: /usr/lib/postfix/pickup: bad command startup -- throttling

This is a symptom, not the cause

> Apr 23 19:00:56 smtp-02 kernel: [15661.808548] showq[21030]: segfault at 2 ip 00007ff20615d251 sp 00007ffd8b03df70 error 4 in NKa5HF1nTzKs.so.x86_64[7ff20614d000+13000]

This is more serious.  What on earth is "NKa5HF1nTzKs.so"?

> Apr 23 19:00:56 smtp-02 postfix/master[18906]: warning: process /usr/lib/postfix/showq pid 21030 killed by signal 11
> Apr 23 19:00:56 smtp-02 postfix/master[18906]: warning: /usr/lib/postfix/showq: bad command startup -- throttling

> Apr 23 19:00:57 smtp-02 kernel: [15662.933957] proxymap[21032]: segfault at 2 ip 00007fc43472f251 sp 00007ffd7e1701d0 error 4
> in NKa5HF1nTzKs.so.x86_64[7fc43471f000+13000]

A crashing "proxymap" will break all services that use it for LDAP, SQL, ... so that's yoiur problem...

> Apr 23 19:00:57 smtp-02 postfix/master[18906]: warning: process /usr/lib/postfix/proxymap pid 21032 killed by signal 11
> Apr 23 19:00:57 smtp-02 postfix/master[18906]: warning: /usr/lib/postfix/proxymap: bad command startup -- throttling
>
> I have another server setup, with the same version of
> Ubuntu 14.04.x
> Postfix 2.11.0
>
> ldd /usr/lib/postfix/smtpd

You're looking in the wrong place, smtpd(8) was not crashing, but proxymap(8) and showq(8) were.
The strange "NKa5HF1nTzKs.so" file is unlikely to show up in ldd output (but do check).  If it
is a real library name and not just base64 noise that overwrote the real name, then perhaps it
was loaded on the fly by some NSS module or something else...

--
        Viktor.

Reply | Threaded
Open this post in threaded view
|

Re: Segfault

Wietse Venema
In reply to this post by Gabriel Marais
Gabriel Marais:
> Apr 23 19:00:15 smtp-02 postfix/master[18906]: warning: process
> /usr/lib/postfix/pickup pid 21010 exit status 1

What did the pickup daemon log *before* this? Error messages
from the pickup daemon are rare.

    $ grep pickup.21010 /the/maillog/file

> Apr 23 19:00:56 smtp-02 kernel: [15661.808548] showq[21030]: segfault at
> 2 ip 00007ff20615d251 sp 00007ffd8b03df70 error 4 in
> NKa5HF1nTzKs.so.x86_64[7ff20614d000+13000]
> Apr 23 19:00:56 smtp-02 postfix/master[18906]: warning: process
> /usr/lib/postfix/showq pid 21030 killed by signal 11

Hmm, I hadn't noticed that showq was segfaulting. By the way, what
is the output from:

    $ find / -name '*NKa5HF1nTzKs.so*'

> Apr 23 19:00:57 smtp-02 kernel: [15662.933957] proxymap[21032]: segfault
> at 2 ip 00007fc43472f251 sp 00007ffd7e1701d0 error 4 in
> NKa5HF1nTzKs.so.x86_64[7fc43471f000+13000]
> Apr 23 19:00:57 smtp-02 postfix/master[18906]: warning: process
> /usr/lib/postfix/proxymap pid 21032 killed by signal 11

And proxymap is segfaulting too. It is not like the warranty on
Postfix 2.11 has suddenly expired. More likely is that the file
for NKa5HF1nTzKs.so (whatever that is) has become corrupted, or the
in-memory cached copy of that file.

Does the file for NKa5HF1nTzKs.so exist on both systems, and are
the contents identical (check with 'md5sum').

Does the server have ECC memory error corrction?

> I have another server setup, with the same version of
> Ubuntu 14.04.x
> Postfix 2.11.0
>
>   ldd /usr/lib/postfix/smtpd

You haven't shown evidence for segfaults in smtpd, but you
hane shown evidence for segfaults in several other programs.

        Wietse