Postfix 2.6.x slow

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
22 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Postfix 2.6.x slow

Eric Vaughn

Are there any new features to postfix 2.6.x that would cause it to be slow?

Other than the obvious suspects; stress adaptive behavior, logging, ulimit (running out of file descriptors).

 

We are a very high volume site, we use postfix only as a proxy to decrypt TLS and then pass traffic downstream to other clients.

(master.cff:smtpd -o smtpd_proxy_filter=x.x.x.x)

 

 

Primary symptoms:

New server (2.6.3) is running much slower than old system was (2.5.1).

CPU utilization is much lower

There are no errors in the logs.  Not even with verbose logging turned on.  (We have since disabled verbose logging for performance reasons.)

There seems to be a ~3.5 second latency for both connection establishment and transaction completion.  (We have yet to identify the source of this latency.)

 

Here is a list of things we have checked so far:

Stress adaptive behavior (has not turned on)

Logging (syslog.conf – was a good suspect, syslog.conf was cleaned up and performance is still slow.)

ulimit (new postfix allows much higher number of open files per process, this was increased to 4096)

disk i/o (sar and iostat)

ssl (is not suspect because cpu utilization is low)

 

default process limit was increased to 2000 during the upgrade (main.cf:default_process_limit = 2000) – This could be a good source as well if downstream clients are not able to handle high concurrency, we will lower to 1000 for retesting.   

 

I wish I had a list of specific questions to ask.  Unfortunately due to the lack of information in the logs, there isn’t much to go on.

 

 

 

Thank You

 

-          Eric

 

 

 

 

 

 

 

 

 

Reply | Threaded
Open this post in threaded view
|

Re: Postfix 2.6.x slow

Wietse Venema
What else did you change besides Postfix?

Any OS upgrades, changes in libraries, ...

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: Postfix 2.6.x slow

Eric Vaughn
In reply to this post by Eric Vaughn
Re: Postfix 2.6.x slow

The list of changes (we upgraded a spare server to swap in as a replacement):
OLD                       NEW
Centos 5.0.               Centos 5.3 (yum update all)
i386.                      x64
2.4 ghrz cpu.             2.83 ghrz cpu
4 gigs ram.               4 gigs ram
Openssl 0.9.8b           0.9.8e (with all security updates included with "yum update all", which should have all available openssl patches)
Old server has /var/spool/postfix on a seperate disk /dev/sdb.  The new has only /dev/sda (/var/spool/postfix is shared with the OS '/'), disks are being added to rectify this descrepancy tomorrow (though this is not a likely source of the issue).


main.cf:default_process_limit was increased from 1000 to 2000.
The only new line added by the upgrade (make install) to main.cf was "postfix_data_directory=/var/lib/postfix"


Victor Duchonvi has already reviewed main.cf, master.cf, and makedefs.out.  There were a few things that he changed as a matter of hygene, but nothing big standing out as a problem source.



Thank you

- Eric






----- Original Message -----
From: [hidden email] <[hidden email]>
To: Postfix users <[hidden email]>
Sent: Mon Oct 05 17:55:02 2009
Subject: Re: Postfix 2.6.x slow

What else did you change besides Postfix?

Any OS upgrades, changes in libraries, ...

        Wietse

Reply | Threaded
Open this post in threaded view
|

Re: Postfix 2.6.x slow

Stan Hoeppner
In reply to this post by Eric Vaughn
Eric Vaughn put forth on 10/5/2009 7:17 PM:
> Are there any new features to postfix 2.6.x that would cause it to be slow?
>
> Other than the obvious suspects; stress adaptive behavior, logging,
> ulimit (running out of file descriptors).
>
> We are a very high volume site, we use postfix only as a proxy to
> decrypt TLS and then pass traffic downstream to other clients.

[snip]

Is the new server plugged into the same switch port(s) as the old
server, or a different switch port, or another switch entirely?  Have
you confirmed the NIC(s) are running at the same link speed as the old
server, and running FDX and with no/minimal packet loss?  Have you
swapped patch cables yet?  Is the new server bound with the same IP
address(es) as the old server, and if not, is it on the same routing
subnet as the old server?  If the disk is SCSI, have you confirmed the
controller and disk are sync'ing at the highest mode supported by each?
 Are you using any kind of remote lookup (e.g. LDAP) per connection to a
remote host that might be injecting latency possibly due to a different
network path than before?  Are you doing a DNS lookup (or multiple) per
connection that may now be injecting latency for some reason, different
route maybe?  Did you point the new host at a different set of DNS
servers than the old host?  Is the kernel tick rate (config_hz)
different going from CentOS 5.0 to 5.3?  Are there any other kernel or
daemon timing difference between 5.0 and 5.3?

That's about all I can think of for now.  You've probably already
covered all of these but I'm throwing them out there just in case.  I've
been in your shoes before, and sometimes it's something "obvious" that
we just completely overlook, then pull our hair out (if you have any
unlike me) for days until the solution pops into our heads.

Good luck.

--
Stan
Reply | Threaded
Open this post in threaded view
|

Re: Postfix 2.6.x slow

Wietse Venema
In reply to this post by Eric Vaughn
Eric Vaughn:
> The list of changes (we upgraded a spare server to swap in as a replacement):
> OLD                       NEW
> Centos 5.0.               Centos 5.3 (yum update all)
> i386.                      x64
> 2.4 ghrz cpu.             2.83 ghrz cpu
> 4 gigs ram.               4 gigs ram
> Openssl 0.9.8b           0.9.8e (with all security updates included with "yum update all", which should have all available openssl patches)

My goodness, they replaced everything.

I think you first want to find out that Postfix can resolve DNS
properly. Do some tests inbound (smtpd) and outbound (smtp). A 5s
delay sounds like a bad resolv.conf file (or even bad resolver
libraries under /var/spool/postfix). Turn off chroot in master.cf.
Turn off all kinds of other stuff until performance changes then
investigate the guilty component.

Unfortunately I cannot provide lots if hand-holding. It's
not part of the deal.

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: Postfix 2.6.x slow

Victor Duchovni
In reply to this post by Eric Vaughn
On Mon, Oct 05, 2009 at 05:17:54PM -0700, Eric Vaughn wrote:

> Are there any new features to postfix 2.6.x that would cause it to be
> slow?

Eric your post premature. You don't yet have measurements showing Postfix
2.6 to be "slow". Lets get the volume comparisons, and tcpdump captures
of both the incoming TLS, and the outgoing decrypted sessions.

Once I see your Postfix 2.6 actually being slower, and not anecdotal
impressions, we'll take it from there...

--
        Viktor.

Disclaimer: off-list followups get on-list replies or get ignored.
Please do not ignore the "Reply-To" header.

To unsubscribe from the postfix-users list, visit
http://www.postfix.org/lists.html or click the link below:
<mailto:[hidden email]?body=unsubscribe%20postfix-users>

If my response solves your problem, the best way to thank me is to not
send an "it worked, thanks" follow-up. If you must respond, please put
"It worked, thanks" in the "Subject" so I can delete these quickly.
Reply | Threaded
Open this post in threaded view
|

Postfix 2.6.x slow - Eric? Update?

Stan Hoeppner
Victor Duchovni put forth on 10/6/2009 11:37 AM:

> On Mon, Oct 05, 2009 at 05:17:54PM -0700, Eric Vaughn wrote:
>
>> Are there any new features to postfix 2.6.x that would cause it to be
>> slow?
>
> Eric your post premature. You don't yet have measurements showing Postfix
> 2.6 to be "slow". Lets get the volume comparisons, and tcpdump captures
> of both the incoming TLS, and the outgoing decrypted sessions.
>
> Once I see your Postfix 2.6 actually being slower, and not anecdotal
> impressions, we'll take it from there...
>

Hi Eric,

I'm really interested to know what the culprit was.  Do you have any new
information to report?  What was the cause/solution?

Thanks.

--
Stan

Reply | Threaded
Open this post in threaded view
|

RE: Postfix 2.6.x slow - Eric? Update?

Eric Vaughn

We suspect a few things.  It seems to be resolved now.  Concurrency
rates were greatly increased during the last test.

1) Postfix 2.6 allows for a higher per process limit.  The OS "ulimit"
by default, may not necessarily support this.  The postfix processes
were running out of file descriptors.  "ulimit" was increased to 4096.
2) selinux was enabled.  This was not high on the suspect list, but it
should be disabled in any high transaction environment.



- Eric







-----Original Message-----
From: [hidden email]
[mailto:[hidden email]] On Behalf Of Stan Hoeppner
Sent: Tuesday, October 20, 2009 10:35 PM
To: [hidden email]
Subject: Postfix 2.6.x slow - Eric? Update?

Victor Duchovni put forth on 10/6/2009 11:37 AM:
> On Mon, Oct 05, 2009 at 05:17:54PM -0700, Eric Vaughn wrote:
>
>> Are there any new features to postfix 2.6.x that would cause it to be
>> slow?
>
> Eric your post premature. You don't yet have measurements showing
Postfix
> 2.6 to be "slow". Lets get the volume comparisons, and tcpdump
captures
> of both the incoming TLS, and the outgoing decrypted sessions.
>
> Once I see your Postfix 2.6 actually being slower, and not anecdotal
> impressions, we'll take it from there...
>

Hi Eric,

I'm really interested to know what the culprit was.  Do you have any new
information to report?  What was the cause/solution?

Thanks.

--
Stan


Reply | Threaded
Open this post in threaded view
|

Re: Postfix 2.6.x slow - Eric? Update?

Victor Duchovni
On Wed, Oct 21, 2009 at 09:34:50AM -0700, Eric Vaughn wrote:

>
> We suspect a few things.  It seems to be resolved now.  Concurrency
> rates were greatly increased during the last test.
>
> 1) Postfix 2.6 allows for a higher per process limit.  The OS "ulimit"
> by default, may not necessarily support this.  The postfix processes
> were running out of file descriptors.  "ulimit" was increased to 4096.
> 2) selinux was enabled.  This was not high on the suspect list, but it
> should be disabled in any high transaction environment.

The systems in question are an extremely unusual Postfix use-case:

    - Postfix is only used for (smtpd(8)) TLS termination, proxying
      decrypted traffic to a down-stream, non TLS system.

    - There are no delivery agents, nothing is ever written to the incoming
      queue, ... The only disk (write) activity is from the syslog server.

    - The back-end systems do moderately expensive processing, yielding moderately
      high "." latency, and messages arrive at high volumes over a WAN link,
      so peak concurrency is high.

    - The "tlsmgr" process multiplexes over 1000 smtpd(8) clients that need
      to read the TLS session cache and seed their entropy pool, ...

There is no evidence at this time that Postfix 2.6 is "slower", and no
reason to expect this to be the case. Rather, in addition to a new Postfix,
the entire system has new hardware, a new O/S, ... and orignally also SELinux
enabled. Too many variables to make direct comparisons. Also the initial
observations of the new system were not sufficiently well instrumented, and
the impression of "slow" performance could not be corroborated by real
measurements.

With SELinux turned off, and proper measurements conducted, no performance
degradation has been observed. Another set of measurements is still
pending where the volume generating clients are driven to peak (rather
than steady-state) capacity. I expect that these tests will also find
2.6 no slower than 2.5. There is nothing in the 2.6 smtpd(8) to slow
down decryption and smtpd_proxy support.

We may discover that > 1000 smtpd(8) processes is slower than the previous
limit of 1000 smtpd processes, even in the face of latency which causes
the majority to be idle at a given time. Perhaps kernel paging activity,
or other overhead makes it difficult to extract more performance.

In any case. I don't see anything in this use-case that bears on real
Postfix MTA deployments.

When the final measurements are done, if anything interesting is found,
and understood, we'll report what we can.

--
        Viktor.

Disclaimer: off-list followups get on-list replies or get ignored.
Please do not ignore the "Reply-To" header.

To unsubscribe from the postfix-users list, visit
http://www.postfix.org/lists.html or click the link below:
<mailto:[hidden email]?body=unsubscribe%20postfix-users>

If my response solves your problem, the best way to thank me is to not
send an "it worked, thanks" follow-up. If you must respond, please put
"It worked, thanks" in the "Subject" so I can delete these quickly.
Reply | Threaded
Open this post in threaded view
|

Re: Postfix 2.6.x slow

Stan Hoeppner
In reply to this post by Eric Vaughn
Eric Vaughn put forth on 10/5/2009 8:23 PM:

> OLD                       NEW
> Centos 5.0.               Centos 5.3 (yum update all)
> i386.                      x64
> 2.4 ghrz cpu.             2.83 ghrz cpu

Hi Eric,

Would you please provide the following:

1. Each server make/model#
2. CPU brand/revision/FSB
3. Memory type (DDR/DDR2?) and bus freq
4. 64bit Linux kernel and 64bit Postfix binary on NEW system, or 64/32?
   Or x64 above is merely capability, but you're still 32bit/32bit?

Thanks.

--
Stan
Reply | Threaded
Open this post in threaded view
|

Re: Postfix 2.6.x slow

Victor Duchovni
On Thu, Oct 22, 2009 at 08:15:56AM -0500, Stan Hoeppner wrote:

> Eric Vaughn put forth on 10/5/2009 8:23 PM:
>
> > OLD                       NEW
> > Centos 5.0.               Centos 5.3 (yum update all)
> > i386.                      x64
> > 2.4 ghrz cpu.             2.83 ghrz cpu
>
> Hi Eric,
>
> Would you please provide the following:
>
> 1. Each server make/model#
> 2. CPU brand/revision/FSB
> 3. Memory type (DDR/DDR2?) and bus freq
> 4. 64bit Linux kernel and 64bit Postfix binary on NEW system, or 64/32?
>    Or x64 above is merely capability, but you're still 32bit/32bit?

There is really no need to pursue this at this time. No evidence has
yet been found to support the new system being slower than the old.

--
        Viktor.

Disclaimer: off-list followups get on-list replies or get ignored.
Please do not ignore the "Reply-To" header.

To unsubscribe from the postfix-users list, visit
http://www.postfix.org/lists.html or click the link below:
<mailto:[hidden email]?body=unsubscribe%20postfix-users>

If my response solves your problem, the best way to thank me is to not
send an "it worked, thanks" follow-up. If you must respond, please put
"It worked, thanks" in the "Subject" so I can delete these quickly.
Reply | Threaded
Open this post in threaded view
|

Re: Postfix 2.6.x slow

Stan Hoeppner
Victor Duchovni put forth on 10/22/2009 10:20 AM:

> On Thu, Oct 22, 2009 at 08:15:56AM -0500, Stan Hoeppner wrote:
>
>> Eric Vaughn put forth on 10/5/2009 8:23 PM:
>>
>>> OLD                       NEW
>>> Centos 5.0.               Centos 5.3 (yum update all)
>>> i386.                      x64
>>> 2.4 ghrz cpu.             2.83 ghrz cpu
>> Hi Eric,
>>
>> Would you please provide the following:
>>
>> 1. Each server make/model#
>> 2. CPU brand/revision/FSB
>> 3. Memory type (DDR/DDR2?) and bus freq
>> 4. 64bit Linux kernel and 64bit Postfix binary on NEW system, or 64/32?
>>    Or x64 above is merely capability, but you're still 32bit/32bit?
>
> There is really no need to pursue this at this time. No evidence has
> yet been found to support the new system being slower than the old.

I think you've demonstrated it's not slower.  I'm wondering why it's not
faster, vs what you described as about equal, in performance.  Granted,
the hardware specs above don't seem radically different, but we don't
know if one is Intel and the other AMD.  That can make a large
difference depending on CPU/platform generation.  I'm just trying to
figure out what potential differences there are in raw/theoretical
hardware horsepower between the old and new servers, and any differences
in the combination of hardware/kernel memory management.   I've read of
compatibility and performance issues when using a 64bit Linux kernel
with 32bit userland.

It's an interesting issue and piques my curiosity.  I'd like to pursue
this aspect to see if it might be a factor, or none at all.

--
Stan

Reply | Threaded
Open this post in threaded view
|

Re: Postfix 2.6.x slow

Victor Duchovni
On Thu, Oct 22, 2009 at 11:18:12AM -0500, Stan Hoeppner wrote:

> > There is really no need to pursue this at this time. No evidence has
> > yet been found to support the new system being slower than the old.
>
> I think you've demonstrated it's not slower.  I'm wondering why it's not
> faster,

In a complex multi-node system, the CPU capacity of one node is not
necessarily on the critical path.

> It's an interesting issue and piques my curiosity.  I'd like to pursue
> this aspect to see if it might be a factor, or none at all.

"There is no there there" at this time.

--
        Viktor.

Disclaimer: off-list followups get on-list replies or get ignored.
Please do not ignore the "Reply-To" header.

To unsubscribe from the postfix-users list, visit
http://www.postfix.org/lists.html or click the link below:
<mailto:[hidden email]?body=unsubscribe%20postfix-users>

If my response solves your problem, the best way to thank me is to not
send an "it worked, thanks" follow-up. If you must respond, please put
"It worked, thanks" in the "Subject" so I can delete these quickly.
Reply | Threaded
Open this post in threaded view
|

Re: Postfix 2.6.x slow

Wietse Venema
In reply to this post by Stan Hoeppner
Stan Hoeppner:
> I think you've demonstrated it's not slower.  I'm wondering why it's not
> faster, vs what you described as about equal, in performance.  Granted,

More than 25 years ago people discovered that it is incredibly hard
to spread one program over multiple CPUs such that it keeps every
CPU busy all the time.

This is the main reason why doubling the number CPUs does not always
halve the execution time.

There are also hardware-level issues but their effect usually pales
in comparison.

        Wietse
Reply | Threaded
Open this post in threaded view
|

Postfix 2.6.x slow

Stan Hoeppner
In reply to this post by Victor Duchovni
Victor Duchovni put forth on 10/22/2009 11:41 AM:

> On Thu, Oct 22, 2009 at 11:18:12AM -0500, Stan Hoeppner wrote:
>
>>> There is really no need to pursue this at this time. No evidence has
>>> yet been found to support the new system being slower than the old.
>> I think you've demonstrated it's not slower.  I'm wondering why it's not
>> faster,
>
> In a complex multi-node system, the CPU capacity of one node is not
> necessarily on the critical path.
>
>> It's an interesting issue and piques my curiosity.  I'd like to pursue
>> this aspect to see if it might be a factor, or none at all.
>
> "There is no there there" at this time.

I have always assumed the limitation is downstream as his postfix is
proxy only.  However, no details of the downstream are forthcoming.  I
also assume no matter how fast the hardware he puts the proxy on, it's
going to still be limited by how fast the downstream systems digest the
smtp sessions.  Like I said, I'm merely trying to identify the
differences between the old and new proxy platform, as that is the only
information so far forthcoming.

I'm not continuing to flog the dead horse here, if indeed this horse is
dead.  So I'll just let it rest.

--
Stan
Reply | Threaded
Open this post in threaded view
|

Postfix 2.6.x slow

Stan Hoeppner
In reply to this post by Wietse Venema
Wietse Venema put forth on 10/22/2009 12:04 PM:
> Stan Hoeppner:
>> I think you've demonstrated it's not slower.  I'm wondering why it's not
>> faster, vs what you described as about equal, in performance.  Granted,
>
> More than 25 years ago people discovered that it is incredibly hard
> to spread one program over multiple CPUs such that it keeps every
> CPU busy all the time.

And rediscovering it every day.  Given your employer, I'll use the
example of Roadrunner.  Ask that system's users how many of those
129,600 cores they are able to keep busy.  Granted, batch jobs probably
use a max of only a few thousand cores, say 12K or so, and I bet on
average they're each busy only 10-15% of the time due to MPI overhead
across that many nodes.

> This is the main reason why doubling the number CPUs does not always
> halve the execution time.

Depends on the application, but I heartily agree.  Obviously for smtp
the bottlenecks are traditionally disk and network, rarely, if ever,
CPU/memory.

> There are also hardware-level issues but their effect usually pales
> in comparison.

Absolutely agreed.  But they often have serious implications on
performance.  More likely are hardware device driver issues than actual
hardware issues.  Here's a good example due to an LSI Logic Linux SCSI
driver change from kernel 2.6.8 to 2.6.9:

From http://bugs.gentoo.org/77334

"The 2.6.10-gentoo-r4 kernel still has the LSI logic SCSI regression
that has been present since 2.6.9

A run of hdparm -t of all my scsi disks shows that the LVD devices (on
channel 1) all are running at their max speed but the SE devices are
running at 1/4 speed (I'm only getting 3MB/sec whereas with the
2.6.8-gentoo-r10 kernel they are getting 14-18MB/sec)"


Imagine doing a distro "security update only" (which included a point
kernel upgrade) on your university Postfix server, with 20,000 mailboxes
and pretty heavy user load, late on a Saturday night, rebooting, and all
comes up fine.  However, you're unaware that it's now dropped your SCSI
throughput from ~18MB/s down to 3MB/s.  Think anyone will notice a
performance difference come Monday morning?  Never, ever, rule out even
the "remotest" of possibilities when troubleshooting.  Gremlins have a
nasty habit of hiding in extremely obscure locations.

In this hypothetical situation, how long would it take Wietse or Victor
to find this university's gremlin and vanquish it?  Pretend you don't
know the answer already, and start with the symptoms described to you in
a frantic phone call or email from you friend, the uni's Postfix op.

--
Stan
Reply | Threaded
Open this post in threaded view
|

Re: Postfix 2.6.x slow

Wietse Venema
Stan Hoeppner:
> running at 1/4 speed (I'm only getting 3MB/sec whereas with the
> [...] kernel they are getting 14-18MB/sec)"

I hope you have those numbers mixed up, and that you meant to write
45MB/s with a good driver and 15MB/s with a bad one. With single-disk
sequential file access of uncached data, 45MB/s is not impressive.

        Wietse
Reply | Threaded
Open this post in threaded view
|

RE: Postfix 2.6.x slow

Eric Vaughn

For our problem, Postfix was not the issue.





-----Original Message-----
From: [hidden email]
[mailto:[hidden email]] On Behalf Of Wietse Venema
Sent: Thursday, October 22, 2009 2:25 PM
To: Postfix users
Subject: Re: Postfix 2.6.x slow

Stan Hoeppner:
> running at 1/4 speed (I'm only getting 3MB/sec whereas with the
> [...] kernel they are getting 14-18MB/sec)"

I hope you have those numbers mixed up, and that you meant to write
45MB/s with a good driver and 15MB/s with a bad one. With single-disk
sequential file access of uncached data, 45MB/s is not impressive.

        Wietse

Reply | Threaded
Open this post in threaded view
|

Re: Postfix 2.6.x slow

Stan Hoeppner
In reply to this post by Wietse Venema
Wietse Venema put forth on 10/22/2009 4:25 PM:
> Stan Hoeppner:
>> running at 1/4 speed (I'm only getting 3MB/sec whereas with the
>> [...] kernel they are getting 14-18MB/sec)"
>
> I hope you have those numbers mixed up, and that you meant to write
> 45MB/s with a good driver and 15MB/s with a bad one. With single-disk
> sequential file access of uncached data, 45MB/s is not impressive.

That's not me talking Wietse, it's a quote from an OP with the problem
who posted comments on the Gentoo kernel bug a few years ago, and his
disks were apparently a little old at that time, SE vs LVD.  The last
gen of SE drives were Ultra Wide SCSI and topped out at 40 MB/s sync on
the SCSI bus.  LVD SCSI is still with us (not for long due to SAS) and
tops out at 320 MB/s sync on the bus.  Obviously we still don't have
drives capable of pushing seq I/O anywhere close to 320 MB/s, though
most easily eclipse 40 MB/s.

Forget the hard numbers for a minute and concentrate on the ratio of
performance degradation.  He was seeing a 5 to 6 fold decrease in
sequential I/O performance after updating his kernel, which contained
the new buggy kernel SCSI driver.  Ransom I/O throughput would be a
little worse yet.

The point I was attempting to make is that, even with todays fast disks,
on a heavily loaded Postfix server, a 6 fold decrease in disk throughput
due to an obscure bug like this would likely wreak havoc for a few
hours, if not days, depending on the skill and experience of the OP,
before the problem were found and fixed.  Ergo, we should never rule out
the rare/obscure/unlikely possible causes of problems that pop up.

--
Stan
Reply | Threaded
Open this post in threaded view
|

Re: Postfix 2.6.x slow

Wietse Venema
Stan Hoeppner:
> The point I was attempting to make is that, even with todays fast disks,
> on a heavily loaded Postfix server, a 6 fold decrease in disk throughput
> due to an obscure bug like this would likely wreak havoc for a few
> hours, if not days, depending on the skill and experience of the OP,
> before the problem were found and fixed.  Ergo, we should never rule out
> the rare/obscure/unlikely possible causes of problems that pop up.

I guess that the lesson from this is: don't install bleeding-edge
kernels on servers that people depend on. Pretty much every OS
distribution has a QA process that catches such anomalies before
too many people suffer.

I have been doing UNIX since 1985. I have learned to be careful.

        Wietse
12