Upgraded to 3.4 today. All logging has Stopped?

classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|

Upgraded to 3.4 today. All logging has Stopped?

rachalmers

I upgraded to and installed 3.4 today. It appears to be running, but all logging has stopped. The last entry in the log file was at the exact time I started the new version.

I’m also having trouble with the Mail program but that’s another story. Which is why I’m trying contact via my iPad.

Local mail, via sendmail seems to be working as does mail going off site again via sendmail. But no logging.
I haven’t changed my main.cf nor the master.cf in ages now, so what’s changed I wonder.
Thanks
Robert

Reply | Threaded
Open this post in threaded view
|

Re: Upgraded to 3.4 today. All logging has Stopped?

Viktor Dukhovni

> On Nov 29, 2018, at 9:15 AM, Robert Chalmers <[hidden email]> wrote:
>
> I upgraded to and installed 3.4 today. It appears to be running, but all logging has stopped. The last entry in the log file was at the exact time I started the new version.

Apple changed the syslog API to use the os_log(3) interface, but apparently
if you compile on a sufficiently old MacOS/X system, you get linked against
the traditional syslog API.

So some people do that, but the os_log(3) stuff is configurable, so you
should be able to configure the logs to be saved.  The in-memory logs
can be queried with log(1).  For example:

  # postfix reload
  postfix/postfix-script: refreshing the Postfix mail system

  # log show --info --style syslog --last 2m --predicate 'eventType == logEvent'  | grep libpostfix-util
  2018-11-29 12:02:26.765215-0500  localhost postlog[59348]: (libpostfix-util.dylib) refreshing the Postfix mail system
  2018-11-29 12:02:26.776443-0500  localhost master[33680]: (libpostfix-util.dylib) reload -- version 3.4-20181125, configuration /var/tmp/postfix/etc

Someone should figure out how create an asl.conf(5) configuration
that causes appropriate Postfix logs to land on disk, and share
the recipe.

I should note that the new API appears to strip everything up to
the last "/" in the log name, so we're losing the "postfix/"
prefixes.  Might need to change the code (on MacOS) to send

        postfix:master

rather than

        postfix/master

--
        Viktor.

Reply | Threaded
Open this post in threaded view
|

Re: Upgraded to 3.4 today. All logging has Stopped?

rachalmers
Interesting - but I'm at a loss as to how to actually solve the logging
problem here. It's a bit iffy with no Postfix mail logs. Dovecot is logging
fine, but that's only half the story.




--
Sent from: http://postfix.1071664.n5.nabble.com/Postfix-Users-f2.html
Reply | Threaded
Open this post in threaded view
|

Re: Upgraded to 3.4 today. All logging has Stopped?

Bill Cole-3
In reply to this post by Viktor Dukhovni
On 29 Nov 2018, at 12:07, Viktor Dukhovni wrote:

>> On Nov 29, 2018, at 9:15 AM, Robert Chalmers <[hidden email]>
>> wrote:
>>
>> I upgraded to and installed 3.4 today. It appears to be running, but
>> all logging has stopped. The last entry in the log file was at the
>> exact time I started the new version.
>
> Apple changed the syslog API to use the os_log(3) interface, but
> apparently
> if you compile on a sufficiently old MacOS/X system, you get linked
> against
> the traditional syslog API.
>
> So some people do that, but the os_log(3) stuff is configurable, so
> you
> should be able to configure the logs to be saved.

The word "should" has 2 distinct meanings...

There's already a default /etc/asl/com.apple.mail file which *according
to Apple's docs* should cause mail messages to go into
/var/log/mail.log, but they do not. Replicating the configuration in
/etc/asl.conf which seems to make some messages land in
/var/log/system.log does not work for any other target. None of the
/etc/asl/com.apple.* files which seem to want to cause messages to go
into log files actually generates any log files.

I agree that there *SHOULD* be a way to cause the broken "unified
logging" subsystem to save custom logs. That would be a good thing. I do
not believe that a working documented mechanism exists to do so. It
seems to me to be unlikely that anyone would stumble across whatever
magic is needed to achieve this result.


> The in-memory logs
> can be queried with log(1).  For example:
>
>   # postfix reload
>   postfix/postfix-script: refreshing the Postfix mail system
>
>   # log show --info --style syslog --last 2m --predicate 'eventType ==
> logEvent'  | grep libpostfix-util
>   2018-11-29 12:02:26.765215-0500  localhost postlog[59348]:
> (libpostfix-util.dylib) refreshing the Postfix mail system
>   2018-11-29 12:02:26.776443-0500  localhost master[33680]:
> (libpostfix-util.dylib) reload -- version 3.4-20181125, configuration
> /var/tmp/postfix/etc
>
> Someone should figure out how create an asl.conf(5) configuration
> that causes appropriate Postfix logs to land on disk, and share
> the recipe.

As I said above, I'm not sure that's possible. However, one could launch
this persistently in the background (either via launchd or even 'batch'
if you're into such things...) and get an emulation of sane mail
logging:

log stream --info --predicate 'senderImagePath CONTAINS "postfix"'
--style syslog >> /var/log/mail.log


--
Bill Cole
[hidden email] or [hidden email]
(AKA @grumpybozo and many *@billmail.scconsult.com addresses)
Available For Hire: https://linkedin.com/in/billcole
Reply | Threaded
Open this post in threaded view
|

Re: Upgraded to 3.4 today. All logging has Stopped?

James Brown
In reply to this post by Viktor Dukhovni

> On 30 Nov 2018, at 4:07 am, Viktor Dukhovni <[hidden email]> wrote:
>
>
>> On Nov 29, 2018, at 9:15 AM, Robert Chalmers <[hidden email]> wrote:
>>
>> I upgraded to and installed 3.4 today. It appears to be running, but all logging has stopped. The last entry in the log file was at the exact time I started the new version.
>
> Apple changed the syslog API to use the os_log(3) interface, but apparently
> if you compile on a sufficiently old MacOS/X system, you get linked against
> the traditional syslog API.
>
> So some people do that, but the os_log(3) stuff is configurable, so you
> should be able to configure the logs to be saved.  The in-memory logs
> can be queried with log(1).  For example:
>
>  # postfix reload
>  postfix/postfix-script: refreshing the Postfix mail system
>
>  # log show --info --style syslog --last 2m --predicate 'eventType == logEvent'  | grep libpostfix-util
>  2018-11-29 12:02:26.765215-0500  localhost postlog[59348]: (libpostfix-util.dylib) refreshing the Postfix mail system
>  2018-11-29 12:02:26.776443-0500  localhost master[33680]: (libpostfix-util.dylib) reload -- version 3.4-20181125, configuration /var/tmp/postfix/etc
>
> Someone should figure out how create an asl.conf(5) configuration
> that causes appropriate Postfix logs to land on disk, and share
> the recipe.
>
> I should note that the new API appears to strip everything up to
> the last "/" in the log name, so we're losing the "postfix/"
> prefixes.  Might need to change the code (on MacOS) to send
>
> postfix:master
>
> rather than
>
> postfix/master
>
> --
> Viktor

Thanks Viktor. It would be great if Postfix would log to disk on newer versions of macOS X like it did before. My Mojave test mail server has Dovecot logging to /var/log/mail.log but Postfix doesn’t.

Has anyone managed to do this? I’d rather not have to compile on old Mac and transfer.

Really need logging to disk for fail2ban to work with Postfix.

James.
Reply | Threaded
Open this post in threaded view
|

Re: Upgraded to 3.4 today. All logging has Stopped?

Wietse Venema
James Brown:

>
> > On 30 Nov 2018, at 4:07 am, Viktor Dukhovni <[hidden email]> wrote:
> >
> >
> >> On Nov 29, 2018, at 9:15 AM, Robert Chalmers <[hidden email]> wrote:
> >>
> >> I upgraded to and installed 3.4 today. It appears to be running, but all logging has stopped. The last entry in the log file was at the exact time I started the new version.
> >
> > Apple changed the syslog API to use the os_log(3) interface, but apparently
> > if you compile on a sufficiently old MacOS/X system, you get linked against
> > the traditional syslog API.
> >
> > So some people do that, but the os_log(3) stuff is configurable, so you
> > should be able to configure the logs to be saved.  The in-memory logs
> > can be queried with log(1).  For example:
> >
> >  # postfix reload
> >  postfix/postfix-script: refreshing the Postfix mail system
> >
> >  # log show --info --style syslog --last 2m --predicate 'eventType == logEvent'  | grep libpostfix-util
> >  2018-11-29 12:02:26.765215-0500  localhost postlog[59348]: (libpostfix-util.dylib) refreshing the Postfix mail system
> >  2018-11-29 12:02:26.776443-0500  localhost master[33680]: (libpostfix-util.dylib) reload -- version 3.4-20181125, configuration /var/tmp/postfix/etc
> >
> > Someone should figure out how create an asl.conf(5) configuration
> > that causes appropriate Postfix logs to land on disk, and share
> > the recipe.
> >
> > I should note that the new API appears to strip everything up to
> > the last "/" in the log name, so we're losing the "postfix/"
> > prefixes.  Might need to change the code (on MacOS) to send
> >
> > postfix:master
> >
> > rather than
> >
> > postfix/master
> >
> > --
> > Viktor
>
> Thanks Viktor. It would be great if Postfix would log to disk on newer versions of macOS X like it did before. My Mojave test mail server has Dovecot logging to /var/log/mail.log but Postfix doesn?t.
>
> Has anyone managed to do this? I?d rather not have to compile on old Mac and transfer.
>
> Really need logging to disk for fail2ban to work with Postfix.

For that, you will have to build Postfix on an older MacOS. Postfix
does not decide where the logging is sent. Postfix logs to the
syslog port as it has been doing for 20+ years.

        Wietse

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: Upgraded to 3.4 today. All logging has Stopped?

Larry Stone
In reply to this post by James Brown
On Jan 9, 2019, at 19:01, James Brown <[hidden email]> wrote:
>
> Thanks Viktor. It would be great if Postfix would log to disk on newer versions of macOS X like it did before. My Mojave test mail server has Dovecot logging to /var/log/mail.log but Postfix doesn’t.

Is this a recent build of Dovecot or was it built on an older version of MacOS before the logging changes? If the former, ask on the Dovecot list how they did it. If the latter, it’s a meaningless data point until Dovecot is rebuilt on a newer version of MacOS.

> Has anyone managed to do this? I’d rather not have to compile on old Mac and transfer.

Not as far as any of us know. It’s been discussed here before and no solution has been found.

— Larry Stone
   [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Upgraded to 3.4 today. All logging has Stopped?

James Brown
On 10 Jan 2019, at 2:01 pm, Larry Stone <[hidden email]> wrote:

On Jan 9, 2019, at 19:01, James Brown <[hidden email]> wrote:

Thanks Viktor. It would be great if Postfix would log to disk on newer versions of macOS X like it did before. My Mojave test mail server has Dovecot logging to /var/log/mail.log but Postfix doesn’t.

Is this a recent build of Dovecot or was it built on an older version of MacOS before the logging changes? If the former, ask on the Dovecot list how they did it. If the latter, it’s a meaningless data point until Dovecot is rebuilt on a newer version of MacOS.

Has anyone managed to do this? I’d rather not have to compile on old Mac and transfer.

Not as far as any of us know. It’s been discussed here before and no solution has been found.

— Larry Stone
  [hidden email]

Hi Larry. It’s a recent build of Dovecot, compiled on Mojave. Steps to compile were:

export CPPFLAGS="-I/usr/local/opt/mysql@5.7/include/mysql -I/usr/local/Cellar/openssl@1.1/1.1.1/include"
export LDFLAGS="-L/usr/local/opt/mysql@5.7/lib -L/usr/local/opt/openssl@1.1/lib"

./configure --with-mysql --with-ssl=openssl

make

sudo make install

The setting file for logging, “etc/dovecot/conf.d/10-logging.conf” does have this:

##
## Log destination.
##

# Log file to use for error messages. "syslog" logs to syslog,
# /dev/stderr logs to stderr.
#log_path = syslog
log_path = /var/log/mail.log

So I’ve had to change this so that it writes directly to the file, and not to syslog.

James.
Reply | Threaded
Open this post in threaded view
|

Re: Upgraded to 3.4 today. All logging has Stopped?

Bill Cole-3
In reply to this post by Larry Stone
On 9 Jan 2019, at 22:01, Larry Stone wrote:

>> Has anyone managed to do this? I’d rather not have to compile on
>> old Mac and transfer.
>
>
> Not as far as any of us know. It’s been discussed here before and no
> solution has been found.


As I suggested back when this thread was young, this command will put
all of Postfix's log messages into /var/log/mail.log:

    log stream --info --predicate 'senderImagePath CONTAINS "postfix"'
--style syslog >> /var/log/mail.log

It's trivial to create a launchd .plist file to run that at boot time
and let you start/stop it with launchctl.

That may be a bit heavy for a busy server because it pipes all of the
log messages through a userspace process, but it will do what you want.

It remains true that macOS has been unfit for most server work since
Sierra as a direct result of conscious choices by Apple to make it so.
Even if the ugly hack above works for you, it's just a battle tactic in
a war that cannot be won. Apple has clearly demonstrated that they want
macOS to be not be used for server duty.

--
Bill Cole
[hidden email] or [hidden email]
(AKA @grumpybozo and many *@billmail.scconsult.com addresses)
Available For Hire: https://linkedin.com/in/billcole
Reply | Threaded
Open this post in threaded view
|

Re: Upgraded to 3.4 today. All logging has Stopped?

Larry Stone
In reply to this post by James Brown




> On Jan 9, 2019, at 9:48 PM, James Brown <[hidden email]> wrote:
>
>> On 10 Jan 2019, at 2:01 pm, Larry Stone <[hidden email]> wrote:
>>
>> Is this a recent build of Dovecot or was it built on an older version of MacOS before the logging changes? If the former, ask on the Dovecot list how they did it. If the latter, it’s a meaningless data point until Dovecot is rebuilt on a newer version of MacOS.
>>
>
> Hi Larry. It’s a recent build of Dovecot, compiled on Mojave.
...

>
> The setting file for logging, “etc/dovecot/conf.d/10-logging.conf” does have this:
>
> ##
> ## Log destination.
> ##
>
> # Log file to use for error messages. "syslog" logs to syslog,
> # /dev/stderr logs to stderr.
> #log_path = syslog
> log_path = /var/log/mail.log
>
> So I’ve had to change this so that it writes directly to the file, and not to syslog.

Ah. So Dovecot has the ability to write logs directly. I believe Wietse has stated in the past that no such capability exists in Postfix and it only logs to the syslog daemon. And it’s the changes Apple has made to syslog that are the issue.

Bill Cole posted (again) a workaround that you can pursue. Beyond that, unless Wietse decides to modify Postfix’s logging to support alternate methods such as Dovecot does (and I have not the slightest clue how involved that might be - you’re welcome to do it yourself if you’re so inclined), we really don’t have a solution given Apple’s decision to move away from the Unix standard for logging.

Bill also stated that MacOS is no longer a suitable platform for being a server and I largely concur. I used to run a full mail server but gave up on that three years ago and moved my mail to an outside service (and as I now have a provider that blocks port 25, not an option for me anymore anyway). I still run Postfix but only for getting system generated emails off the system to my outside mail service (I have some system status processes that alert me to various issues that can occur) so logging is not the concern for me today that it was when I was running the full server.

--
Larry Stone
[hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Upgraded to 3.4 today. All logging has Stopped?

rachalmers
I ran into this myself a little while back. Used Bill’s logging example until I ironed out a few glitches. Now everything is working ok I don’t use it.

I’m on a Mac by the way. OSX 10.14.2 and run an apache vhosts web setup, Postfix with Dovecot + MySQL, and Clamd, freshclam etc etc. Postfix and Dovecot compiled myself to get around Apple’s broken installs. I think everything else is either brew or macports, like Unbound, openssl, dmarcian and so on.

It’s a 2TB SSD system now, with 16GB RAM, on a SKY Fibre MAX line. And yes, it’s dynamic DNS much to my disappointment and they won’t change it. And for some obscure reason SKY have listed the netblock in the spamhause rbl  database. Which I have to keep unlisting it from. But for the radio stream I use NOIP.com, and for the mail server when/if the IP changes - and it’s noticable as soon as it does, I just go to the DNS config on my provider and reset it to the current. Fortunately it doesn’t change often because the server and router  is never turned off. Rarely anyway. So I’m making do, and that’s hardly Apple’s fault.

As a server it runs ok for my needs. But I don’t have an office full of users so can’t comment on it’s scalability.

Dovecot logging works fine.
Postfix if I use Bill Cole’s option. It’s also possible to fine tune that command line to get better granularity - see the man pages and the online wiki.




> On 10 Jan 2019, at 15:54, Larry Stone <[hidden email]> wrote:
>
>
>
>
>
>> On Jan 9, 2019, at 9:48 PM, James Brown <[hidden email]> wrote:
>>
>>> On 10 Jan 2019, at 2:01 pm, Larry Stone <[hidden email]> wrote:
>>>
>>> Is this a recent build of Dovecot or was it built on an older version of MacOS before the logging changes? If the former, ask on the Dovecot list how they did it. If the latter, it’s a meaningless data point until Dovecot is rebuilt on a newer version of MacOS.
>>>
>>
>> Hi Larry. It’s a recent build of Dovecot, compiled on Mojave.
> ...
>>
>> The setting file for logging, “etc/dovecot/conf.d/10-logging.conf” does have this:
>>
>> ##
>> ## Log destination.
>> ##
>>
>> # Log file to use for error messages. "syslog" logs to syslog,
>> # /dev/stderr logs to stderr.
>> #log_path = syslog
>> log_path = /var/log/mail.log
>>
>> So I’ve had to change this so that it writes directly to the file, and not to syslog.
>
> Ah. So Dovecot has the ability to write logs directly. I believe Wietse has stated in the past that no such capability exists in Postfix and it only logs to the syslog daemon. And it’s the changes Apple has made to syslog that are the issue.
>
> Bill Cole posted (again) a workaround that you can pursue. Beyond that, unless Wietse decides to modify Postfix’s logging to support alternate methods such as Dovecot does (and I have not the slightest clue how involved that might be - you’re welcome to do it yourself if you’re so inclined), we really don’t have a solution given Apple’s decision to move away from the Unix standard for logging.
>
> Bill also stated that MacOS is no longer a suitable platform for being a server and I largely concur. I used to run a full mail server but gave up on that three years ago and moved my mail to an outside service (and as I now have a provider that blocks port 25, not an option for me anymore anyway). I still run Postfix but only for getting system generated emails off the system to my outside mail service (I have some system status processes that alert me to various issues that can occur) so logging is not the concern for me today that it was when I was running the full server.
>
> --
> Larry Stone
> [hidden email]
>


Reply | Threaded
Open this post in threaded view
|

Re: Upgraded to 3.4 today. All logging has Stopped?

Wietse Venema
In reply to this post by Larry Stone
Larry Stone:

> > # Log file to use for error messages. "syslog" logs to syslog,
> > # /dev/stderr logs to stderr.
> > #log_path = syslog
> > log_path = /var/log/mail.log
> >
> > So I?ve had to change this so that it writes directly to the file, and not to syslog.
>
> Ah. So Dovecot has the ability to write logs directly. I believe
> Wietse has stated in the past that no such capability exists in
> Postfix and it only logs to the syslog daemon. And it?s the changes
> Apple has made to syslog that are the issue.

Is that better than Bill Cole's solution to run a log exporter at
system startup?

If both Dovecot and Postfix write to the same logfile, that would
be a disaster.

- The only way to make multiple logfile writers safe is that each
  logfile writer flushes its own buffers after every log call, and
  that would be disastrous for performance. See the Postfix
  LINUX_README for a discussion. It may be OK for MacOS but it is
  not good for real servers.

- If individual programs write directly to the logfile, flushing
  after every log call is also required to avoid losing logs when
  a program crashes, and that is when logs are needed most.

- The only way to make logging performant is to have a single writer
  that has a limited-size write buffer (like syslogd and rsyslogd).

Therefore,

- Postfix and Dovecot cannot share logfiles. But there is nothing
  to enforce that, because there are no mandatory locks.

- Postfix needs its own logger daemon, which brings major challenges
  when Postfix is not (yet) running.

  - What happens with logging during Postfix startup?
    Hack the log client code to directly write to the logfile?  Will
    it even be allowed to write outside the Postfix queue? If every
    program opens the logfile as root, it has to make sure that the
    file is not a symlink, has no multiple hard links, etc.

  - What happens with logging from non-daemon programs when Postix
    is down? Unless the logfile is world-writable, those prograns
    will have nowhere to log. This affects programs that invoke
    /bin/mail before Postfix is up; we should not assume that such
    programs will always run as root.

  - Log rotation support. Postfix cannot keep appending to the
    same file forever. It may be OK for MacOS but it is not good
    for real servers. Basically re-invent the log rotation wheel.

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: Upgraded to 3.4 today. All logging has Stopped?

Larry Stone
>
> On Jan 10, 2019, at 11:08 AM, Wietse Venema <[hidden email]> wrote:
>
> If both Dovecot and Postfix write to the same logfile, that would
> be a disaster.
>


Thanks to Wietse for that detailed explanation of all the issues involved with attempting to roll your own logging system. Lots of issues I never thought about and as a result, my knowledge of the subject has been greatly expanded.

--
Larry Stone
[hidden email]


Reply | Threaded
Open this post in threaded view
|

Re: Upgraded to 3.4 today. All logging has Stopped?

James Brown
In reply to this post by Wietse Venema
On 11 Jan 2019, at 4:08 am, Wietse Venema <[hidden email]> wrote:

>
> Larry Stone:
>>> # Log file to use for error messages. "syslog" logs to syslog,
>>> # /dev/stderr logs to stderr.
>>> #log_path = syslog
>>> log_path = /var/log/mail.log
>>>
>>> So I?ve had to change this so that it writes directly to the file, and not to syslog.
>>
>> Ah. So Dovecot has the ability to write logs directly. I believe
>> Wietse has stated in the past that no such capability exists in
>> Postfix and it only logs to the syslog daemon. And it?s the changes
>> Apple has made to syslog that are the issue.
>
> Is that better than Bill Cole's solution to run a log exporter at
> system startup?
>
> If both Dovecot and Postfix write to the same logfile, that would
> be a disaster.
>
> - The only way to make multiple logfile writers safe is that each
>  logfile writer flushes its own buffers after every log call, and
>  that would be disastrous for performance. See the Postfix
>  LINUX_README for a discussion. It may be OK for MacOS but it is
>  not good for real servers.
>
> - If individual programs write directly to the logfile, flushing
>  after every log call is also required to avoid losing logs when
>  a program crashes, and that is when logs are needed most.
>
> - The only way to make logging performant is to have a single writer
>  that has a limited-size write buffer (like syslogd and rsyslogd).
>
> Therefore,
>
> - Postfix and Dovecot cannot share logfiles. But there is nothing
>  to enforce that, because there are no mandatory locks.
>
> - Postfix needs its own logger daemon, which brings major challenges
>  when Postfix is not (yet) running.
>
>  - What happens with logging during Postfix startup?
>    Hack the log client code to directly write to the logfile?  Will
>    it even be allowed to write outside the Postfix queue? If every
>    program opens the logfile as root, it has to make sure that the
>    file is not a symlink, has no multiple hard links, etc.
>
>  - What happens with logging from non-daemon programs when Postix
>    is down? Unless the logfile is world-writable, those prograns
>    will have nowhere to log. This affects programs that invoke
>    /bin/mail before Postfix is up; we should not assume that such
>    programs will always run as root.
>
>  - Log rotation support. Postfix cannot keep appending to the
>    same file forever. It may be OK for MacOS but it is not good
>    for real servers. Basically re-invent the log rotation wheel.
>
> Wietse

Thanks Wietse, Larry, Robert and Bill. I really appreciate your help.

Wietse, thanks for pointing out all the problems of Postfix logging without syslog.

I have created a script that runs Bill’s log command to send it to a file. Not the same log file that Dovecot is using. Created a LaunchDaemon to open the script at startup.

Seems to work perfectly, so thanks again everyone.

James.