Keep Postfix running in the foreground

classic Classic list List threaded Threaded
50 messages Options
123
Reply | Threaded
Open this post in threaded view
|

Re: Keep Postfix running in the foreground

Viktor Dukhovni


> On Apr 2, 2018, at 12:42 PM, Wietse Venema <[hidden email]> wrote:
>
> To make the master 'pid 1' one would have to use 'exec
> $daemon_directory/master' in the 'postfix-script' file.
>
> However, that fails on every system that I know:
>
>    postfix-master[xxx]: fatal: unable to set session and process group ID: Operation not permitted

A simple minder process that propagates signal to its child process
would probably solve this issue.

        sigprop cmd [args]

run "cmd" as a child process, catch and propagate all signals that can
be caught to the child, exit with the child's exit code.

Because "sigprop" has just one child it would not need a process
group, ... it would just kill the foreground master process.

With that out of the way, perhaps the solution then is for the foreground
master process to be that minder, running a child master process to do
all the work, with the parent managing just the one child, this avoids
the need for a separate (if general purpose) minder.  A violation of
the Unix philosophy perhaps, but perhaps a reasonable approach?

--
        Viktor.

Reply | Threaded
Open this post in threaded view
|

Re: Keep Postfix running in the foreground

Wietse Venema
In reply to this post by Wietse Venema
Wietse Venema:
> To make the master 'pid 1' one would have to use 'exec
> $daemon_directory/master' in the 'postfix-script' file.
>
> However, that fails on every system that I know:
>
>     postfix-master[xxx]: fatal: unable to set session and process group ID: Operation not permitted

Just for the heck of it, what happens when you change postfix-script
to do this (delete line starting with -, add lines starting with +):

        start-fg)
            # Foreground start-up is incompatible with multi-instance mode.
            # We can't use "exec $daemon_directory/master" here: that would
            # break process group management, and "postfix stop" would kill
            # too many processes.
            case $instances in
-           "") exec $daemon_directory/master
+           "") exec $daemon_directory/master -d
+               $FATAL "could not start-fg $daemon_directory/master"
                ;;
             *) $FATAL "start-fg does not support multi_instance_directories"
                exit 1
                ;;
            esac

See what happens with 'postfix stop' and if it kills too much.

        Wietse

> The master daemon must run in its own process group and session,
> so that after receiving a "postfix stop" signal, the master can
> terminate all Postfix daemon processes including processes that
> 'hang', by sending a signal to the process group.
>
> If the master would not have its own process group and session, it
> would kill too many processes. People who ran the master that way
> found that Postfix would kill its parent, i.e. the real system init
> process, and shutdown the system. That would be unacceptable.
>
> If the master would not kill all Postfix daemon processes including
> processes that 'hang', it would be impossible to restart Postfix
> without a human manually killing off the hanging process. That would
> also be unacceptable.
>
> Wietse
>
Reply | Threaded
Open this post in threaded view
|

Re: Keep Postfix running in the foreground

John Allen
In reply to this post by Viktor Dukhovni
Dumb question I suspect - what is the attraction of docker? What does it do that I might need?
JohnA

On April 2, 2018 1:22:29 PM Viktor Dukhovni <[hidden email]> wrote:

On Apr 2, 2018, at 12:42 PM, Wietse Venema <[hidden email]> wrote:

To make the master 'pid 1' one would have to use 'exec
$daemon_directory/master' in the 'postfix-script' file.

However, that fails on every system that I know:

postfix-master[xxx]: fatal: unable to set session and process group ID: Operation not permitted

A simple minder process that propagates signal to its child process
would probably solve this issue.

 sigprop cmd [args]

run "cmd" as a child process, catch and propagate all signals that can
be caught to the child, exit with the child's exit code.

Because "sigprop" has just one child it would not need a process
group, ... it would just kill the foreground master process.

With that out of the way, perhaps the solution then is for the foreground
master process to be that minder, running a child master process to do
all the work, with the parent managing just the one child, this avoids
the need for a separate (if general purpose) minder.  A violation of
the Unix philosophy perhaps, but perhaps a reasonable approach?

-- 
 Viktor.

Reply | Threaded
Open this post in threaded view
|

Re: Keep Postfix running in the foreground

Ralph Seichter
On 02.04.2018 19:55, John Allen wrote:

> what is the attraction of docker? What does it do that I might need?

You might need it because a Docker container is the recommended method
to deploy Discourse, which I am doing right now... SCNR. ;-)

-Ralph

Reply | Threaded
Open this post in threaded view
|

Re: Keep Postfix running in the foreground

A. Schulze
In reply to this post by Wietse Venema


Am 02.04.2018 um 19:30 schrieb Wietse Venema:
> -           "") exec $daemon_directory/master
> +           "") exec $daemon_directory/master -d
> +               $FATAL "could not start-fg $daemon_directory/master"

version 3.3.0 don't contain the "exec $daemon_directory/master" but only "$daemon_directory/master"
anyway, the patched version run master as PID 1. fine!

root@postfix:/# ps afx
  PID TTY      STAT   TIME COMMAND
  336 pts/0    Ss     0:00 /bin/bash
  342 pts/0    R+     0:00  \_ ps afx
    1 ?        Ss     0:00 /usr/lib/postfix/master -d
  334 ?        S      0:00 pickup -l -t unix -d -u
  335 ?        S      0:00 qmgr -l -t unix -d -u

or from outside/docker host:
23584 ?        Ssl    0:00  \_ docker-containerd --config /var/run/docker/contai
23794 ?        Sl     0:00      \_ docker-containerd-shim -namespace moby -workd
23811 ?        Ss     0:00          \_ /usr/lib/postfix/master -d
24193 ?        S      0:00              \_ pickup -l -t unix -d -u
24194 ?        S      0:00              \_ qmgr -l -t unix -d -u

But "docker stop $postfix_container" don't let exit master:
23584 ?        Ssl    0:00  \_ docker-containerd --config /var/run/docker/contai
23794 ?        Sl     0:00      \_ docker-containerd-shim -namespace moby -workd
23811 ?        Ss     0:00          \_ /usr/lib/postfix/master -d


I tried "postfix stop" inside the container. That also terminate all child processes but not the master itself.
So a container receiving "stop", still run with only master as PID 1
expected: master process exit and container terminate.

that's better then without the patch as postfix processes are shut down more properly...

> See what happens with 'postfix stop' and if it kills too much.
in a container it don't hurt: there is nothing other

Andreas
Reply | Threaded
Open this post in threaded view
|

Re: Keep Postfix running in the foreground

Wietse Venema
A. Schulze:
>
>
> Am 02.04.2018 um 19:30 schrieb Wietse Venema:
> > -           "") exec $daemon_directory/master
> > +           "") exec $daemon_directory/master -d
> > +               $FATAL "could not start-fg $daemon_directory/master"
>
> version 3.3.0 don't contain the "exec $daemon_directory/master" but only "$daemon_directory/master"

Yes, my mistake.

> anyway, the patched version run master as PID 1. fine!

Good enough for proof of concept. The only misfeature is that it
passes -d to all child processes. I will add a new flag that does
less than '-d', only the things that containers need.

BTW on a non-container host, "postfix stop" stops the master
as expected. You're not supposed to 'postfix stop' from inside
containers anyway.

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: Keep Postfix running in the foreground

John Stoffel-2
In reply to this post by Ralph Seichter
>>>>> "Ralph" == Ralph Seichter <[hidden email]> writes:

Ralph> On 02.04.2018 19:55, John Allen wrote:
>> what is the attraction of docker? What does it do that I might need?

Ralph> You might need it because a Docker container is the recommended method
Ralph> to deploy Discourse, which I am doing right now... SCNR. ;-)

But... isn't discourse running in it's own container, so you'd be
spinning up postfix it another container... when it seems simpler to
have a central host to handle all email from the various contrainers
you spin up...

But I'm happy to learn why.

Reply | Threaded
Open this post in threaded view
|

Re: Keep Postfix running in the foreground

Eray Aslan-2
In reply to this post by Viktor Dukhovni
On Mon, Apr 02, 2018 at 01:21:41PM -0400, Viktor Dukhovni wrote:
> A simple minder process that propagates signal to its child process
> would probably solve this issue.

On a somewhat related note, dumb-init[1] and tini[2] are populer
choices to run as pid 1 inside containers.  Perhaps, they offer an
easier solution.

--
Eray

[1] https://github.com/Yelp/dumb-init
[2] https://github.com/krallin/tini
Reply | Threaded
Open this post in threaded view
|

Re: Keep Postfix running in the foreground

Wietse Venema
In reply to this post by Wietse Venema
> > anyway, the patched version run master as PID 1. fine!
>
> Good enough for proof of concept. The only misfeature is that it
> passes -d to all child processes. I will add a new flag that does
> less than '-d', only the things that containers need.

postfix-3.4-20180403 will try to run the master daemon as PID 1.

From the postfix(1) manpage:

   start-fg
          Like start, but keep the master  daemon  running  in  the  fore-
          ground,  if  possible  as  PID  1.  [...]

I updated both the postfix-script file and the master daemon.

I'd appreciate it if someone could verify that this will run the
master daemon with PID 1, and that 'postfix stop' in the container
will stop the master daemon. If it doesn't, then Linux does weird
stuff with PID 1 processes.

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: Keep Postfix running in the foreground

A. Schulze

Wietse Venema:

> I'd appreciate it if someone could verify that this will run the
> master daemon with PID 1, and that 'postfix stop' in the container
> will stop the master daemon.

I'll verify that in the next days ...

Andreas

Reply | Threaded
Open this post in threaded view
|

Re: Keep Postfix running in the foreground

Ralph Seichter
In reply to this post by John Stoffel-2
On 03.04.2018 02:37, John Stoffel wrote:

> But... isn't discourse running in it's own container, so you'd be
> spinning up postfix it another container...

John Allen asked "what is the attraction of docker", and I was just
mucking about with a new Discourse installation, so I mentioned this as
*my* reason for using Docker, tongue-in-cheek. ;-)

While Discourse is distributed only as a Docker image and does not leave
me much choice, I don't run Postfix in Docker, nor do I personally see
any reason to do so, but I know customers who lately want to Docker All
The Things.

-Ralph

Reply | Threaded
Open this post in threaded view
|

Re: Keep Postfix running in the foreground

Eray Aslan-2
In reply to this post by Wietse Venema
On Tue, Apr 03, 2018 at 07:46:42PM -0400, Wietse Venema wrote:
> I updated both the postfix-script file and the master daemon.
>
> I'd appreciate it if someone could verify that this will run the
> master daemon with PID 1, and that 'postfix stop' in the container
> will stop the master daemon. If it doesn't, then Linux does weird
> stuff with PID 1 processes.

Unless I am missing something, postfix stop does not stop the master
daemon inside a container:

$ docker run -d -v /dev/log:/dev/log eraya/postfix:3.4_pre20180403-r1
Unable to find image 'eraya/postfix:3.4_pre20180403-r1' locally
3.4_pre20180403-r1: Pulling from eraya/postfix
973ffedf3041: Already exists
4d8110e24f25: Already exists
bba655fd008e: Already exists
9250e489a277: Already exists
a925f9420666: Already exists
Digest: sha256:534a3645fbb1fedcfb51391cfb5cde7efa0b19057077a67900679b8a47ec875f
Status: Downloaded newer image for eraya/postfix:3.4_pre20180403-r1
a49e9ce8d75de2eb9900bea47188b6ce7bf773cb7b0e10f7f7f5792340d51231
$ docker ps
CONTAINER ID        IMAGE                              COMMAND                  CREATED             STATUS              PORTS               NAMES
a49e9ce8d75d        eraya/postfix:3.4_pre20180403-r1   "/usr/sbin/postfix s…"   7 seconds ago       Up 6 seconds        25/tcp              nostalgic_lumiere
$ docker exec a49e9ce8d75d ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.1  71540  2876 ?        Ss   14:52   0:00 /usr/libexec/postfix/master -i
postfix     76  0.0  0.1  71524  2832 ?        S    14:52   0:00 pickup -l -t unix -u
postfix     77  0.0  0.1  71572  2864 ?        S    14:52   0:00 qmgr -l -t unix -u
root        78  0.0  0.0  17556  1184 ?        Rs   14:52   0:00 ps aux
$ docker exec a49e9ce8d75d postconf mail_version
mail_version = 3.4-20180403
$ docker exec a49e9ce8d75d postfix stop
$ docker ps
CONTAINER ID        IMAGE                              COMMAND                  CREATED              STATUS              PORTS               NAMES
a49e9ce8d75d        eraya/postfix:3.4_pre20180403-r1   "/usr/sbin/postfix s…"   About a minute ago   Up 59 seconds       25/tcp              nostalgic_lumiere

i.e. master daemon still running after postfix stop.  versus starting
with an init:

$ docker run -d -v /dev/log:/dev/log eraya/postfix:3.4_pre20180403
172645c305b89f9149d3cd14559da04c42bd3b369eaeb08253ec55b7032a61ed
$ docker ps
CONTAINER ID        IMAGE                           COMMAND                  CREATED             STATUS              PORTS               NAMES
172645c305b8        eraya/postfix:3.4_pre20180403   "/usr/local/bin/dumb…"   8 seconds ago       Up 7 seconds        25/tcp              naughty_edison
$ docker exec 172645c305b8 ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.0    188     4 ?        Ss   14:48   0:00 /usr/local/bin/dumb-init postfix start-fg
root         5  0.0  0.0  18352  1696 ?        Ss   14:48   0:00 /bin/sh /usr/libexec/postfix/postfix-script start-fg
root        77  0.0  0.1  71540  2880 ?        Ss   14:48   0:00 /usr/libexec/postfix/master
postfix     78  0.0  0.1  71524  2836 ?        S    14:48   0:00 pickup -l -t unix -u
postfix     79  0.0  0.1  71572  2864 ?        S    14:48   0:00 qmgr -l -t unix -u
root        80  0.0  0.0  17556  1184 ?        Rs   14:48   0:00 ps aux
$ docker exec 172645c305b8 postconf mail_version
mail_version = 3.4-20180403
$ docker exec 172645c305b8 postfix stop
$ docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

i.e. no running container after postfix stop.

--
Eray
Reply | Threaded
Open this post in threaded view
|

Re: Keep Postfix running in the foreground

Wietse Venema
Eray Aslan:
> On Tue, Apr 03, 2018 at 07:46:42PM -0400, Wietse Venema wrote:
> > I updated both the postfix-script file and the master daemon.
> >
> > I'd appreciate it if someone could verify that this will run the
> > master daemon with PID 1, and that 'postfix stop' in the container
> > will stop the master daemon. If it doesn't, then Linux does weird
> > stuff with PID 1 processes.

Thanks for the detailed reports.

> Unless I am missing something, postfix stop does not stop the master
> daemon inside a container:

Confirming that Linux does weird stuff with signals and PID 1 processes
(it ignores the equivalent of "kill -9 myself" that I added to the code).

> $ docker exec a49e9ce8d75d ps aux
> USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
> root         1  0.0  0.1  71540  2876 ?        Ss   14:52   0:00 /usr/libexec/postfix/master -i
> postfix     76  0.0  0.1  71524  2832 ?        S    14:52   0:00 pickup -l -t unix -u
> postfix     77  0.0  0.1  71572  2864 ?        S    14:52   0:00 qmgr -l -t unix -u
> root        78  0.0  0.0  17556  1184 ?        Rs   14:52   0:00 ps aux

That looks as intended: the master runs with PID 1.

> USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
> root         1  0.0  0.0    188     4 ?        Ss   14:48   0:00 /usr/local/bin/dumb-init postfix start-fg
> root         5  0.0  0.0  18352  1696 ?        Ss   14:48   0:00 /bin/sh /usr/libexec/postfix/postfix-script start-fg
> root        77  0.0  0.1  71540  2880 ?        Ss   14:48   0:00 /usr/libexec/postfix/master
> postfix     78  0.0  0.1  71524  2836 ?        S    14:48   0:00 pickup -l -t unix -u
> postfix     79  0.0  0.1  71572  2864 ?        S    14:48   0:00 qmgr -l -t unix -u

And that looks as expected when master is not running as PID 1.

> $ docker exec 172645c305b8 postfix stop
> $ docker ps
> CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
>
> i.e. no running container after postfix stop.

The dumb-init program terminates after 'postfix start-fg' terminates.

    if (killed_pid == child_pid) {
        forward_signal(SIGTERM);  // send SIGTERM to any remaining children
        DEBUG("Child exited with status %d. Goodbye.\n", exit_status);
        exit(exit_status);
    }

I.e. no Linux-specific hackery.

Just for the heck of it, can you replace in src/master/master_sig.c
this code:

    if (kill(pid, SIGKILL) < 0)
        msg_fatal("%s: kill myself: %m", myname);

With this code:

    exit(0);

And see if that fixes the PID=1 behavior?

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: Keep Postfix running in the foreground

Viktor Dukhovni


> On Apr 4, 2018, at 1:08 PM, Wietse Venema <[hidden email]> wrote:
>
> Just for the heck of it, can you replace in src/master/master_sig.c
> this code:
>
>    if (kill(pid, SIGKILL) < 0)
>        msg_fatal("%s: kill myself: %m", myname);
>
> With this code:
>
>    exit(0);
>
> And see if that fixes the PID=1 behavior?

Perhaps instead of "exit(0)" it should be "_exit(0)",
since exit(3) is not async-signal-safe.

--
        Viktor.

Reply | Threaded
Open this post in threaded view
|

Re: Keep Postfix running in the foreground

Wietse Venema
Wietse Venema:

> Just for the heck of it, can you replace in src/master/master_sig.c
> this code:
>
>    if (kill(pid, SIGKILL) < 0)
>        msg_fatal("%s: kill myself: %m", myname);
>
> With this code:
>
>    exit(0);
>
> And see if that fixes the PID=1 behavior?

Viktor Dukhovni:
> Perhaps instead of "exit(0)" it should be "_exit(0)",
> since exit(3) is not async-signal-safe.

According to RTFM, the exit() function does not return so it can't 'fail'.

FreeBSD:
     The exit() and _Exit() functions never return.
     The _exit() system call can never return.
Linux:
     The exit() function does not return.
     (Similar text for _exit and_Exit).

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: Keep Postfix running in the foreground

Viktor Dukhovni


> On Apr 4, 2018, at 1:31 PM, Wietse Venema <[hidden email]> wrote:
>
> According to RTFM, the exit() function does not return so it can't 'fail'.
>
> FreeBSD:
>     The exit() and _Exit() functions never return.
>     The _exit() system call can never return.
> Linux:
>     The exit() function does not return.
>     (Similar text for _exit and_Exit).

Yes, but it can dead-lock trying to flush stdio buffers, and the like.
And it can dead-lock with threads running malloc(), ... it is not safe
to call in a signal handler, while _exit() should be.

--
        Viktor.

Reply | Threaded
Open this post in threaded view
|

Re: Keep Postfix running in the foreground

Wietse Venema
Viktor Dukhovni:

>
>
> > On Apr 4, 2018, at 1:31 PM, Wietse Venema <[hidden email]> wrote:
> >
> > According to RTFM, the exit() function does not return so it can't 'fail'.
> >
> > FreeBSD:
> >     The exit() and _Exit() functions never return.
> >     The _exit() system call can never return.
> > Linux:
> >     The exit() function does not return.
> >     (Similar text for _exit and_Exit).
>
> Yes, but it can dead-lock trying to flush stdio buffers, and the like.
> And it can dead-lock with threads running malloc(), ... it is not safe
> to call in a signal handler, while _exit() should be.

Eray, can you please do the test and ignore this discussion?

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: Keep Postfix running in the foreground

A. Schulze
In reply to this post by Wietse Venema


Am 04.04.2018 um 19:08 schrieb Wietse Venema:
> Eray Aslan:
>> On Tue, Apr 03, 2018 at 07:46:42PM -0400, Wietse Venema wrote:
>>> I updated both the postfix-script file and the master daemon.
>>>
>>> I'd appreciate it if someone could verify that this will run the
>>> master daemon with PID 1, and that 'postfix stop' in the container
>>> will stop the master daemon. If it doesn't, then Linux does weird
>>> stuff with PID 1 processes.

I found the same...

> Just for the heck of it, can you replace in src/master/master_sig.c
> this code:
>
>     if (kill(pid, SIGKILL) < 0)
>         msg_fatal("%s: kill myself: %m", myname);
>
> With this code:
>
>     exit(0);
>
> And see if that fixes the PID=1 behavior?

it does in any way. Thanks, Wietse!

I tried both, exit(0) and _exit(0) :-)

Andreas
Reply | Threaded
Open this post in threaded view
|

Re: Keep Postfix running in the foreground

Wietse Venema
A. Schulze:

> Am 04.04.2018 um 19:08 schrieb Wietse Venema:
> > Eray Aslan:
> >> On Tue, Apr 03, 2018 at 07:46:42PM -0400, Wietse Venema wrote:
> >>> I updated both the postfix-script file and the master daemon.
> >>>
> >>> I'd appreciate it if someone could verify that this will run the
> >>> master daemon with PID 1, and that 'postfix stop' in the container
> >>> will stop the master daemon. If it doesn't, then Linux does weird
> >>> stuff with PID 1 processes.
>
> I found the same...
>
> > Just for the heck of it, can you replace in src/master/master_sig.c
> > this code:
> >
> >     if (kill(pid, SIGKILL) < 0)
> >         msg_fatal("%s: kill myself: %m", myname);
> >
> > With this code:
> >
> >     exit(0);
> >
> > And see if that fixes the PID=1 behavior?
>
> it does in any way. Thanks, Wietse!
>
> I tried both, exit(0) and _exit(0) :-)

Andreas, thanks for the encouraging words :-)

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: Keep Postfix running in the foreground

Wietse Venema
Wietse Venema:

> A. Schulze:
> > Am 04.04.2018 um 19:08 schrieb Wietse Venema:
> > > Eray Aslan:
> > >> On Tue, Apr 03, 2018 at 07:46:42PM -0400, Wietse Venema wrote:
> > >>> I updated both the postfix-script file and the master daemon.
> > >>>
> > >>> I'd appreciate it if someone could verify that this will run the
> > >>> master daemon with PID 1, and that 'postfix stop' in the container
> > >>> will stop the master daemon. If it doesn't, then Linux does weird
> > >>> stuff with PID 1 processes.
> >
> > I found the same...
> >
> > > Just for the heck of it, can you replace in src/master/master_sig.c
> > > this code:
> > >
> > >     if (kill(pid, SIGKILL) < 0)
> > >         msg_fatal("%s: kill myself: %m", myname);
> > >
> > > With this code:
> > >
> > >     exit(0);
> > >
> > > And see if that fixes the PID=1 behavior?
> >
> > it does in any way. Thanks, Wietse!
> >
> > I tried both, exit(0) and _exit(0) :-)
>
> Andreas, thanks for the encouraging words :-)

I also need you guys to verify that with the Postfix master running
as PID=1, "docker stop" will no longer leave the master daemon
running until Docker times out and forcibly terminates everything.

By default, "docker stop" should send signal SIGTERM (signal 15) which
is what Postfix expects, but it is good to verify.

        Wietse
123