postqueue -f delayed

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

postqueue -f delayed

PeterDaem
Hi...

flushing the queue with 'postqueue -f'' normally produces instant flush but sometimes it takes some time to do it... it always works! but sometimes with a long delay...

just out of curiosity... why does this happen? is it qmgr daemon waiting for anything? is there any way for force it?


Thanks...

Pete.
Reply | Threaded
Open this post in threaded view
|

Re: postqueue -f delayed

Wietse Venema
Pedro David Marco:
> Hi...
> flushing the queue with 'postqueue -f'' normally produces instant flush but sometimes it takes some time to do it... it always works! but sometimes with a long delay...
> just out of curiosity... why does this happen? is it qmgr daemon waiting for anything? is there any way for force it?
>

Insufficient data.
Reply | Threaded
Open this post in threaded view
|

Re: postqueue -f delayed

Ron Wheeler
In reply to this post by PeterDaem
You might want to take a look at what is in the queue.

Flushing the queue means communicating with other mail servers and the reason that mail is in the queue is that it was "too hard" to deliver it the first time.
A broken or overloaded remote could still be slow.

Ron

On 2020-10-26 6:07 a.m., Pedro David Marco wrote:
Hi...

flushing the queue with 'postqueue -f'' normally produces instant flush but sometimes it takes some time to do it... it always works! but sometimes with a long delay...

just out of curiosity... why does this happen? is it qmgr daemon waiting for anything? is there any way for force it?


Thanks...

Pete.

-- 
Ron Wheeler
Artifact Software
438-345-3369
[hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: postqueue -f delayed

PeterDaem


>On Monday, October 26, 2020, 05:09:41 PM GMT+1, Ron Wheeler <[hidden email]> wrote:

>You might want to take a look at what is in the queue.

>Flushing the queue means communicating with other mail servers and the reason that mail is in the queue is that it was "too hard" to deliver it the first time.
>A broken or overloaded remote could still be slow.

>Ron


Thanks Ron,

Just a real example: 100 emails in deferred queue due to connection timed out (remote host was down for a while). Once the remote is up again, i run postqueue -f  for quick
delivery...
Sometimes it works inmediatelly, and sometimes there is a delay... (with no postfix log activity at all)



Thanks Wietse.. please, what extra data may be needed? is the previous example enough?

Thanks!


----
Pete.
Reply | Threaded
Open this post in threaded view
|

Re: postqueue -f delayed

Ron Wheeler
Could be just that the other end was busy receiving someone else's mail. Takes 2 to tango!
No big attachments?


On 2020-10-26 12:22 p.m., Pedro David Marco wrote:


>On Monday, October 26, 2020, 05:09:41 PM GMT+1, Ron Wheeler [hidden email] wrote:

>You might want to take a look at what is in the queue.

>Flushing the queue means communicating with other mail servers and the reason that mail is in the queue is that it was "too hard" to deliver it the first time.
>A broken or overloaded remote could still be slow.

>Ron


Thanks Ron,

Just a real example: 100 emails in deferred queue due to connection timed out (remote host was down for a while). Once the remote is up again, i run postqueue -f  for quick
delivery...
Sometimes it works inmediatelly, and sometimes there is a delay... (with no postfix log activity at all)



Thanks Wietse.. please, what extra data may be needed? is the previous example enough?

Thanks!


----
Pete.

-- 
Ron Wheeler
Artifact Software
438-345-3369
[hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: postqueue -f delayed

PeterDaem


>On Monday, October 26, 2020, 05:31:05 PM GMT+1, Ron Wheeler <[hidden email]> wrote:
>

>Could be just that the other end was busy receiving someone else's mail. Takes 2 to tango!
>No big attachments?



Thanks Ron...  size no bigger than 500KB... if remote is busy...  in the log at least i should see the postfix attempt, am i right? but there is nothing at all in the log...

----
Pete.
Reply | Threaded
Open this post in threaded view
|

Re: postqueue -f delayed

Ron Wheeler
I think that you should only see the attempt as a successful send. Are you logging successful sends?

I would not expect any error as long as the delay is not so long that Postfix decides that it is never going to go.
As long as the attempt succeeds within the timeout delay, Postfix considers it a success, does not complain and moves on to the next one.

I am not sure of the following:
- how many time Postfix retries before putting something in the queue?
- how often Postfix goes through the queue retrying old failed sends?
- what make Postfix give up retrying automatically?

If I were in your situation, I would be looking to see if there is anything that could be done to make Postfix deal with the root problem of stuff getting caught in the queue and not being delivered after the remote system resumes normal operation.
Having to do a manual flush is what makes the delay visible. If it went on its own, you would never know of the occasional delay.

If you are very old, you will remember when networking was young and e-mail was sent over dial-up connections that connected only once or twice a day.
The email system has to deal with the historical world where connections where not "always on" so a successful send does not imply anything about time.

Ron


On 2020-10-26 12:44 p.m., Pedro David Marco wrote:


>On Monday, October 26, 2020, 05:31:05 PM GMT+1, Ron Wheeler [hidden email] wrote:
>

>Could be just that the other end was busy receiving someone else's mail. Takes 2 to tango!
>No big attachments?



Thanks Ron...  size no bigger than 500KB... if remote is busy...  in the log at least i should see the postfix attempt, am i right? but there is nothing at all in the log...

----
Pete.

-- 
Ron Wheeler
Artifact Software
438-345-3369
[hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: postqueue -f delayed

Peter Blair
At 26 October, 2020 Ron Wheeler wrote:
 
> If you are very old, you will remember when networking was young and e-mail
> was sent over dial-up connections that connected only once or twice a day.
> The email system has to deal with the historical world where connections
> where not "always on" so a successful send does not imply anything about
> time.

All of the good tech started with "uu": uucp, uuencode, uunet :P
Reply | Threaded
Open this post in threaded view
|

Re: postqueue -f delayed

Ron Wheeler
I came through the ARPAnet-DECnet and 2780/3780 stream.

On 2020-10-26 1:49 p.m., Peter Blair wrote:
At 26 October, 2020 Ron Wheeler wrote:
 
If you are very old, you will remember when networking was young and e-mail
was sent over dial-up connections that connected only once or twice a day.
The email system has to deal with the historical world where connections
where not "always on" so a successful send does not imply anything about
time.
All of the good tech started with "uu": uucp, uuencode, uunet :P

-- 
Ron Wheeler
Artifact Software
438-345-3369
[hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: postqueue -f delayed

Noel Jones-2
In reply to this post by Ron Wheeler

On 10/26/2020 12:46 PM, Ron Wheeler wrote:

>
> I am not sure of the following:
> - how many time Postfix retries before putting something in the queue?
> - how often Postfix goes through the queue retrying old failed sends?
> - what make Postfix give up retrying automatically?
>


Documentation:
http://www.postfix.org/QSHAPE_README.html
http://www.postfix.org/TUNING_README.html

The default values for all the queue controls are carefully chosen,
and don't need to be changed except for unusual circumstances, such
as a bulk mail server or a server that only has intermittent
connection to the internet.
If you feel the need to twiddle the knobs, start with small changes
and test the effect.

As a general rule, don't flush the queue; let postfix deal with
deferred mail on its own. Frequent flushes can reduce performance.

In your case, a modest change to maximal_backoff_time may help
delivery to destinations that are sometimes offline. Make a small
change and test. Note that reducing the backoff can delay new mail
if there is a lot of deferred mail not yet deliverable.
Don't flush the queue.


   -- Noel Jones
Reply | Threaded
Open this post in threaded view
|

Re: postqueue -f delayed

Viktor Dukhovni
In reply to this post by PeterDaem
On Mon, Oct 26, 2020 at 10:07:25AM +0000, Pedro David Marco wrote:

> Flushing the queue with 'postqueue -f' normally produces instant
> flush but sometimes it takes some time to do it... it always works!

It never produces "instant flush", what it does is reset the queue
manager's delay timer for the next deferred queue scan, so that if no
deferred queue scan is currently in progress, it starts now, or if
already in progress, the next scan will start as soon as the current one
completes.

Furthermore, instead of retrying just the messages whose retry time is
in the past, for the next scan (and any remaining portion of the current
scan) all messages will be retried.

> but sometimes with a long delay...  just out of curiosity... why does
> this happen? is it qmgr daemon waiting for anything? is there any way
> for force it?

As Wietse noted, without more information about what's in the queue at
the time, etc. it is hard to say.  One would expective to see "qmgr" log
messages showing mail entering the active queue, e.g.:

    Oct 26 14:43:56 amnesiac postfix/qmgr[97795]: E0BFA3BC92C:
        from=<[hidden email]>,
        size=11617, nrcpt=1 (queue active)

but perhaps your logging subsystem is losing messages.

On Mon, Oct 26, 2020 at 04:22:21PM +0000, Pedro David Marco wrote:

> >On Monday, October 26, 2020, 05:09:41 PM GMT+1, Ron Wheeler <[hidden email]> wrote:  
> >You might want to take a look at what is in the queue.
>  
> >Flushing the queue means communicating with other mail servers and
> >the reason that mail is in the queue is that it was "too hard" to
> >deliver it the first time.  A broken or overloaded remote could still
> >be slow.

Ron is not well informed on this topic, and is just guessing.

> Just a real example: 100 emails in deferred queue due to connection
> timed out (remote host was down for a while). Once the remote is up
> again, i run postqueue -f  for quickdelivery...Sometimes it works
> immediatelly, and sometimes there is a delay... (with no postfix log
> activity at all)

Your logging subsystem may be losing messages, are you seeing logging
from the queue manager at all?  With "postqueue -f", and deferred
messages in the queue, you should be seeing "qmgr" log messages about
new mail entering the queue, which would show up promptly, unless
you're using a particularly sluggish transport table (slow LDAP,
overloaded MySQL, ...).

On Mon, Oct 26, 2020 at 04:44:11PM +0000, Pedro David Marco wrote:

>  >On Monday, October 26, 2020, 05:31:05 PM GMT+1, Ron Wheeler <[hidden email]> wrote:  >
> >Could be just that the other end was busy receiving someone else's mail. Takes 2 to tango!
> >No big attachments?
>  
> Thanks Ron...  size no bigger than 500KB... if remote is busy...  in
> the log at least i should see the postfix attempt, am i right? but
> there is nothing at all in the log...

Message content size does not matter in this context.  Queue manager
latency depends on the number of recipients in a message (up to a limit
on the number brought into memory at once) not the message size.  With a
sufficiency low-latency transport table (none or indexed files) the
queue manager activates messages "in real time".

--
    Viktor.
Reply | Threaded
Open this post in threaded view
|

Re: postqueue -f delayed

Bill Cole-3
In reply to this post by PeterDaem
On 26 Oct 2020, at 6:07, Pedro David Marco wrote:

> Hi...
> flushing the queue with 'postqueue -f'' normally produces instant
> flush but sometimes it takes some time to do it... it always works!
> but sometimes with a long delay...

Can you be more specific about "long"?


--
Bill Cole
[hidden email] or [hidden email]
(AKA @grumpybozo and many *@billmail.scconsult.com addresses)
Not Currently Available For Hire
Reply | Threaded
Open this post in threaded view
|

Re: postqueue -f delayed

Wietse Venema
Bill Cole:
> On 26 Oct 2020, at 6:07, Pedro David Marco wrote:
>
> > Hi...
> > flushing the queue with 'postqueue -f'' normally produces instant
> > flush but sometimes it takes some time to do it... it always works!
> > but sometimes with a long delay...
>
> Can you be more specific about "long"?

And what Postfix activity you consider "delayed".  As other people
have noted, you won't see "activity" when all delivery agents are
already "busy" connecting to unreachable destinations.

That is exactly what happens when you keep hitting Postfix with
"postfix flush" commands.

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: postqueue -f delayed

Ron Wheeler
In reply to this post by Viktor Dukhovni
You got me!!! I have only been running corporate e-mail on Postfix for a couple of decades and still learning the basics.
It does not require a lot of expertise until something goes wrong!

I knew that you or Wietse or one of the other experts would correct my guesses.

You guys give great support on a product that is very complex but does damn near everything possible with mail.

Ron

On 2020-10-26 2:59 p.m., Viktor Dukhovni wrote:
On Mon, Oct 26, 2020 at 10:07:25AM +0000, Pedro David Marco wrote:

Flushing the queue with 'postqueue -f' normally produces instant
flush but sometimes it takes some time to do it... it always works!
It never produces "instant flush", what it does is reset the queue
manager's delay timer for the next deferred queue scan, so that if no
deferred queue scan is currently in progress, it starts now, or if
already in progress, the next scan will start as soon as the current one
completes.

Furthermore, instead of retrying just the messages whose retry time is
in the past, for the next scan (and any remaining portion of the current
scan) all messages will be retried.

but sometimes with a long delay...  just out of curiosity... why does
this happen? is it qmgr daemon waiting for anything? is there any way
for force it?
As Wietse noted, without more information about what's in the queue at
the time, etc. it is hard to say.  One would expective to see "qmgr" log
messages showing mail entering the active queue, e.g.:

    Oct 26 14:43:56 amnesiac postfix/qmgr[97795]: E0BFA3BC92C:
        from=[hidden email],
        size=11617, nrcpt=1 (queue active)

but perhaps your logging subsystem is losing messages.

On Mon, Oct 26, 2020 at 04:22:21PM +0000, Pedro David Marco wrote:

On Monday, October 26, 2020, 05:09:41 PM GMT+1, Ron Wheeler [hidden email] wrote:  
You might want to take a look at what is in the queue.
 
Flushing the queue means communicating with other mail servers and
the reason that mail is in the queue is that it was "too hard" to
deliver it the first time.  A broken or overloaded remote could still
be slow.
Ron is not well informed on this topic, and is just guessing.

Just a real example: 100 emails in deferred queue due to connection
timed out (remote host was down for a while). Once the remote is up
again, i run postqueue -f  for quickdelivery...Sometimes it works
immediatelly, and sometimes there is a delay... (with no postfix log
activity at all)
Your logging subsystem may be losing messages, are you seeing logging
from the queue manager at all?  With "postqueue -f", and deferred
messages in the queue, you should be seeing "qmgr" log messages about
new mail entering the queue, which would show up promptly, unless
you're using a particularly sluggish transport table (slow LDAP,
overloaded MySQL, ...).

On Mon, Oct 26, 2020 at 04:44:11PM +0000, Pedro David Marco wrote:

 >On Monday, October 26, 2020, 05:31:05 PM GMT+1, Ron Wheeler [hidden email] wrote:  > 
Could be just that the other end was busy receiving someone else's mail. Takes 2 to tango!
No big attachments?
 
Thanks Ron...  size no bigger than 500KB... if remote is busy...  in
the log at least i should see the postfix attempt, am i right? but
there is nothing at all in the log...
Message content size does not matter in this context.  Queue manager
latency depends on the number of recipients in a message (up to a limit
on the number brought into memory at once) not the message size.  With a
sufficiency low-latency transport table (none or indexed files) the
queue manager activates messages "in real time".


-- 
Ron Wheeler
Artifact Software
438-345-3369
[hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: postqueue -f delayed

PeterDaem
In reply to this post by Wietse Venema
Thanks, Ron, Wietse, and Viktor... i will put an eye on this, having in mind all your remarks...

----
Pete

On Monday, October 26, 2020, 10:46:51 PM GMT+1, Wietse Venema <[hidden email]> wrote:


Bill Cole:

> On 26 Oct 2020, at 6:07, Pedro David Marco wrote:
>
> > Hi...
> > flushing the queue with 'postqueue -f'' normally produces instant
> > flush but sometimes it takes some time to do it... it always works!
> > but sometimes with a long delay...
>
> Can you be more specific about "long"?


And what Postfix activity you consider "delayed".  As other people
have noted, you won't see "activity" when all delivery agents are
already "busy" connecting to unreachable destinations.

That is exactly what happens when you keep hitting Postfix with
"postfix flush" commands.

    Wietse