Local delivery to mbox / inode issue

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Local delivery to mbox / inode issue

Dominic Raferd
I am using incrond to monitor an mbox file (in /var/mail) for changes, but it is failing to trigger when postfix adds an incoming mail to the file. (It triggers fine however if I touch the file.) 

I may be barking up the wrong tree but I wonder if this is because instead of merely appending to the existing mbox file, postfix/local rewrites the file so that its inode changes (which I know breaks incrond's ability to monitor a file). If so, is there a way to get postfix to append to the file without changing the inode, and if so what are the disadvantages?

I have postfix 3.3.0, local has default settings, and /var/mail is on filesystem ext4 (options rw,relatime,data=ordered) under Ubuntu 18.04.1 (GNU/Linux 4.15). 
Reply | Threaded
Open this post in threaded view
|

Re: Local delivery to mbox / inode issue

Wietse Venema
Dominic Raferd:
> I am using incrond to monitor an mbox file (in /var/mail) for changes, but
> it is failing to trigger when postfix adds an incoming mail to the file.

Possible causes:

- Your file system does not set the file mtime when Postfix appends
to the file. Fix: don't disable mtime updates for append operations.

- There is a clock drift problem where the host with Postfix has
different time than the host with the file system. Fix: run NTP on
both hosts.

> (It triggers fine however if I touch the file.)
>
> I may be barking up the wrong tree but I wonder if this is because instead
> of merely appending to the existing mbox file, postfix/local rewrites the
> file so that its inode changes (which I know breaks incrond's ability to
> monitor a file). If so, is there a way to get postfix to append to the file
> without changing the inode, and if so what are the disadvantages?

Have you verified that the inode number changes?

Postfix opens an mbox file with O_APPEND mode which is standardized
by the POSIX API. If that causes the file inode number to change,
then you need to use a different file system.

        Wietse

> I have postfix 3.3.0, local has default settings, and /var/mail is on
> filesystem ext4 (options rw,relatime,data=ordered) under Ubuntu 18.04.1
> (GNU/Linux 4.15).
Reply | Threaded
Open this post in threaded view
|

Re: Local delivery to mbox / inode issue

Dominic Raferd
Thanks for the swift response - see below.

On Thu, 6 Dec 2018 at 16:10, Wietse Venema <[hidden email]> wrote:
Dominic Raferd:
> I am using incrond to monitor an mbox file (in /var/mail) for changes, but
> it is failing to trigger when postfix adds an incoming mail to the file.

Possible causes:

- Your file system does not set the file mtime when Postfix appends
to the file. Fix: don't disable mtime updates for append operations.

- it does set the mtime when Postfix appends to the file 

- There is a clock drift problem where the host with Postfix has
different time than the host with the file system. Fix: run NTP on
both hosts.

- Postfix is on the same host as the filesystem
> (It triggers fine however if I touch the file.)
>
> I may be barking up the wrong tree but I wonder if this is because instead
> of merely appending to the existing mbox file, postfix/local rewrites the
> file so that its inode changes (which I know breaks incrond's ability to
> monitor a file). If so, is there a way to get postfix to append to the file
> without changing the inode, and if so what are the disadvantages?

Have you verified that the inode number changes?

no, I will check how to do this

Postfix opens an mbox file with O_APPEND mode which is standardized
by the POSIX API. If that causes the file inode number to change,
then you need to use a different file system.

In this case I suspect that the inode issue is not the cause of incrond's problem.
Reply | Threaded
Open this post in threaded view
|

Re: Local delivery to mbox / inode issue

Bill Cole-3
On 6 Dec 2018, at 11:15, Dominic Raferd wrote:

>> Have you verified that the inode number changes?
>>
>
>
> no, I will check how to do this


'ls -li' is your friend.
Reply | Threaded
Open this post in threaded view
|

Re: Local delivery to mbox / inode issue

Dominic Raferd


On Thu, 6 Dec 2018 at 16:37, Bill Cole <[hidden email]> wrote:
On 6 Dec 2018, at 11:15, Dominic Raferd wrote:

>> Have you verified that the inode number changes?
>>
>
>
> no, I will check how to do this


'ls -li' is your friend.

Thanks, I have now used this to confirm that the inode number does not change when postfix/local updates the mbox file. So the problem with incrond lies elsewhere.
Reply | Threaded
Open this post in threaded view
|

Re: Local delivery to mbox / inode issue

Bill Cole-3
On 6 Dec 2018, at 12:13, Dominic Raferd wrote:

> On Thu, 6 Dec 2018 at 16:37, Bill Cole <
> [hidden email]> wrote:
>
>> On 6 Dec 2018, at 11:15, Dominic Raferd wrote:
>>
>>>> Have you verified that the inode number changes?
>>>>
>>>
>>>
>>> no, I will check how to do this
>>
>>
>> 'ls -li' is your friend.
>>
>
> Thanks, I have now used this to confirm that the inode number does not
> change when postfix/local updates the mbox file. So the problem with
> incrond lies elsewhere.

One thing to check is the sysctl parameter 'fs.inotify.max_user_watches'
(which is what its name implies.)  Some (EL-family, for example)
distributions default to a low value (8K) which is a very easy limit to
hit with a tool like incron that is designed to exploit inotify to its
fullest.


--
Bill Cole
[hidden email] or [hidden email]
(AKA @grumpybozo and many *@billmail.scconsult.com addresses)
Available For Hire: https://linkedin.com/in/billcole
Reply | Threaded
Open this post in threaded view
|

Re: Local delivery to mbox / inode issue

Matus UHLAR - fantomas
In reply to this post by Dominic Raferd
On 06.12.18 15:45, Dominic Raferd wrote:
>I am using incrond to monitor an mbox file (in /var/mail) for changes,

hmmm, why?
maybe there's other way to implement your requirement
--
Matus UHLAR - fantomas, [hidden email] ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Saving Private Ryan...
Private Ryan exists. Overwrite? (Y/N)
Reply | Threaded
Open this post in threaded view
|

Re: Local delivery to mbox / inode issue

Dominic Raferd
On Fri, 7 Dec 2018 at 09:15, Matus UHLAR - fantomas <[hidden email]> wrote:
On 06.12.18 15:45, Dominic Raferd wrote:
>I am using incrond to monitor an mbox file (in /var/mail) for changes,

hmmm, why?
maybe there's other way to implement your requirement

I think I have it working now. The problem was not postfix (of course), it was that I was modifying the mbox file using 
postlock $MBOXFILE sed -if $SCRIPTFILE $MBOXFILE 
which in fact replaces it, I am now using:
postlock $MBOXFILE /bin/bash -c "sed -f $SCRIPTFILE $MBOXFILE|sponge $MBOXFILE" # not sure if bash subshell is necessary?
and also comparing the inode before and after - if the inode changes (which is not normally the case) then I restart incrond.

The use case is my home-grown quarantining system. Mail server periodically emails me about quarantined emails - I can reply to these emails using short codes, this reply is placed by postfix in the monitored $MBOXFILE. When $MBOXFILE changes, incrond triggers a script which looks at the latest email in it and interprets it to take the necessary actions i.e. removing unwanted mails from quarantine, releasing wanted ones, whitelisting sender etc.
Reply | Threaded
Open this post in threaded view
|

Re: Local delivery to mbox / inode issue

Matus UHLAR - fantomas
>> On 06.12.18 15:45, Dominic Raferd wrote:
>> >I am using incrond to monitor an mbox file (in /var/mail) for changes,

>On Fri, 7 Dec 2018 at 09:15, Matus UHLAR - fantomas <[hidden email]>
>wrote:
>> hmmm, why?
>> maybe there's other way to implement your requirement

On 07.12.18 10:22, Dominic Raferd wrote:
>The use case is my home-grown quarantining system. Mail server periodically
>emails me about quarantined emails - I can reply to these emails using
>short codes, this reply is placed by postfix in the monitored $MBOXFILE.
>When $MBOXFILE changes, incrond triggers a script which looks at the latest
>email in it and interprets it to take the necessary actions i.e. removing
>unwanted mails from quarantine, releasing wanted ones, whitelisting sender
>etc.

You can run script at the time of mail delivery by using .forward that pipes
mail to a script. You don't need to watch file for changes.

--
Matus UHLAR - fantomas, [hidden email] ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
"They say when you play that M$ CD backward you can hear satanic messages."
"That's nothing. If you play it forward it will install Windows."
Reply | Threaded
Open this post in threaded view
|

Re: Local delivery to mbox / inode issue

Dominic Raferd


On Fri, 7 Dec 2018 at 10:40, Matus UHLAR - fantomas <[hidden email]> wrote:
>> On 06.12.18 15:45, Dominic Raferd wrote:
>> >I am using incrond to monitor an mbox file (in /var/mail) for changes,

>On Fri, 7 Dec 2018 at 09:15, Matus UHLAR - fantomas <[hidden email]>
>wrote:
>> hmmm, why?
>> maybe there's other way to implement your requirement

On 07.12.18 10:22, Dominic Raferd wrote:
>The use case is my home-grown quarantining system. Mail server periodically
>emails me about quarantined emails - I can reply to these emails using
>short codes, this reply is placed by postfix in the monitored $MBOXFILE.
>When $MBOXFILE changes, incrond triggers a script which looks at the latest
>email in it and interprets it to take the necessary actions i.e. removing
>unwanted mails from quarantine, releasing wanted ones, whitelisting sender
>etc.

You can run script at the time of mail delivery by using .forward that pipes
mail to a script. You don't need to watch file for changes.

That is neat, but not sure in that situation how I give the script permissions to operate on the quarantined files (i.e. run amavisd-release), and update whitelist files.