Sometimes loosing mails when using Procmail as content filter

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Sometimes loosing mails when using Procmail as content filter

Michael Neurohr
Hi,
I'm using Procmail to filter incoming messages with Spamassassin and ClamAV.
Today I recognized that mails are getting lost when a single mail
contains multiple recipients that are hosted on this server. I've no
clue what I'm doing wrong. Filtering the mails if only one of the
recipients is hosted on this server works fine.

I'm using Postfix version 2.6.6.

It looks like the mail is lost directly during the first Procmail rule.
I already tried to define a lock for the first rule, but it does not help.

The first mail (out of multiple recipients in that single mail) arrives
correctly at its recipient, all other are getting lost.

Can someone tell me what I'm doing wrong?

Thanks,
Michael

The corresponding log lines are:

maillog
=====================================================================
Jun 11 08:52:30 mx0 postfix/smtpd[31969]: connect from
testrelay.tu-graz.ac.at[129.27.2.201]
Jun 11 08:52:30 mx0 postfix/smtpd[31969]: 8581482404B8:
client=testrelay.tu-graz.ac.at[129.27.2.201]
Jun 11 08:52:30 mx0 postfix/cleanup[31977]: 8581482404B8:
message-id=<[hidden email]>
Jun 11 08:52:30 mx0 postfix/qmgr[31967]: 8581482404B8:
from=<[hidden email]>, size=2241, nrcpt=4 (queue active)
Jun 11 08:52:30 mx0 postfix/smtpd[31969]: disconnect from
testrelay.tu-graz.ac.at[129.27.2.201]
Jun 11 08:52:35 mx0 postfix/pickup[31968]: D3AC582437EE: uid=5000
from=<[hidden email]>
Jun 11 08:52:35 mx0 postfix/pipe[31978]: 8581482404B8:
to=<[hidden email]>, relay=procmail, delay=5.4,
delays=0.13/0/0/5.3, dsn=2.0.0, status=sent (delivered via procmail service)
Jun 11 08:52:35 mx0 postfix/pipe[31978]: 8581482404B8:
to=<[hidden email]>, relay=procmail, delay=5.4,
delays=0.13/0/0/5.3, dsn=2.0.0, status=sent (delivered via procmail service)
Jun 11 08:52:35 mx0 postfix/pipe[31978]: 8581482404B8:
to=<[hidden email]>, relay=procmail, delay=5.4,
delays=0.13/0/0/5.3, dsn=2.0.0, status=sent (delivered via procmail service)
Jun 11 08:52:35 mx0 postfix/pipe[31978]: 8581482404B8:
to=<[hidden email]>, relay=procmail, delay=5.4,
delays=0.13/0/0/5.3, dsn=2.0.0, status=sent (delivered via procmail service)
Jun 11 08:52:35 mx0 postfix/qmgr[31967]: 8581482404B8: removed
Jun 11 08:52:35 mx0 postfix/cleanup[31977]: D3AC582437EE:
message-id=<[hidden email]>
Jun 11 08:52:35 mx0 postfix/qmgr[31967]: D3AC582437EE:
from=<[hidden email]>, size=2659, nrcpt=1 (queue active)
Jun 11 08:52:35 mx0 dovecot: lmtp(32093): Connect from local
Jun 11 08:52:35 mx0 dovecot: lmtp(32093, [hidden email]):
HyvbNLP8l1NdfQAAtCQUIQ: sieve: msgid=<[hidden email]>:
stored mail into mailbox 'INBOX'
Jun 11 08:52:35 mx0 postfix/lmtp[32003]: D3AC582437EE:
to=<[hidden email]>,
relay=mx0.filestore.su[private/dovecot-lmtp], delay=0.13,
delays=0.02/0/0.01/0.1, dsn=2.0.0, status=sent (250 2.0.0
<[hidden email]> HyvbNLP8l1NdfQAAtCQUIQ Saved)
Jun 11 08:52:35 mx0 dovecot: lmtp(32093): Disconnect from local:
Successful quit
Jun 11 08:52:35 mx0 postfix/qmgr[31967]: D3AC582437EE: removed
=====================================================================

procmail.log
=====================================================================
procmail: [32072] Wed Jun 11 08:52:30 2014
procmail: Locking "/var/mail/vhosts/backup-mail/[hidden email]/.lock"
procmail: Assigning
"LASTFOLDER=/var/mail/vhosts/backup-mail/[hidden email]/new/1402469550.32072_1.mx0.ovz"
procmail: Unlocking
"/var/mail/vhosts/backup-mail/[hidden email]/.lock"
procmail: Executing "/usr/local/bin/clamassassin"
procmail: Match on "^X-Virus-Status: No"
procmail: Executing "/usr/bin/spamassassin"
procmail: [32072] Wed Jun 11 08:52:35 2014
procmail: Executing
"/usr/sbin/sendmail,-f,[hidden email],[hidden email]"
procmail: [32072] Wed Jun 11 08:52:35 2014
procmail: Assigning "LASTFOLDER=/usr/sbin/sendmail -f
[hidden email] [hidden email]"
 Subject: Re: test
  Folder: /usr/sbin/sendmail -f [hidden email] vero   2532
=====================================================================

My procmail script is:
=====================================================================
SHELL=/bin/sh
SENDMAILFLAGS="-f $E_SENDER $E_RECIPIENT"
LOGFILE="/var/log/procmail"
VERBOSE=on

# First make a backup of the mail
:0c:
/var/mail/vhosts/backup-mail/$E_RECIPIENT/

# Scan for viruses
:0fw
| /usr/local/bin/clamassassin

# If no virus has been found, scan for Spam
:0fw
* ^X-Virus-Status: No
| /usr/bin/spamassassin

# Last action: inject back to Postfix
:0w
| /usr/sbin/sendmail $SENDMAILFLAGS
=====================================================================

It is connected to Postfix as follows:
=====================================================================
smtp      inet  n       -       n       -       -       smtpd
      -o content_filter=procmail:dummy
      -o receive_override_options=no_address_mappings
      -o smtpd_recipient_restrictions=reject_unauth_destination
=====================================================================
=====================================================================
procmail  unix  -       n       n       -       10       pipe
  flags=Rq user=vmail null_sender= argv=/usr/bin/procmail -m
E_SENDER=$sender E_RECIPIENT=$recipient /etc/procmailrc
=====================================================================

postconf -n produces:
=====================================================================
alias_database = hash:/etc/aliases
alias_maps = hash:/etc/aliases
append_dot_mydomain = no
biff = no
command_directory = /usr/sbin
config_directory = /etc/postfix
daemon_directory = /usr/libexec/postfix
data_directory = /var/lib/postfix
html_directory = no
inet_interfaces = all
inet_protocols = ipv4, ipv6
mail_owner = postfix
mailbox_size_limit = 0
mailq_path = /usr/bin/mailq
manpage_directory = /usr/share/man
message_size_limit = 26214400
mydestination = localhost, $myhostname
myhostname = mx0.filestore.su
mynetworks = 127.0.0.0/8 [::1]/128 [fe80::]/10
myorigin = $mydomain
newaliases_path = /usr/bin/newaliases
queue_directory = /var/spool/postfix
readme_directory = /usr/share/doc/postfix-2.6.6/README_FILES
recipient_delimiter = +
relayhost =
sample_directory = /usr/share/doc/postfix-2.6.6/samples
sendmail_path = /usr/sbin/sendmail
setgid_group = postdrop
smtp_bind_address6 = 2a04:f81:783f:783f:43e1:787a:4be1:29a8
smtpd_banner = $myhostname ESMTP $mail_name (Debian/GNU)
smtpd_client_restrictions = reject_rbl_client ix.dnsbl.manitu.net
smtpd_recipient_restrictions = permit_sasl_authenticated,
reject_non_fqdn_sender, reject_non_fqdn_recipient,
reject_unknown_sender_domain, reject_unknown_recipient_domain,
permit_mynetworks,        reject_unauth_destination, permit
smtpd_sasl_auth_enable = yes
smtpd_sasl_path = private/auth
smtpd_sasl_type = dovecot
smtpd_tls_CAfile = /var/mail/ssl/ca-bundle.crt
smtpd_tls_auth_only = yes
smtpd_tls_cert_file = /var/mail/ssl/mx0.filestore.su.pem
smtpd_tls_key_file = /var/mail/ssl/mx0.filestore.su.key
smtpd_use_tls = yes
virtual_alias_maps = mysql:/etc/postfix/mysql-virtual-alias-maps.cf
virtual_mailbox_domains =
mysql:/etc/postfix/mysql-virtual-mailbox-domains.cf
virtual_mailbox_maps = mysql:/etc/postfix/mysql-virtual-mailbox-maps.cf
virtual_transport = lmtp:unix:private/dovecot-lmtp
=====================================================================
Reply | Threaded
Open this post in threaded view
|

Re: Sometimes losing mails when using Procmail as content filter

D'Arcy Cain
On Wed, 11 Jun 2014 09:37:44 +0200
Michael Neurohr <[hidden email]> wrote:
> I'm using Procmail to filter incoming messages with Spamassassin and
> ClamAV. Today I recognized that mails are getting lost when a single
> mail contains multiple recipients that are hosted on this server.
> I've no clue what I'm doing wrong. Filtering the mails if only one of
> the recipients is hosted on this server works fine.

You post a lot of information but I don't see /var/log/procmail.
Perhaps it has a clue.

I also use procmail to run spamassassin but I do it from individual
user .procmailrc files (generated from their preferences) and I wonder
if that has something to do with it.  What about SA and CAV system
configs?  Could there be something in there that uses a common file to
check for dups?  I have this in my .procmailrc files:

:0 Wh: msgid.lock
| formail -D 65536 $HOME/.msgid.cache

If something similar is happening with those programs but with a common
database (e.g. /var/db/msgid.cache) that might have the effect that you
are seeing.

Just spitballing here.  The procmail log might shed more light.

--
D'Arcy J.M. Cain
System Administrator, Vex.Net
http://www.Vex.Net/ IM:[hidden email]
VoIP: sip:[hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Sometimes loosing mails when using Procmail as content filter

Wietse Venema
In reply to this post by Michael Neurohr
Michael Neurohr:
> Hi,
> I'm using Procmail to filter incoming messages with Spamassassin and ClamAV.
> Today I recognized that mails are getting lost when a single mail
> contains multiple recipients that are hosted on this server. I've no
> clue what I'm doing wrong. Filtering the mails if only one of the
> recipients is hosted on this server works fine.

Configure Postfix to make single-recipient deliveries.

Set "procmail_destination_recipient_limit = 1" which is required
just like it is required for local(8) deliveries.

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: Sometimes loosing mails when using Procmail as content filter

Michael Neurohr
On 11.06.2014 14:17, Wietse Venema wrote:
> Configure Postfix to make single-recipient deliveries.
>
> Set "procmail_destination_recipient_limit = 1" which is required
> just like it is required for local(8) deliveries.

Yes, that did it. Many thanks!
Is that setting needed for all filters that are used like unix pipes?

Michael
Reply | Threaded
Open this post in threaded view
|

Re: Sometimes losing mails when using Procmail as content filter

Michael Neurohr
In reply to this post by D'Arcy Cain
On 11.06.2014 13:02, D'Arcy J.M. Cain wrote:
> You post a lot of information but I don't see /var/log/procmail.
> Perhaps it has a clue.

I wrote a wrong info before the procmail log. All I'm seeing in
/var/log/procmail is:

=====================================================================
procmail: [32072] Wed Jun 11 08:52:30 2014
procmail: Locking "/var/mail/vhosts/backup-mail/[hidden email]/.lock"
procmail: Assigning
"LASTFOLDER=/var/mail/vhosts/backup-mail/[hidden email]/new/1402469550.32072_1.mx0.ovz"
procmail: Unlocking
"/var/mail/vhosts/backup-mail/[hidden email]/.lock"
procmail: Executing "/usr/local/bin/clamassassin"
procmail: Match on "^X-Virus-Status: No"
procmail: Executing "/usr/bin/spamassassin"
procmail: [32072] Wed Jun 11 08:52:35 2014
procmail: Executing
"/usr/sbin/sendmail,-f,[hidden email],[hidden email]"
procmail: [32072] Wed Jun 11 08:52:35 2014
procmail: Assigning "LASTFOLDER=/usr/sbin/sendmail -f
[hidden email] [hidden email]"
 Subject: Re: test
  Folder: /usr/sbin/sendmail -f [hidden email] vero   2532
=====================================================================

The processing of the other users do not produce any log entries. That
fact is really wired...


But as Wietse wrote, the problem was the multiple recipient deliveries.
The problem is solved now :-)

Michael
Reply | Threaded
Open this post in threaded view
|

Re: Sometimes loosing mails when using Procmail as content filter

Noel Jones-2
In reply to this post by Michael Neurohr
On 6/11/2014 9:10 AM, Michael Neurohr wrote:

> On 11.06.2014 14:17, Wietse Venema wrote:
>> Configure Postfix to make single-recipient deliveries.
>>
>> Set "procmail_destination_recipient_limit = 1" which is required
>> just like it is required for local(8) deliveries.
>
> Yes, that did it. Many thanks!
> Is that setting needed for all filters that are used like unix pipes?
>
> Michael
>

That setting is needed for any transport or filter that expects only
one recipient at a time.

Very generally, this means final delivery, but there are exceptions.



  -- Noel Jones