Double Mails delivered with aliases.

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Double Mails delivered with aliases.

Søren Peter Skou

Hiya,

 

I’m having sort of a strange problem with my postfix installation.

 

I have Local users and users with forwards. Some can have more than one forward, say we have Original-Recipient [hidden email] that forwards to local [hidden email] and remote [hidden email] . At first, my remote user did not get the mail at all. I suspected the remote end, but some debugging showed that it was the Postfix that simply did not deliver to remote destinations when forwarding. Having thought about this for some time I ended up with a solution that at least managed to ship mails to the remote user. But now the local user gets 2 mails, one that is clearly destined for [hidden email], but also one that is destined for [hidden email].

 

Postfix receives the mail, handles greylisting, quota, spf, dkim and other lovelies. Then sends it off via “content_filter=scan:127.0.0.1:10025” to clamsmtp. This returns on port 10026 with this listener from master.cf

 

127.0.0.1:10026 inet    n       -       -       -       -       smtpd

        -o spamassassin_destination_recipient_limit=1

        -o content_filter=spamassassin

        -o receive_override_options=no_header_body_checks,no_unknown_recipient_checks

        -o smtpd_helo_restrictions=

        -o smtpd_client_restrictions=

        -o smtpd_sender_restrictions=

        -o smtpd_recipient_restrictions=permit_mynetworks,reject

        -o mynetworks_style=host

        -o default_transport=smtp:

        -o smtpd_authorized_xforward_hosts=127.0.0.0/8

 

 

Here it then sends it further into the content_filter=spamassassin

 

spamassassin unix - n n - - pipe

    flags=DROXhu user=vmail:vmail argv=/usr/local/bin/xMTPDeliver ${sender} ${user} ${nexthop}

 

xMTPDeliver examines ${user} and ${nexthop} to see where it is supposed to go local/remote. If it is local, we use dovecot-lda to deliver it directly into users spam-folder if marked as spam, otherwise into users inbox. Fairly straight forward.

 

 

#!/bin/bash

 

# Author: [hidden email] - 20170507

# Description:

# Check if USER@DOMAIN is a local delivery.

# If it is - deliver through dovecot-LDA after scanning.

# Otherwise ship via SMTP (probably localhost..)

 

SMTP="/usr/sbin/sendmail -oi"

LMTP="/usr/lib/dovecot/deliver"

SPAM="/usr/bin/spamc"

LOG="/usr/bin/logger -p mail.info -t filter"

 

## Final Destination delivery.

SENDER=$1

USER=$2

DOMAIN=$3

 

# Note: aliases/forwards should be resolved at this point -

# otherwise they will be put in queue again.

# If 0 - then

IS_LOCAL=`/usr/sbin/postmap -q ${USER}@${DOMAIN} mysql:/etc/postfix/mysql-virtual-mailbox-maps.cf | wc -l`

 

## This local. Send to spamc + dovecot-LDA

if [ ${IS_LOCAL} -gt 0 ] ; then

        ${LOG} "Delivering to local user (${USER}@${DOMAIN})"

        ${SPAM} -f -e ${LMTP} -f ${SENDER} -d ${USER}@${DOMAIN}

else

## External domain - deliver through SMTP

        ${LOG} "Delivering to remote user (${USER}@${DOMAIN})"

        ${SPAM} -f -e ${SMTP} -f ${SENDER} -t ${USER}@${DOMAIN}

 

 

But somehow, somewhere – [hidden email] gets 2 mails, second mail [hidden email] does not even mention [hidden email] – but through and through looks like a mail to [hidden email] .. The debug shows that 2 mails are being generated as expected, one goes through the IS_LOCAL statement, whereas the other goes through the “Deliver through SMTP” and I suspect it is here something goes awry, that it sees “Oi, this one is really for [hidden email] – which is also aliased to [hidden email]”. But I cannot see where to fix that particular problem.

 

Any takers (or suggestions for changes)

 

Best regards

Søren P. Skou

Reply | Threaded
Open this post in threaded view
|

Re: Double Mails delivered with aliases.

Phil Stracchino
On 08/17/17 09:27, Søren Peter Skou wrote:

> Hiya,
>
> I’m having sort of a strange problem with my postfix installation.
>
> I have Local users and users with forwards. Some can have more than one
> forward, say we have Original-Recipient [hidden email] <mailto:[hidden email]> that
> forwards to local [hidden email] <mailto:[hidden email]> and remote [hidden email]
> <mailto:[hidden email]> . At first, my remote user did not get the mail at
> all. I suspected the remote end, but some debugging showed that it was
> the Postfix that simply did not deliver to remote destinations when
> forwarding. Having thought about this for some time I ended up with a
> solution that at least managed to ship mails to the remote user. But now
> the local user gets 2 mails, one that is clearly destined for [hidden email]
> <mailto:[hidden email]>, but also one that is destined for [hidden email]
> <mailto:[hidden email]>.


Short version:
If you have a delivery address aliasing scheme set up that results in
mail being delivered to duplicate email addresses for the same user, it
is not Postfix's responsibility to de-duplicate those emails.  That's on
you.


--
  Phil Stracchino
  Babylon Communications
  [hidden email]
  [hidden email]
  Landline: +1.603.293.8485
  Mobile:   +1.603.998.6958
Reply | Threaded
Open this post in threaded view
|

SV: Double Mails delivered with aliases.

Søren Peter Skou

> Short version:
> If you have a delivery address aliasing scheme set up that results in
> mail being delivered to duplicate email addresses for the same user, it
> is not Postfix's responsibility to de-duplicate those emails.  That's on
> you.
Yeah, I'm aware of this. Also, So far it is doing exactly as expected - Postfix starts 2 different processes to handle this. One that handles the local delivery for [hidden email], and another for the [hidden email]  ... Oh, I'm looking the wrong place, it is not Postfix per se that does things wrong, it is the "remote delivery" of the xMTPSender that handles things "wrong" (seen from my point of view).

I'll dig a bit in the run of that particular one to see how I can solve that.

Thank you,
Søren P. Skou
Reply | Threaded
Open this post in threaded view
|

Re: Double Mails delivered with aliases.

Matus UHLAR - fantomas
In reply to this post by Søren Peter Skou
On 17.08.17 13:27, Søren Peter Skou wrote:
>I have Local users and users with forwards.

how do they forward?

[...]
>Postfix receives the mail, handles greylisting, quota, spf, dkim and other
> lovelies.  Then sends it off via "content_filter=scan:127.0.0.1:10025" to
> clamsmtp.  This returns on port 10026 with this listener from master.cf
[...]
>Here it then sends it further into the content_filter=spamassassin
>
>spamassassin unix - n n - - pipe
>    flags=DROXhu user=vmail:vmail argv=/usr/local/bin/xMTPDeliver ${sender} ${user} ${nexthop}
>
>xMTPDeliver examines ${user} and ${nexthop} to see where it is supposed to
> go local/remote.  If it is local, we use dovecot-lda to deliver it
> directly into users spam-folder if marked as spam, otherwise into users
> inbox.  Fairly straight forward.

why do you use this xMTPDeliver at all?
--
Matus UHLAR - fantomas, [hidden email] ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
    One OS to rule them all, One OS to find them,
One OS to bring them all and into darkness bind them
Reply | Threaded
Open this post in threaded view
|

SV: Double Mails delivered with aliases.

Søren Peter Skou
(and now to the list instead of directly, apologies 'bout that)

> >I have Local users and users with forwards.
> how do they forward?
Using virtual_alias_maps - Lookup into a database, that returns the field alias as destination (alias due to a backend expecting that particular fieldname). If there is more than one forward, it is commaseparated.

The SQL itself is fairly straightforward :
  select alias as destination from MAILAUTH where username = '%u' and domain = '%d' and is_alias = 'yes' limit 1
for the [hidden email] it would be

select alias as destination from MAILAUTH where username = 'a' and domain = 'a.test' and is_alias = 'yes' limit 1

+-------------------+
| destination       |
+-------------------+
| [hidden email],[hidden email] |
+-------------------+

This works as expected, Postfix sees this, and fires off one process for [hidden email] and one for [hidden email]. And everything is then handed off to xMTPDeliver

>
> why do you use this xMTPDeliver at all?
At first, I didn't have that. I delivered directly to users mailboxes. The issue started with people having aliases/forwards that are not local. This means that the dovecot-lda did not really do anything for the remotes and they ended up in "Nope" land basically. Hence my test for "IS_LOCAL" - This works, the remote gets one as expected, but during the "remote" part, the local part all of a sudden gets one more. I'm now investigating the sendmail part of the xMTPDeliver to get it to do what I want. But if you have a better suggestion that ensures local deliver ends up in either INBOX or SPAM, and that remote gets theirs as well, I'm all ears 😊

Best regards,
Søren P. Skou
Reply | Threaded
Open this post in threaded view
|

Re: SV: Double Mails delivered with aliases.

Matus UHLAR - fantomas
On 17.08.17 14:22, Søren Peter Skou wrote:
>> >I have Local users and users with forwards.
>> how do they forward?
>Using virtual_alias_maps - Lookup into a database, that returns the field
> alias as destination (alias due to a backend expecting that particular
> fieldname).  If there is more than one forward, it is commaseparated.

>This works as expected, Postfix sees this, and fires off one process for
> [hidden email] and one for [hidden email].  And everything is then handed off to
> xMTPDeliver

you define content_filter named "spamassassin", which in fact calls
xMTPDeliver instead of spamassassin. That's pretty confusing.

You haven't posted whole master nor whole xMTPDeliver (pastebin, please).
Both can contain something that causes multiple deliveries.

>> why do you use this xMTPDeliver at all?
>At first, I didn't have that. I delivered directly to users mailboxes. The
> issue started with people having aliases/forwards that are not local.
> This means that the dovecot-lda did not really do anything for the remotes
> and they ended up in "Nope" land basically.  Hence my test for "IS_LOCAL"
> - This works, the remote gets one as expected, but during the "remote"
> part, the local part all of a sudden gets one more.  I'm now investigating
> the sendmail part of the xMTPDeliver to get it to do what I want.  But if
> you have a better suggestion that ensures local deliver ends up in either
> INBOX or SPAM, and that remote gets theirs as well, I'm all ears 😊

I believe you should only send local mail to dovecot (not all via
content_filter), you seem to pass everything to is which apparently caused
your problem.

I recommend you to reconfigure your mail flow and not use xMTPDeliver nor
dovecot_lda as content filter.

--
Matus UHLAR - fantomas, [hidden email] ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
The 3 biggets disasters: Hiroshima 45, Tschernobyl 86, Windows 95
Reply | Threaded
Open this post in threaded view
|

SV: SV: Double Mails delivered with aliases.

Søren Peter Skou
Realized that my usage of "local" for my users, is wrong terminology, they are all virtual users.

[...]
> You haven't posted whole master nor whole xMTPDeliver (pastebin, please).
> Both can contain something that causes multiple deliveries.
Pastebin coming up 😊
main.cf : https://pastebin.com/rQTFc50q
master.cf : https://pastebin.com/zfpG2sBB
xMTPDeliver: https://pastebin.com/GrUPuDgJ

I've come to the conclusion that the culprit lies in the "Not local" part of xMTPDeliver. Once it ships is off using sendmail (postfix) -oi, it is re-entered back into the queue, and Postfix sees the address once more, and delivers locally, since the [hidden email] resolves to both [hidden email] and [hidden email].
 
[....]
> I believe you should only send local mail to dovecot (not all via
> content_filter), you seem to pass everything to is which apparently caused
> your problem.
All incoming mail needs to be scanned for Vira/Malware, is why I pass it to a content_filter, 127.0.0.1:10025 has clamsmtp running. Clamsmtp returns on port 10026 if clean. In master.cf 10026 is then defined with a content_filter=spamassassin. This calls xMTPDeliver which handles the spamc calls. From there it depends if it is locally delivered, then deliver using DovecotLDA to deliver it into the Users INBOX or SPAM folder depending on the results from spamc. If it is remote, use sendmail -oi (which in fact is postfix..). The latter is where the culprit arises.



Reply | Threaded
Open this post in threaded view
|

Re: SV: SV: Double Mails delivered with aliases.

Matus UHLAR - fantomas
>> You haven't posted whole master nor whole xMTPDeliver (pastebin, please).
>> Both can contain something that causes multiple deliveries.

On 18.08.17 09:38, Søren Peter Skou wrote:

>Pastebin coming up 😊
>main.cf : https://pastebin.com/rQTFc50q
>master.cf : https://pastebin.com/zfpG2sBB
>xMTPDeliver: https://pastebin.com/GrUPuDgJ
>
>I've come to the conclusion that the culprit lies in the "Not local" part
> of xMTPDeliver.  Once it ships is off using sendmail (postfix) -oi, it is
> re-entered back into the queue, and Postfix sees the address once more,
> and delivers locally, since the [hidden email] resolves to both [hidden email] and
> [hidden email].
>
>[....]
>> I believe you should only send local mail to dovecot (not all via
>> content_filter), you seem to pass everything to is which apparently caused
>> your problem.

> All incoming mail needs to be scanned for Vira/Malware, is why I pass it
> to a content_filter, 127.0.0.1:10025 has clamsmtp running.  Clamsmtp
> returns on port 10026 if clean.In master.cf 10026 is then defined with a
> content_filter=spamassassin.  This calls xMTPDeliver which handles the
> spamc calls.

I see it simple: stop sing content_filter for mail delivery. It is not
designed to do that.
configure the content_filter to push mail back to queue and use postfix to
decide when to forward and when to deliver the mail locally.

--
Matus UHLAR - fantomas, [hidden email] ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
"They say when you play that M$ CD backward you can hear satanic messages."
"That's nothing. If you play it forward it will install Windows."