postfix + forwadgroup + external amavis with haproxy and no_address_mappings

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

postfix + forwadgroup + external amavis with haproxy and no_address_mappings

mami64
Hi
I have debian 9 and postfix 3.1.14. Generally, I have distributed mail
traffic over several machines

- separately for sent mail - here I have postfix
- separately for incoming e-mails - here I have postfix + external amavis


The general outline is this:

1) mail arrives at postfix
2) postfix transfers it to Amavis
    - it really is a local haproxy which directs to one of three amavis

3) mail returns from amavis on a given ip: port (which is filtered from
outside the firewall)
4) using LMTP to dovecot cluster and then to maildirs and then to sieve
      virtual_transport = lmtp: inet: 10.0.100.5: 24




Some my restryctions
smtpd_client_restrictions =
# local map with host and network wgo must go to amavis or without amavisa
        check_client_access cidr:/etc/postfix/amavis_bypass,
        reject_unauth_pipelining,
        permit

/etc/postfix/amavis_bypass

#without amavis
86.xxx.xxx.0/24 OK
89.xxx.xxx.0/24 Ok
10.0.100.21/32 OK
10.0.100.22/32 OK
10.0.100.23/32 OK
10.0.100.24/32 OK
10.0.100.25/32 OK
89.206.41.19/32 OK
#other go to amavis
0.0.0.0/0 FILTER smtp-amavis:[127.0.0.1]:10628



master.cf:
smtp-amavis     unix    -       -       -       -       80       smtp
        -o smtp_data_done_timeout=6000s
        -o smtp_send_xforward_command=yes
        -o disable_dns_lookups=yes

#80 cosnnections - and in my amavis I have 90 (10+overtime )


#returns from amavis  IP .199

86.xxx.xxx.199:10027 inet n  -   n   -   -      smtpd
    -o smtpd_proxy_timeout=900s
    -o content_filter=
    -o mynetworks_style=host
    -o mynetworks=10.0.100.0/24,86.xxx.xxx.199/32,
    -o local_recipient_maps=
    -o relay_recipient_maps=
    -o strict_rfc821_envelopes=yes
    -o smtp_tls_security_level=none
    -o smtpd_tls_security_level=none
    -o smtpd_restriction_classes=
    -o smtpd_delay_reject=no
    -o smtpd_client_restrictions=permit_mynetworks,reject
    -o smtpd_helo_restrictions=
    -o smtpd_sender_restrictions=
    -o smtpd_recipient_restrictions=permit_mynetworks,reject
    -o smtpd_end_of_data_restrictions=
    -o smtpd_error_sleep_time=0
    -o smtpd_soft_error_limit=1001
    -o smtpd_hard_error_limit=1000
    -o smtpd_client_connection_count_limit=0
    -o smtpd_client_connection_rate_limit=0
    -o
receive_override_options=no_header_body_checks,no_unknown_recipient_checks,no_address_mappings


All works fine but sometimes my "users" use a mial forwarding .... In
that forwarding have (100-200 email) like

[hidden email] ---> [hidden email], [hidden email],
[hidden email], [hidden email]

And all forward e-mail was "releback" in smtp and go to amavis.

In amavis I get:

Apr 16 15:11:11 amavis2 amavis[10499]: (10499-01) ESMTP
[86.xxx.xxx.155]:10628
/var/amavis/tmp/amavis-20200416T151111-10499-r3E5zU6i: <[hidden email]> ->
<[hidden email]>,<[hidden email]>,<[hidden email]>,<[hidden email]>,<[hidden email]>,<[hidden email]>
SIZE=2129 BODY=7BIT Received: from myserver.domainltd.pl
([86.xxx.xxx.199]) by localhost (amavis2.localdomain [86.xxx.xxx.155])
(amavisd-new, port 10628) with ESMTP; Thu, 16 Apr 2020 15:11:11 +0200 (CEST)


Apr 16 15:11:11 amavis2 amavis[10499]: (10499-01) spam_scan:
score=-0.198 autolearn=no autolearn_force=no
tests=[BAYES_00=-1.9,DCC_REPUT_70_89=0.1,HTML_IMAGE_RATIO_06=0.001,HTML_MESSAGE=0.001,IQ_EMAIL_KASA_2=0.5,RCVD_IN_DNSWL_NONE=-0.0001,SUBJ_ALL_CAPS=0.5,UNIVERSAL_HTMLv20160523_1=0.1,UNIVERSAL_HTMLv20160523_2=0.1,UNIVERSAL_HTMLv20160523_3=0.1,UNIVERSAL_HTMLv20160523_5=0.1,URIBL_BLOCKED=0.2]
recips=22
Apr 16 15:11:11 amavis2 amavis[10499]: (10499-01) spam_scan:
score=-0.198 autolearn=no autolearn_force=no
tests=[BAYES_00=-1.9,DCC_REPUT_70_89=0.1,HTML_IMAGE_RATIO_06=0.001,HTML_MESSAGE=0.001,IQ_EMAIL_KASA_2=0.5,RCVD_IN_DNSWL_NONE=-0.0001,SUBJ_ALL_CAPS=0.5,UNIVERSAL_HTMLv20160523_1=0.1,UNIVERSAL_HTMLv20160523_2=0.1,UNIVERSAL_HTMLv20160523_3=0.1,UNIVERSAL_HTMLv20160523_5=0.1,URIBL_BLOCKED=0.2]
recips=4
Apr 16 15:11:11 amavis2 amavis[10499]: (10499-01)  spam_scan:
score=-0.198 autolearn=no autolearn_force=no
tests=[BAYES_00=-1.9,DCC_REPUT_70_89=0.1,HTML_IMAGE_RATIO_06=0.001,HTML_MESSAGE=0.001,IQ_EMAIL_KASA_2=0.5,RCVD_IN_DNSWL_NONE=-0.0001,SUBJ_ALL_CAPS=0.5,UNIVERSAL_HTMLv20160523_1=0.1,UNIVERSAL_HTMLv20160523_2=0.1,UNIVERSAL_HTMLv20160523_3=0.1,UNIVERSAL_HTMLv20160523_5=0.1,URIBL_BLOCKED=0.2]
recips=82
Apr 16 15:11:11 amavis2 amavis[10499]: (10499-01) spam_scan:
score=-0.198 autolearn=no autolearn_force=no
tests=[BAYES_00=-1.9,DCC_REPUT_70_89=0.1,HTML_IMAGE_RATIO_06=0.001,HTML_MESSAGE=0.001,IQ_EMAIL_KASA_2=0.5,RCVD_IN_DNSWL_NONE=-0.0001,SUBJ_ALL_CAPS=0.5,UNIVERSAL_HTMLv20160523_1=0.1,UNIVERSAL_HTMLv20160523_2=0.1,UNIVERSAL_HTMLv20160523_3=0.1,UNIVERSAL_HTMLv20160523_5=0.1,URIBL_BLOCKED=0.2]
recips=72

and searching all e-mail from forwarded e-mail list to local awl (mysql)
in amavis

what is stupid.......

sometimes i get

delay=127.0.0.1[127.0.0.1]:10628, conn_use=3, delay=6773,
delays=6517/5.8/0/250, dsn=4.4.2, status=deferred (lost connection with
127.0.0.1[127.0.0.1] while sending end of data -- message may be sent
more than once)


now i change "smtp_connection_reuse_time_limit=400s"

because i get in postfix log:

"delay=127.0.0.1[127.0.0.1]:10628, conn_use=3, delay=6773,
delays=6517/5.8/0/250, dsn=4.4.2, status=deferred (lost connection with
127.0.0.1[127.0.0.1] while sending end of data -- message may be sent
more than once)"

and in log amavis I found terminate connections after 300s
"smtp_connection_reuse_time_limit" is default 300s
 



I solve this problem by adding:
in master.cf

1)smtp       inet  n       -       y       -       100      smtpd -o
receive_override_options=no_address_mappings

2)remove "no_address_mappings" in transport:
    ......
    86.xxx.xxx.199:10027 inet n  -   n   -   -      smtpd
    -o smtpd_proxy_timeout=900s
    ......


Works fine but all incomming "aliasgroup" from my allow network (without
amavis) not working - this is obvious (no_address_mappings in smtp)


and change map /etc/postfix/amavis_bypass
...
#without amavis
86.xxx.xxx.0/24 FILTER smtp:10.0.100.5:10025
.....

and I add another local transport like:

10.0.100.5:10025 inet n  -   n   -   -  smtpd
    -o content_filter=
    -o mynetworks_style=host
    -o mynetworks=10.0.100.0/24
    -o local_recipient_maps=
    -o relay_recipient_maps=
    -o strict_rfc821_envelopes=yes
    -o smtp_tls_security_level=none
    -o smtpd_tls_security_level=none
    -o smtpd_restriction_classes=
    -o smtpd_delay_reject=no
    -o smtpd_client_restrictions=permit_mynetworks,reject
    -o smtpd_helo_restrictions=
    -o smtpd_sender_restrictions=
    -o smtpd_recipient_restrictions=permit_mynetworks,reject
    -o smtpd_end_of_data_restrictions=
    -o smtpd_error_sleep_time=0
    -o smtpd_soft_error_limit=1001
    -o smtpd_hard_error_limit=1000
    -o smtpd_client_connection_count_limit=0
    -o smtpd_client_connection_rate_limit=0
   -o
receive_override_options=no_header_body_checks,no_unknown_recipient_checks


This working - My question is. Is there a simpler solution? Because now
my "mail route" is:

- incomming e-mail
- if IP (whitlisted) go to: 
   - local transport 10.0.100.5 and go to lmtp

- if IP (from 0.0.0.0) go to:
   - local haproxy
   - local haproxy go to amavis
   - amavis scanned
   - amavis return to postfix
- postfix local transport 10.0.100.5 and go to lmtp


Reply | Threaded
Open this post in threaded view
|

Re: postfix + forwadgroup + external amavis with haproxy and no_address_mappings

mami64
Hi
In that configurations cannot work delimiter

in main.cf exists

recipient_delimiter = +


On 28.04.2020 10:15, natan maciej milaszewski wrote:

> Hi
> I have debian 9 and postfix 3.1.14. Generally, I have distributed mail
> traffic over several machines
>
> - separately for sent mail - here I have postfix
> - separately for incoming e-mails - here I have postfix + external amavis
>
>
> The general outline is this:
>
> 1) mail arrives at postfix
> 2) postfix transfers it to Amavis
>     - it really is a local haproxy which directs to one of three amavis
>
> 3) mail returns from amavis on a given ip: port (which is filtered from
> outside the firewall)
> 4) using LMTP to dovecot cluster and then to maildirs and then to sieve
>       virtual_transport = lmtp: inet: 10.0.100.5: 24
>
>
>
>
> Some my restryctions
> smtpd_client_restrictions =
> # local map with host and network wgo must go to amavis or without amavisa
>         check_client_access cidr:/etc/postfix/amavis_bypass,
>         reject_unauth_pipelining,
>         permit
>
> /etc/postfix/amavis_bypass
>
> #without amavis
> 86.xxx.xxx.0/24 OK
> 89.xxx.xxx.0/24 Ok
> 10.0.100.21/32 OK
> 10.0.100.22/32 OK
> 10.0.100.23/32 OK
> 10.0.100.24/32 OK
> 10.0.100.25/32 OK
> 89.206.41.19/32 OK
> #other go to amavis
> 0.0.0.0/0 FILTER smtp-amavis:[127.0.0.1]:10628
>
>
>
> master.cf:
> smtp-amavis     unix    -       -       -       -       80       smtp
>         -o smtp_data_done_timeout=6000s
>         -o smtp_send_xforward_command=yes
>         -o disable_dns_lookups=yes
>
> #80 cosnnections - and in my amavis I have 90 (10+overtime )
>
>
> #returns from amavis  IP .199
>
> 86.xxx.xxx.199:10027 inet n  -   n   -   -      smtpd
>     -o smtpd_proxy_timeout=900s
>     -o content_filter=
>     -o mynetworks_style=host
>     -o mynetworks=10.0.100.0/24,86.xxx.xxx.199/32,
>     -o local_recipient_maps=
>     -o relay_recipient_maps=
>     -o strict_rfc821_envelopes=yes
>     -o smtp_tls_security_level=none
>     -o smtpd_tls_security_level=none
>     -o smtpd_restriction_classes=
>     -o smtpd_delay_reject=no
>     -o smtpd_client_restrictions=permit_mynetworks,reject
>     -o smtpd_helo_restrictions=
>     -o smtpd_sender_restrictions=
>     -o smtpd_recipient_restrictions=permit_mynetworks,reject
>     -o smtpd_end_of_data_restrictions=
>     -o smtpd_error_sleep_time=0
>     -o smtpd_soft_error_limit=1001
>     -o smtpd_hard_error_limit=1000
>     -o smtpd_client_connection_count_limit=0
>     -o smtpd_client_connection_rate_limit=0
>     -o
> receive_override_options=no_header_body_checks,no_unknown_recipient_checks,no_address_mappings
>
>
> All works fine but sometimes my "users" use a mial forwarding .... In
> that forwarding have (100-200 email) like
>
> [hidden email] ---> [hidden email], [hidden email],
> [hidden email], [hidden email]
>
> And all forward e-mail was "releback" in smtp and go to amavis.
>
> In amavis I get:
>
> Apr 16 15:11:11 amavis2 amavis[10499]: (10499-01) ESMTP
> [86.xxx.xxx.155]:10628
> /var/amavis/tmp/amavis-20200416T151111-10499-r3E5zU6i: <[hidden email]> ->
> <[hidden email]>,<[hidden email]>,<[hidden email]>,<[hidden email]>,<[hidden email]>,<[hidden email]>
> SIZE=2129 BODY=7BIT Received: from myserver.domainltd.pl
> ([86.xxx.xxx.199]) by localhost (amavis2.localdomain [86.xxx.xxx.155])
> (amavisd-new, port 10628) with ESMTP; Thu, 16 Apr 2020 15:11:11 +0200 (CEST)
>
>
> Apr 16 15:11:11 amavis2 amavis[10499]: (10499-01) spam_scan:
> score=-0.198 autolearn=no autolearn_force=no
> tests=[BAYES_00=-1.9,DCC_REPUT_70_89=0.1,HTML_IMAGE_RATIO_06=0.001,HTML_MESSAGE=0.001,IQ_EMAIL_KASA_2=0.5,RCVD_IN_DNSWL_NONE=-0.0001,SUBJ_ALL_CAPS=0.5,UNIVERSAL_HTMLv20160523_1=0.1,UNIVERSAL_HTMLv20160523_2=0.1,UNIVERSAL_HTMLv20160523_3=0.1,UNIVERSAL_HTMLv20160523_5=0.1,URIBL_BLOCKED=0.2]
> recips=22
> Apr 16 15:11:11 amavis2 amavis[10499]: (10499-01) spam_scan:
> score=-0.198 autolearn=no autolearn_force=no
> tests=[BAYES_00=-1.9,DCC_REPUT_70_89=0.1,HTML_IMAGE_RATIO_06=0.001,HTML_MESSAGE=0.001,IQ_EMAIL_KASA_2=0.5,RCVD_IN_DNSWL_NONE=-0.0001,SUBJ_ALL_CAPS=0.5,UNIVERSAL_HTMLv20160523_1=0.1,UNIVERSAL_HTMLv20160523_2=0.1,UNIVERSAL_HTMLv20160523_3=0.1,UNIVERSAL_HTMLv20160523_5=0.1,URIBL_BLOCKED=0.2]
> recips=4
> Apr 16 15:11:11 amavis2 amavis[10499]: (10499-01)  spam_scan:
> score=-0.198 autolearn=no autolearn_force=no
> tests=[BAYES_00=-1.9,DCC_REPUT_70_89=0.1,HTML_IMAGE_RATIO_06=0.001,HTML_MESSAGE=0.001,IQ_EMAIL_KASA_2=0.5,RCVD_IN_DNSWL_NONE=-0.0001,SUBJ_ALL_CAPS=0.5,UNIVERSAL_HTMLv20160523_1=0.1,UNIVERSAL_HTMLv20160523_2=0.1,UNIVERSAL_HTMLv20160523_3=0.1,UNIVERSAL_HTMLv20160523_5=0.1,URIBL_BLOCKED=0.2]
> recips=82
> Apr 16 15:11:11 amavis2 amavis[10499]: (10499-01) spam_scan:
> score=-0.198 autolearn=no autolearn_force=no
> tests=[BAYES_00=-1.9,DCC_REPUT_70_89=0.1,HTML_IMAGE_RATIO_06=0.001,HTML_MESSAGE=0.001,IQ_EMAIL_KASA_2=0.5,RCVD_IN_DNSWL_NONE=-0.0001,SUBJ_ALL_CAPS=0.5,UNIVERSAL_HTMLv20160523_1=0.1,UNIVERSAL_HTMLv20160523_2=0.1,UNIVERSAL_HTMLv20160523_3=0.1,UNIVERSAL_HTMLv20160523_5=0.1,URIBL_BLOCKED=0.2]
> recips=72
>
> and searching all e-mail from forwarded e-mail list to local awl (mysql)
> in amavis
>
> what is stupid.......
>
> sometimes i get
>
> delay=127.0.0.1[127.0.0.1]:10628, conn_use=3, delay=6773,
> delays=6517/5.8/0/250, dsn=4.4.2, status=deferred (lost connection with
> 127.0.0.1[127.0.0.1] while sending end of data -- message may be sent
> more than once)
>
>
> now i change "smtp_connection_reuse_time_limit=400s"
>
> because i get in postfix log:
>
> "delay=127.0.0.1[127.0.0.1]:10628, conn_use=3, delay=6773,
> delays=6517/5.8/0/250, dsn=4.4.2, status=deferred (lost connection with
> 127.0.0.1[127.0.0.1] while sending end of data -- message may be sent
> more than once)"
>
> and in log amavis I found terminate connections after 300s
> "smtp_connection_reuse_time_limit" is default 300s
>  
>
>
>
> I solve this problem by adding:
> in master.cf
>
> 1)smtp       inet  n       -       y       -       100      smtpd -o
> receive_override_options=no_address_mappings
>
> 2)remove "no_address_mappings" in transport:
>     ......
>     86.xxx.xxx.199:10027 inet n  -   n   -   -      smtpd
>     -o smtpd_proxy_timeout=900s
>     ......
>
>
> Works fine but all incomming "aliasgroup" from my allow network (without
> amavis) not working - this is obvious (no_address_mappings in smtp)
>
>
> and change map /etc/postfix/amavis_bypass
> ...
> #without amavis
> 86.xxx.xxx.0/24 FILTER smtp:10.0.100.5:10025
> .....
>
> and I add another local transport like:
>
> 10.0.100.5:10025 inet n  -   n   -   -  smtpd
>     -o content_filter=
>     -o mynetworks_style=host
>     -o mynetworks=10.0.100.0/24
>     -o local_recipient_maps=
>     -o relay_recipient_maps=
>     -o strict_rfc821_envelopes=yes
>     -o smtp_tls_security_level=none
>     -o smtpd_tls_security_level=none
>     -o smtpd_restriction_classes=
>     -o smtpd_delay_reject=no
>     -o smtpd_client_restrictions=permit_mynetworks,reject
>     -o smtpd_helo_restrictions=
>     -o smtpd_sender_restrictions=
>     -o smtpd_recipient_restrictions=permit_mynetworks,reject
>     -o smtpd_end_of_data_restrictions=
>     -o smtpd_error_sleep_time=0
>     -o smtpd_soft_error_limit=1001
>     -o smtpd_hard_error_limit=1000
>     -o smtpd_client_connection_count_limit=0
>     -o smtpd_client_connection_rate_limit=0
>    -o
> receive_override_options=no_header_body_checks,no_unknown_recipient_checks
>
>
> This working - My question is. Is there a simpler solution? Because now
> my "mail route" is:
>
> - incomming e-mail
> - if IP (whitlisted) go to: 
>    - local transport 10.0.100.5 and go to lmtp
>
> - if IP (from 0.0.0.0) go to:
>    - local haproxy
>    - local haproxy go to amavis
>    - amavis scanned
>    - amavis return to postfix
> - postfix local transport 10.0.100.5 and go to lmtp
>
>

Reply | Threaded
Open this post in threaded view
|

Re: postfix + forwadgroup + external amavis with haproxy and no_address_mappings

Matus UHLAR - fantomas
In reply to this post by mami64
On 28.04.20 10:15, natan maciej milaszewski wrote:
>I have debian 9 and postfix 3.1.14. Generally, I have distributed mail
>traffic over several machines

>#other go to amavis
>0.0.0.0/0 FILTER smtp-amavis:[127.0.0.1]:10628

>master.cf:
>smtp-amavis     unix    -       -       -       -       80       smtp
>        -o smtp_data_done_timeout=6000s
>        -o smtp_send_xforward_command=yes
>        -o disable_dns_lookups=yes

I believe you should use lmtp instead of smtp for amavis connections.

>#80 cosnnections - and in my amavis I have 90 (10+overtime )
>
>
>#returns from amavis  IP .199
>
>86.xxx.xxx.199:10027 inet n  -   n   -   -      smtpd
>    -o smtpd_proxy_timeout=900s
>    -o content_filter=
>    -o mynetworks_style=host
>    -o mynetworks=10.0.100.0/24,86.xxx.xxx.199/32,
>    -o local_recipient_maps=
>    -o relay_recipient_maps=
>    -o strict_rfc821_envelopes=yes
>    -o smtp_tls_security_level=none
>    -o smtpd_tls_security_level=none
>    -o smtpd_restriction_classes=
>    -o smtpd_delay_reject=no
>    -o smtpd_client_restrictions=permit_mynetworks,reject
>    -o smtpd_helo_restrictions=
>    -o smtpd_sender_restrictions=
>    -o smtpd_recipient_restrictions=permit_mynetworks,reject
>    -o smtpd_end_of_data_restrictions=
>    -o smtpd_error_sleep_time=0
>    -o smtpd_soft_error_limit=1001
>    -o smtpd_hard_error_limit=1000
>    -o smtpd_client_connection_count_limit=0
>    -o smtpd_client_connection_rate_limit=0
>    -o
>receive_override_options=no_header_body_checks,no_unknown_recipient_checks,no_address_mappings


>All works fine but sometimes my "users" use a mial forwarding .... In
>that forwarding have (100-200 email) like

forwarding how?

>[hidden email] ---> [hidden email], [hidden email],
>[hidden email], [hidden email]
>
>And all forward e-mail was "releback" in smtp and go to amavis.

do you want to say that users send the same mail to postfix, not from any of
whitelisted addreses?
Maybe you should whitelist localhost (127.0.0.1) too.


>sometimes i get
>
>delay=127.0.0.1[127.0.0.1]:10628, conn_use=3, delay=6773,
>delays=6517/5.8/0/250, dsn=4.4.2, status=deferred (lost connection with
>127.0.0.1[127.0.0.1] while sending end of data -- message may be sent
>more than once)

lmtp should help here.

>"smtp_connection_reuse_time_limit" is default 300s

connection reuse won't help here. timeouts and smtp are the problem here.


>I solve this problem by adding:
>in master.cf
>
>1)smtp       inet  n       -       y       -       100      smtpd -o
>receive_override_options=no_address_mappings
>
>2)remove "no_address_mappings" in transport:
>    ......
>    86.xxx.xxx.199:10027 inet n  -   n   -   -      smtpd
>    -o smtpd_proxy_timeout=900s
>    ......

No, you don't solve the problem,  you work around the problem.


>and change map /etc/postfix/amavis_bypass
>...
>#without amavis
>86.xxx.xxx.0/24 FILTER smtp:10.0.100.5:10025
>.....


>and I add another local transport like:
>
>10.0.100.5:10025 inet n  -   n   -   -  smtpd

you are only making this complicated.


>This working - My question is. Is there a simpler solution? Because now
>my "mail route" is:
>
>- incomming e-mail
>- if IP (whitlisted) go to: 
>   - local transport 10.0.100.5 and go to lmtp
>
>- if IP (from 0.0.0.0) go to:
>   - local haproxy
>   - local haproxy go to amavis
>   - amavis scanned
>   - amavis return to postfix
>- postfix local transport 10.0.100.5 and go to lmtp

use LMTP for filtering.

--
Matus UHLAR - fantomas, [hidden email] ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
I'm not interested in your website anymore.
If you need cookies, bake them yourself.