Outsourced anti-spam and Issues with VRFY

classic Classic list List threaded Threaded
22 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Outsourced anti-spam and Issues with VRFY

Charles Marcus
Hi all,

We are testing a new outsourced anti-spam service (Edgewave/RedCondor).
We are letting their systems check for valid recipients using the VRFY
command, but their default verifier uses <[hidden email]>,
instead of <[hidden email]> as the FROM address when verifying.
Because I employ anti-spoofing (anything from/to one of our addresses
must be sent through our server or we reject it), we had to set up a
custom verifier in the Edgewave control panel.

Everything is working, mail is flowing, but grepping on
(lost|warning|error|fatal|panic), I see a lot of these:

> 2013-07-31T10:48:39-04:00 myhost postfix-25/smtpd[17729]: lost
> connection after VRFY from smtp446.redcondor.net[208.80.204.46]

with occasional

> 2013-07-31T10:46:28-04:00 myhost postfix-25/smtpd[17993]: too many
> errors after VRFY from smtp644.redcondor.net[208.80.206.44]

tossed in.

Is this normal? Or is this indicative of a problem that should be fixed?

Thanks

Reply | Threaded
Open this post in threaded view
|

Re: Outsourced anti-spam and Issues with VRFY

DTNX Postmaster
On Jul 31, 2013, at 18:19, Charles Marcus <[hidden email]> wrote:

> We are testing a new outsourced anti-spam service (Edgewave/RedCondor). We are letting their systems check for valid recipients using the VRFY command, but their default verifier uses <[hidden email]>, instead of <[hidden email]> as the FROM address when verifying. Because I employ anti-spoofing (anything from/to one of our addresses must be sent through our server or we reject it), we had to set up a custom verifier in the Edgewave control panel.
>
> Everything is working, mail is flowing, but grepping on (lost|warning|error|fatal|panic), I see a lot of these:
>
>> 2013-07-31T10:48:39-04:00 myhost postfix-25/smtpd[17729]: lost connection after VRFY from smtp446.redcondor.net[208.80.204.46]
>
> with occasional
>
>> 2013-07-31T10:46:28-04:00 myhost postfix-25/smtpd[17993]: too many errors after VRFY from smtp644.redcondor.net[208.80.206.44]
>
> tossed in.
>
> Is this normal? Or is this indicative of a problem that should be fixed?

No, not normal. Apparently their verifier takes a shortcut by just
dropping the connection instead of sending a QUIT, like it should. And
it also does that quite often it seems, which leads to the second error.

First, make sure that your 'custom verifier' includes all the
appropriate steps. Depending on how that is set up, you yourself might
be the culprit ;-)

If that does not work, you could try debugging this with the
'notify_classes' parameter;

http://www.postfix.org/postconf.5.html#notify_classes

This will send a session transcript to the postmaster for every session
that is rejected or otherwise ends in an error. Using the information
from that you should be able to find where the problem originates.

Mvg,
Joni

Reply | Threaded
Open this post in threaded view
|

Re: Outsourced anti-spam and Issues with VRFY

Wietse Venema
DTNX Postmaster:
> >> 2013-07-31T10:48:39-04:00 myhost postfix-25/smtpd[17729]: lost connection after VRFY from smtp446.redcondor.net[208.80.204.46]

Not a real problem...

> > with occasional
> >
> >> 2013-07-31T10:46:28-04:00 myhost postfix-25/smtpd[17993]: too many errors after VRFY from smtp644.redcondor.net[208.80.206.44]

That *is* a problem. Postfix will slow down and eventually hang
up when a client sends too many commands that cause an error
reply (as in VERY with a non-existent recipient).

To make this robust the verified MUST disconnect after each probe.

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: Outsourced anti-spam and Issues with VRFY

Charles Marcus
On 2013-07-31 1:23 PM, [hidden email] (Wietse Venema) [hidden email] wrote:
That *is* a problem. Postfix will slow down and eventually hang
up when a client sends too many commands that cause an error
reply (as in VERY with a non-existent recipient).

To make this robust the verified MUST disconnect after each probe.

When you say 'the verifieD MUST disconnect...', I'm assuming you meant 'verifieR' - meaning, their server, the one connecting to mine sending VRFY probes?

Thanks, I'm passing this on to their support team.

-- 

Best regards,

Charles
Reply | Threaded
Open this post in threaded view
|

Re: Outsourced anti-spam and Issues with VRFY

Wietse Venema
Charles Marcus:

> On 2013-07-31 1:23 PM, [hidden email] (Wietse Venema)
> <[hidden email] (Wietse Venema)> wrote:
> > That*is*  a problem. Postfix will slow down and eventually hang
> > up when a client sends too many commands that cause an error
> > reply (as in VERY with a non-existent recipient).
> >
> > To make this robust the verified MUST disconnect after each probe.
>
> When you say 'the verifieD MUST disconnect...', I'm assuming you meant
> 'verifieR' - meaning, their server, the one connecting to mine sending
> VRFY probes?

Yes. Even when the verifier sends only known recipients, Postfix
will limit the number of VRFY commands that can be sent in succession
(that's why the verifier must also hang up after success).

That limit (smtpd_junk_command_limit) is configurable, but the same
limit applies to all clients.

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: Outsourced anti-spam and Issues with VRFY

Charles Marcus
On 2013-07-31 2:53 PM, [hidden email] (Wietse Venema)
<[hidden email] (Wietse Venema)> wrote:

> Charles Marcus:
>> When you say 'the verifieD MUST disconnect...', I'm assuming you
>> meant 'verifieR' - meaning, their server, the one connecting to mine
>> sending VRFY probes?
> Yes. Even when the verifier sends only known recipients, Postfix
> will limit the number of VRFY commands that can be sent in succession
> (that's why the verifier must also hang up after success).
>
> That limit (smtpd_junk_command_limit) is configurable, but the same
> limit applies to all clients.

Understood, thanks again Wietse.

Obviously the right thing to do is get them to fix their verifier to
hang up.

--

Best regards,

Charles


Reply | Threaded
Open this post in threaded view
|

Re: Outsourced anti-spam and Issues with VRFY

Charles Marcus
In reply to this post by Charles Marcus
On 2013-07-31 1:44 PM, Charles Marcus [hidden email] wrote:
On 2013-07-31 1:23 PM, [hidden email] (Wietse Venema) [hidden email] wrote:
That *is* a problem. Postfix will slow down and eventually hang
up when a client sends too many commands that cause an error
reply (as in VERY with a non-existent recipient).

To make this robust the verified MUST disconnect after each probe.

When you say 'the verifieD MUST disconnect...', I'm assuming you meant 'verifieR' - meaning, their server, the one connecting to mine sending VRFY probes?

Thanks, I'm passing this on to their support team.

Ok, I got the following response... is doing as they are recommending an acceptable way to resolve this issue?

We are set up for performance with VRFY probes and by modifying your postfix config file so postfix will not nave a performance issue by setting postfix option smtpd_soft_error_limit to be larger than smtpd_hard_error_limit.

This can be done by setting set "-o smtpd_soft_error_limit=1001 -o smtpd_hard_error_limit=1000" in his postfix config file.

I'm fine doing this, as long as this isn't going to cause other issues or have unintended consequences.

Note: my postfix server *only* accepts inbound connections (including VRFY probes) from their netblocks.

Thanks,

-- 

Best regards,

Charles
Reply | Threaded
Open this post in threaded view
|

Re: Outsourced anti-spam and Issues with VRFY

DTNX Postmaster
On Aug 4, 2013, at 21:08, Charles Marcus <[hidden email]> wrote:

> On 2013-07-31 1:44 PM, Charles Marcus <[hidden email]> wrote:
>> On 2013-07-31 1:23 PM, [hidden email] (Wietse Venema) <[hidden email] (Wietse Venema)> wrote:
>>> That *is*
>>>  a problem. Postfix will slow down and eventually hang
>>> up when a client sends too many commands that cause an error
>>> reply (as in VERY with a non-existent recipient).
>>>
>>> To make this robust the verified MUST disconnect after each probe.
>>
>> Thanks, I'm passing this on to their support team.
>
> Ok, I got the following response... is doing as they are recommending an acceptable way to resolve this issue?
>
>> We are set up for performance with VRFY probes and by modifying your postfix config file so postfix will not nave a performance issue by setting postfix option smtpd_soft_error_limit to be larger than smtpd_hard_error_limit.
>>
>> This can be done by setting set "-o smtpd_soft_error_limit=1001 -o smtpd_hard_error_limit=1000" in his postfix config file.
>
> I'm fine doing this, as long as this isn't going to cause other issues or have unintended consequences.
>
> Note: my postfix server *only* accepts inbound connections (including VRFY probes) from their netblocks.

It's your server, but if it were mine, I'd steer clear of them if this is the argument they come back with.

Even on busier setups a proper disconnect should not be a problem, as long as they are not being daft and sending a VRFY probe for every incoming message, but rather using some kind of caching mechanism.

In other words, I just would not trust them to be the gateway for all incoming mail. Who knows what else they take performance shortcuts on, and crooked workarounds like these have a tendency to come back and bite you in the arse at the most inconvenient times.

Just my opinion, though. YMMV :-)

Mvg,
Joni

Reply | Threaded
Open this post in threaded view
|

Re: Outsourced anti-spam and Issues with VRFY

Wietse Venema
In reply to this post by Charles Marcus
Charles Marcus:
> > We are set up for performance with VRFY probes and by modifying
> > your postfix config file so postfix will not nave a performance
> > issue by setting postfix option smtpd_soft_error_limit to be larger
> > than smtpd_hard_error_limit.

That is nonsense, as demonstrated below:

    # postconf smtpd_hard_error_limit=1 smtpd_soft_error_limit=2
    # postfix reload
    # telnet 127.0.0.1 smtp
    Trying 127.0.0.1...
    Connected to localhost.
    Escape character is '^]'.
    220 hades.porcupine.org ESMTP Postfix
    hello foo
    502 5.5.2 Error: command not recognized
    421 4.7.0 hades.porcupine.org Error: too many errors
    Connection closed by foreign host.

These people never tested this recommendation, just like they
never tested their software against Postfix or else they would
have been aware of the smtpd_junk_command_limit feature.

It should be safe to dumb down Postfix defenses, provided that
no-one else can connect to your SMTP server.

However given the poor quality assurance with respect to Postfix,
I would be suspicious about the quality assurance of their code.

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: Outsourced anti-spam and Issues with VRFY

Charles Marcus
On 2013-08-04 7:30 PM, [hidden email] (Wietse Venema)
<[hidden email] (Wietse Venema)> wrote:

> Charles Marcus:
>>> We are set up for performance with VRFY probes and by modifying
>>> your postfix config file so postfix will not nave a performance
>>> issue by setting postfix option smtpd_soft_error_limit to be larger
>>> than smtpd_hard_error_limit.
>
> That is nonsense, as demonstrated below:
>
>      # postconf smtpd_hard_error_limit=1 smtpd_soft_error_limit=2
>      # postfix reload
>      # telnet 127.0.0.1 smtp
>      Trying 127.0.0.1...
>      Connected to localhost.
>      Escape character is '^]'.
>      220 hades.porcupine.org ESMTP Postfix
>      hello foo
>      502 5.5.2 Error: command not recognized
>      421 4.7.0 hades.porcupine.org Error: too many errors
>      Connection closed by foreign host.
>
> These people never tested this recommendation, just like they
> never tested their software against Postfix or else they would
> have been aware of the smtpd_junk_command_limit feature.
>
> It should be safe to dumb down Postfix defenses, provided that
> no-one else can connect to your SMTP server.

Thanks Wietse,

After your hint I read up on this command at:

http://www.postfix.org/STRESS_README.html#legacy

but I'm still unsure how to implement this properly to address this
particular issue...

Would it be to lower the junk setting to 1? Would I also need to lower
the others (timeout and hard_error_limit)? Or maybe use different values?

 > However given the poor quality assurance with respect to Postfix,
 > I would be suspicious about the quality assurance of their code.

I understand. All I can say that they are considerably more effective
than the last 2 or 3 solutions we have used (webroot, postini, then
maildistiller), and on top of that, they are 1/3 the cost. So, I'd like
to continue using them if I can eliminate these errors.

Also - I hate to ask (it isn't your job to do their job), but could you
suggest off the top of your head what they *should* be doing? Would
properly closing all VRFY probe connections really impact performance on
their side that much - especially if they are caching these responses
(so those wouldn't even need to be sent downstream to my server)? I
really hope I don't find out they aren't caching them for at least a few
hours to a day or so.

Thanks again,

Charles

Reply | Threaded
Open this post in threaded view
|

Re: Outsourced anti-spam and Issues with VRFY

Mikael Bak-2
On 08/05/2013 02:15 PM, Charles Marcus wrote:
> Also - I hate to ask (it isn't your job to do their job), but could you
> suggest off the top of your head what they *should* be doing? Would
> properly closing all VRFY probe connections really impact performance on
> their side that much - especially if they are caching these responses
> (so those wouldn't even need to be sent downstream to my server)? I
> really hope I don't find out they aren't caching them for at least a few
> hours to a day or so.
>

I could be wrong.
I have the impression that they should use something similar to postfix'
"reject_unverified_recipient". That's what our anti spam solution does.

HTH,
Mikael

Reply | Threaded
Open this post in threaded view
|

Re: Outsourced anti-spam and Issues with VRFY

Noel Jones-2
In reply to this post by Charles Marcus
On 8/5/2013 7:15 AM, Charles Marcus wrote:

> On 2013-08-04 7:30 PM, [hidden email] (Wietse Venema)
> <[hidden email] (Wietse Venema)> wrote:
>> Charles Marcus:
>>>> We are set up for performance with VRFY probes and by modifying
>>>> your postfix config file so postfix will not nave a performance
>>>> issue by setting postfix option smtpd_soft_error_limit to be larger
>>>> than smtpd_hard_error_limit.
>>
>> That is nonsense, as demonstrated below:
>>
>>      # postconf smtpd_hard_error_limit=1 smtpd_soft_error_limit=2
>>      # postfix reload
>>      # telnet 127.0.0.1 smtp
>>      Trying 127.0.0.1...
>>      Connected to localhost.
>>      Escape character is '^]'.
>>      220 hades.porcupine.org ESMTP Postfix
>>      hello foo
>>      502 5.5.2 Error: command not recognized
>>      421 4.7.0 hades.porcupine.org Error: too many errors
>>      Connection closed by foreign host.
>>
>> These people never tested this recommendation, just like they
>> never tested their software against Postfix or else they would
>> have been aware of the smtpd_junk_command_limit feature.
>>
>> It should be safe to dumb down Postfix defenses, provided that
>> no-one else can connect to your SMTP server.
>
> Thanks Wietse,
>
> After your hint I read up on this command at:
>
> http://www.postfix.org/STRESS_README.html#legacy
>
> but I'm still unsure how to implement this properly to address this
> particular issue...
>
> Would it be to lower the junk setting to 1? Would I also need to
> lower the others (timeout and hard_error_limit)? Or maybe use
> different values?

Set those three limits to 100 or higher.  Those controls are
intended to prevent random clients from wasting your time.  Since
you don't allow connections from random clients, it's safe to
increase them.

# main.cf
smtpd_hard_error_limit = 100
smtpd_soft_error_limit = 100
smtpd_junk_command_limit = 100


>
>> However given the poor quality assurance with respect to Postfix,
>> I would be suspicious about the quality assurance of their code.

I'm guessing their advice assumed you use the default setting for
smtpd_hard_error_limit.

I'm also willing to accept that they could offer effective filtering
services even if they aren't postfix experts.



  -- Noel Jones
Reply | Threaded
Open this post in threaded view
|

Re: Outsourced anti-spam and Issues with VRFY

Charles Marcus
On 2013-08-05 9:21 AM, Noel Jones <[hidden email]> wrote:
> Set those three limits to 100 or higher.  Those controls are
> intended to prevent random clients from wasting your time.  Since
> you don't allow connections from random clients, it's safe to
> increase them.
>
> # main.cf
> smtpd_hard_error_limit = 100
> smtpd_soft_error_limit = 100
> smtpd_junk_command_limit = 100

Thanks Noel... I'll do this, unless I can get them to change their VRFY
service to properly close these connections - or stop using a MAIL FROM
that is in my domain name for their SMTP RCPT TO option so we could use
that option.

Same question to you though - do you think that *not* closing VRFY
probes/connections properly is improving their performance in any
meaningful way?

Thanks again,

Charles

Reply | Threaded
Open this post in threaded view
|

Re: Outsourced anti-spam and Issues with VRFY

Noel Jones-2
On 8/5/2013 9:09 AM, Charles Marcus wrote:

> On 2013-08-05 9:21 AM, Noel Jones <[hidden email]> wrote:
>> Set those three limits to 100 or higher.  Those controls are
>> intended to prevent random clients from wasting your time.  Since
>> you don't allow connections from random clients, it's safe to
>> increase them.
>>
>> # main.cf
>> smtpd_hard_error_limit = 100
>> smtpd_soft_error_limit = 100
>> smtpd_junk_command_limit = 100
>
> Thanks Noel... I'll do this, unless I can get them to change their
> VRFY service to properly close these connections - or stop using a
> MAIL FROM that is in my domain name for their SMTP RCPT TO option so
> we could use that option.
>
> Same question to you though - do you think that *not* closing VRFY
> probes/connections properly is improving their performance in any
> meaningful way?
>

Depends on the volume.  At high volume if they can batch up
recipients and VRFY a bunch in one blast it would help.  OTOH, if
the connection is just sitting there idle for 5 minutes between each
VRFY, not likely much difference.

But that assumes everyone has fast transaction startup...

And this depends on their software too.  Maybe they've optimized for
lots of open idle connections, and I can imagine some random MTA
might take a long time to set up an SMTP transaction.

I don't suppose an open idle connection from an somewhat authorized
client will bother anything, so just go with it.



  -- Noel Jones
Reply | Threaded
Open this post in threaded view
|

Re: Outsourced anti-spam and Issues with VRFY

Charles Marcus
On 2013-08-05 10:53 AM, Noel Jones <[hidden email]> wrote:
> I don't suppose an open idle connection from an somewhat authorized
> client will bother anything, so just go with it.

Ok - and by 'go with it', you mean just adjust the settings per your
last email and be done with it, right?

I asked Edgewave to escalate this issue, so we'll see what their tier
2/3 tech says - if they don't/can't change the way their system works,
I'll make these changes.

Thanks again for taking the time to reply Noel!

--

Best regards,

Charles


Reply | Threaded
Open this post in threaded view
|

Re: Outsourced anti-spam and Issues with VRFY

Noel Jones-2
On 8/5/2013 10:30 AM, Charles Marcus wrote:
> On 2013-08-05 10:53 AM, Noel Jones <[hidden email]> wrote:
>> I don't suppose an open idle connection from an somewhat authorized
>> client will bother anything, so just go with it.
>
> Ok - and by 'go with it', you mean just adjust the settings per your
> last email and be done with it, right?

That's right.

>
> I asked Edgewave to escalate this issue, so we'll see what their
> tier 2/3 tech says - if they don't/can't change the way their system
> works, I'll make these changes.

If you like their service, I don't think it's unreasonable to make
this harmless change to your system to accommodate them.

Just don't ask them for postfix advice  ;)


  -- Noel Jones
Reply | Threaded
Open this post in threaded view
|

Re: Outsourced anti-spam and Issues with VRFY

Wietse Venema
Noel Jones:
> On 8/5/2013 10:30 AM, Charles Marcus wrote:
> > On 2013-08-05 10:53 AM, Noel Jones <[hidden email]> wrote:
> >> I don't suppose an open idle connection from an somewhat authorized
> >> client will bother anything, so just go with it.
> >
> > Ok - and by 'go with it', you mean just adjust the settings per your
> > last email and be done with it, right?
>
> That's right.

This should be safe.

Apparently they are optimizing for Sendmail-like MTAs that have a
large process startup cost when they make one connection per (vrfy)
request. Postfix does not have that problem.

> Just don't ask them for postfix advice  ;)

Indeed.

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: Outsourced anti-spam and Issues with VRFY

Stan Hoeppner
In reply to this post by Charles Marcus
On 8/5/2013 9:09 AM, Charles Marcus wrote:

> On 2013-08-05 9:21 AM, Noel Jones <[hidden email]> wrote:
>> Set those three limits to 100 or higher.  Those controls are
>> intended to prevent random clients from wasting your time.  Since
>> you don't allow connections from random clients, it's safe to
>> increase them.
>>
>> # main.cf
>> smtpd_hard_error_limit = 100
>> smtpd_soft_error_limit = 100
>> smtpd_junk_command_limit = 100
>
> Thanks Noel... I'll do this, unless I can get them to change their VRFY
> service to properly close these connections - or stop using a MAIL FROM
> that is in my domain name for their SMTP RCPT TO option so we could use
> that option.
...

I'm sure you've already covered this Charles but just in case you
haven't I'll mention it anyway.

No matter what you do here with this outsourced service, I'd suggest you
document all Postfix config changes you're making, or save a copy of
your main/master.cf files as they were before embarking down this path,
should you later decide to ditch this service.  You don't want these
custom setting biting you in the backside if/when you switch back to
direct-to-MX, or possibly to another outsourced service.

--
Stan


Reply | Threaded
Open this post in threaded view
|

Re: Outsourced anti-spam and Issues with VRFY

Charles Marcus
On 2013-08-05 11:33 PM, Stan Hoeppner <[hidden email]> wrote:
> I'm sure you've already covered this Charles but just in case you
> haven't I'll mention it anyway.
>
> No matter what you do here with this outsourced service, I'd suggest you
> document all Postfix config changes you're making, or save a copy of
> your main/master.cf files as they were before embarking down this path,
> should you later decide to ditch this service.  You don't want these
> custom setting biting you in the backside if/when you switch back to
> direct-to-MX, or possibly to another outsourced service.

Thanks Stan, good idea, but I'm ahead of you.

I have a 'special' section in main.cf for all of my custom settings (at
the very bottom of the file), and I have these particular changes under
my 'work around stupid Edgewave behavior (not closing VRFY probes)'
section... ;)

--

Best regards,

Charles


Reply | Threaded
Open this post in threaded view
|

Re: Outsourced anti-spam and Issues with VRFY

Charles Marcus
In reply to this post by Noel Jones-2
On 2013-08-05 9:21 AM, Noel Jones <[hidden email]> wrote:
> Set those three limits to 100 or higher.  Those controls are
> intended to prevent random clients from wasting your time.  Since
> you don't allow connections from random clients, it's safe to
> increase them.
>
> # main.cf
> smtpd_hard_error_limit = 100
> smtpd_soft_error_limit = 100
> smtpd_junk_command_limit = 100

Ok, I made these changes, but wanted to confirm something...

I'm still seeing lots of these in the logs (but I'm fairly sure this was
to be expected?):

> 2013-08-06T06:31:55-04:00 myhost postfix-25/smtpd[12390]: lost connection after VRFY from smtp644.redcondor.net[208.80.206.44]

What I was expecting was that the 'too many errors' issue would be
resolved by these changes, but alas, I had one about an hour after
making the change (and to be sure I did fully restart postfix):

> 2013-08-06T07:33:25-04:00 myhost postfix-25/smtpd[12873]: too many errors after VRFY from smtp644.redcondor.net[208.80.206.44]

So... does this mean that I need to adjust the values higher? Or dos
this mean that this problem is caused by something else?

Thanks,

--

Best regards,

Charles

12