Rejecting International Email

classic Classic list List threaded Threaded
54 messages Options
123
Reply | Threaded
Open this post in threaded view
|

Re: Rejecting International Email

Jorey Bump
Paweł Leśniak wrote, at 04/30/2008 09:05 AM:

> Erwan David pisze:
>> It may work, but be aware that language is difficult to detect, and
>> you can receive mails written in english but using Big5 charset (or
>> any charset containing ascii).
>>
>> Even my mail will be sent in UTF-8 (because I made a citation of
>> your name, and I did not configure to use ISO-8859-2)
>>  
> OK. With UTF-8 it's hard to detect in what language is the message
> written (but SpamAssassin still can do that by checking which portion of
> UTF charset You're using). But You don't use ISO-8859-2 or KOI-8R to
> write a message ever. Unless You write in these charsets and that's when
> You are from country which needs this encodings. I know it's not a
> *real* solution. But still in my opinion it's way better than rejecting
> mail by IP/Domain.

I routinely get mail in foreign character sets from colleagues who have
received such mail and are CC'ing me in the reply. Some mail clients are
configured to use the character set of the original sender. So, you can
still get burned by this technique.

In any case, blocking by character set doesn't yield enough results to
be worthwhile, especially if you have to deal with even a single false
positive. Adding scores for character sets may be useful, but outright
rejection is a bad idea, in my experience.

Reply | Threaded
Open this post in threaded view
|

Re: Rejecting International Email

Jorey Bump
In reply to this post by Terry Carmen
Terry Carmen wrote, at 04/29/2008 11:24 PM:
> Jorey Bump wrote:
>> Why do you assume such a close relationship between geography and the
>> Internet?
>
> Because the IP blocks are allocated by region, then by country.

This may be somewhat accurate for determining the location of the relay,
but not the user. A user anywhere in the world can use a relay anywhere
in the world.

>> In fact, I stopped blocking based on country when someone who lived a
>> mile away from me couldn't email my client who lived a block away. Why
>> not? Because unknown to him, his .com domain's web host and email
>> provider was based in Singapore.
> So you update your CIDR list and move on.

I'm allergic to maintaining unnecessary whitelists. I'd rather spend the
time with my kids.

> Only you can decide if any lost business is worth the reduction in spam.

That's a bit backwards from an admin's perspective. The goal is to
eliminate spam without creating any false positives. I'd rephrase this:

"Only you can decide if the reduction in spam is worth the lost business."

Say that to any business person, and I guarantee the only words they'll
hear are "lost business", and look at you like you're crazy to even
suggest that the possibility is acceptable.


Reply | Threaded
Open this post in threaded view
|

Re: Rejecting International Email

d.hill
In reply to this post by Terry Carmen
On Wed, 30 Apr 2008 at 08:58 -0400, [hidden email] confabulated:

> We tried greylisting. It's a little helpful, but the delay was really
> annoying the users. And for large companies that bounce outbound mail around
> to multiple servers after a 4xx reject, some messages would never get through
> because the greylist time would expire before that sender/recipient/sending
> ip came back around.

Sounds like you were greylisting on the entire IP address. Here, I only
greylist on the first three octets of the IP address. I saw the exact same
results as you at first using the entire IP address.
Reply | Threaded
Open this post in threaded view
|

Re: Rejecting International Email

Henrik K
On Wed, Apr 30, 2008 at 01:42:01PM +0000, D Hill wrote:

> On Wed, 30 Apr 2008 at 08:58 -0400, [hidden email] confabulated:
>
>> We tried greylisting. It's a little helpful, but the delay was really  
>> annoying the users. And for large companies that bounce outbound mail
>> around to multiple servers after a 4xx reject, some messages would
>> never get through because the greylist time would expire before that
>> sender/recipient/sending ip came back around.
>
> Sounds like you were greylisting on the entire IP address. Here, I only  
> greylist on the first three octets of the IP address. I saw the exact
> same results as you at first using the entire IP address.

Just use selective greylisting. No point greylisting something that's
clearly a legit server.

See http://hege.li/howto/spam/etc/postfix/in/ for some examples.

Reply | Threaded
Open this post in threaded view
|

Re: Rejecting International Email

Terry Carmen
In reply to this post by M. Fioretti-2

> Last but not least: the assumption that potential customers who get a
> rejection message with a toll free number (which would mean spamming: why
> _answer_ to messages which you are sure are spam?) would be:
>
> savy enough to understand what to do and above all,
>
> bothered to call to be whitelisted instead of just email another supplier
> is as naive as the one that C-R systems are a good idea: the only people
> who waste their time this way are likely to be those to whom you ALREADY
> owe money.
>
> This said, if you haven't got the point yet, in spite of several detailed,
> real world example by several people, I have really nothing to add. It's
> your (actually, your clients...) problem only.
>  
Actually, you don't appear to "get it", but I don't really care.

The client is happy, the client's customers are happy and the client's
vendors are happy. Everybody is happy except for a few people on this
mailing list.

Terry






They have decided that the tradeoff (if any) is acceptable and are happy
with it.
Reply | Threaded
Open this post in threaded view
|

Re: Rejecting International Email

Paweł Leśniak
 
> Actually, you don't appear to "get it", but I don't really care.
>
> The client is happy, the client's customers are happy and the client's
> vendors are happy. Everybody is happy except for a few people on this
> mailing list.
>
Personally I do not agree.
You seem to be sure that client is aware of trade-of. Clients' customers
are happy today. As long as they don't have to connect via network which
is blacklisted. It's Internet, not intranet. Who knows the routes?
Happiness when You are on deadline with tasks and can't send emails -
priceless. Of course they don't know that emails are sent to /dev/null,
so they don't care at the beginning, because thay think everything is
OK. Do you care now?

"Everybody" is really virtual to me.

P.


Reply | Threaded
Open this post in threaded view
|

Re: Rejecting International Email

Carlwill
In reply to this post by Terry Carmen
It appears my ignorance has ruffled a few feathers and I deeply
apologize for this. I am in no way trying to upset anyone and while I
do agree that most spam is sent to and from the US, I manage a Postfix
server for a defense contractor and have specific requirements to make
sure I only receive email from the US. Obviously I don't have a lot of
Postfix / Email experience but I am learning a lot from all your
suggestions and advice so please don't have any ill feelings towards
my original post.

I am going to look into more suggested ways of stopping un-wanted
emails with foreign characters and language subsets which are banned
per DoD standards.

My sincere apologies to all and thanks for the great info again!

- Carlos
Reply | Threaded
Open this post in threaded view
|

Re: Rejecting International Email

John Baker-5
Well, I generally lurk here but I don't see what you have to apologize
for. It was worthwhile question, some people just get easily excitable.

I don't know exact stats but most spam seems to come from world wide
botnets. I've noticed when particularly large batches come in that are
clearly created from the same scripts and part of the same "job" they'll
    originate from IP's on different continents and in different countries.

I'm sort of courious about the defense contractor only wanting mail from
US ip's. Do they think that this is specifically more secure? With the
global nature of today's market for just about everything you'd think
that they would have to have some business dealings with suppliers from
other countries.



Carlos Williams wrote:

> It appears my ignorance has ruffled a few feathers and I deeply
> apologize for this. I am in no way trying to upset anyone and while I
> do agree that most spam is sent to and from the US, I manage a Postfix
> server for a defense contractor and have specific requirements to make
> sure I only receive email from the US. Obviously I don't have a lot of
> Postfix / Email experience but I am learning a lot from all your
> suggestions and advice so please don't have any ill feelings towards
> my original post.
>
> I am going to look into more suggested ways of stopping un-wanted
> emails with foreign characters and language subsets which are banned
> per DoD standards.
>
> My sincere apologies to all and thanks for the great info again!
>
> - Carlos


--
John Baker
Network Systems Administrator
Marlboro College
Phone: 451-7551 off campus; 551 on campus
Reply | Threaded
Open this post in threaded view
|

Re: Rejecting International Email

M. Fioretti-2
In reply to this post by Paweł Leśniak
On Wed, Apr 30, 2008 16:06:45 PM +0200, Paweł Leśniak wrote:

>> Actually, you don't appear to "get it", but I don't really care.
>>
>> The client is happy, the client's customers are happy and the client's
>> vendors are happy. Everybody is happy except for a few people on this
>> mailing list.
>>
> Personally I do not agree.
> You seem to be sure that client is aware of trade-of.

Exactly. From all Terry has said so far, it really sounds like:

a) he continues to believe, or at least market to his client, the idea
   that "only accepting connections from **systems** inside the
   geographic area they service" is **equal** to "only accepting
   connections from **REAL** customers or potential customers who LIVE
   AND WORK "inside the geographic area they service""

b) his clients are in the same situation as those people who start
   using C-R systems without having a clue as to what they really do
   and are happy because they don't realize that they aren't blocking
   just spam.

Final note to Terry: if your clients are happy, OK. Although you're
making them a disservice if you haven't clearly explained that the
assumption in point a) above, upon which you are building the service
you sell them, has very shaky basis in reality: *they* may still be
one of the exceptions, I'm not in a position to deny it or to care at
all, but they should be sure to be one of such exceptions.

The _only_ thing which made unhappy "a few people on this list" (or
me, at least), is simply the idea to endorse, even by just being
silent, the assumption in point a) as a general criterion valid in
anything but very few, fery special situations.

Then again, there's nothing Postfix anymore in this thread and there
is almost unanimity on the matter, so why continue on the list?

             Marco
--
Your own civil rights and the quality of your life heavily depend on how
software is used *around* you:            http://digifreedom.net/node/84
Reply | Threaded
Open this post in threaded view
|

Re: Rejecting International Email

Bill Cole-3
In reply to this post by Carlwill
At 10:06 AM -0400 4/30/08, Carlos Williams wrote:
>I manage a Postfix
>server for a defense contractor and have specific requirements to make
>sure I only receive email from the US.

That is not possible to assure unless you operate your mail server on
a default-reject basis and only accept cryptographically signed mail
coming in on SMTP sessions that have passed rigorous authentication
as a client that you know to be in the US. That effectively demands
TLS with verified client certs.

Of course, some of that may not be necessary depending on what you
mean by "email from the US." For example, I recently worked in an
environment where getting mail to people I worked with in the same
building in Auburn Hills, MI (suburban Detroit) involved the mail
going over one private WAN to and Exchange server New York, then to
another Exchange server Germany, then to a Sendmail system and across
the Internet to another Sendmail system in Germany, another Exchange
server in Germany, and then back over another private WAN to an
Exchange server in Auburn Hills. Was that mail "from the US" or not?
Does it matter that the sender and recipient worked for the US units
of major .de conglomerates?

Any mechanism that you try to use short of digitally signing all mail
and positively identifying all SMTP clients is going to have edge
cases that fail. An IP address allocated under RIPE to a European
company may be held by a machine in the US, while one allocated under
ARIN might be owned by a Japanese company and assigned to a machine
in Africa. You can find .com and .net domains associated with
machines on any continent. You can see perfectly legitimate mail
coming out of a machine in the US with a sender using a .com (or even
.us) domain name, even though he has never been closer to the US than
Delhi.

Bottom line: whoever came up with this requirement probably had a
weak understanding of how Internet email works in the real world


--
Bill Cole
[hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Rejecting International Email

Mark Goodge
Bill Cole wrote:
> At 10:06 AM -0400 4/30/08, Carlos Williams wrote:
>> I manage a Postfix
>> server for a defense contractor and have specific requirements to make
>> sure I only receive email from the US.

[snip]
> Bottom line: whoever came up with this requirement probably had a weak
> understanding of how Internet email works in the real world

The problem, of course, is that many of us work for organisations or
people with a weak understanding of how the Internet works in the real
world, and we have to do our best to provide solutions to rather
ridiculous requirements.

The OP's question could probably have been better phrased as "What
solution would best approximate the contractor's requirements while
still allowing the system to function correctly?" There are some tools,
such as geo-IP coding and content filtering on language, which would
probably be close enough under normal circumstances, provided that the
OP is happy with the occasional false positive. But, as you say, there
isn't a system which can be implemented solely within his own network
which will absolutely meet his needs, as digital signing requires
compliance from all the senders as well and no other solution will be
100% reliable.

Mark

Reply | Threaded
Open this post in threaded view
|

Re: Rejecting International Email

M. Fioretti-2
On Thu, May 01, 2008 07:16:58 AM +0100, Mark Goodge wrote:

> Bill Cole wrote:
>> At 10:06 AM -0400 4/30/08, Carlos Williams wrote:
>>> I manage a Postfix
>>> server for a defense contractor and have specific requirements to make
>>> sure I only receive email from the US.
>
> [snip]
>> Bottom line: whoever came up with this requirement probably had a weak
>> understanding of how Internet email works in the real world
>
> The problem, of course, is that many of us work for organisations or
> people with a weak understanding of how the Internet works in the real
> world, and we have to do our best to provide solutions to rather
> ridiculous requirements.

of course. The only point of this discussion was to make clear that
such requirements are ridiculous, that they cannot be implemented at
the mail server level in a way that doesn't create false positives
(=lost business) and that the programmer has the responsibility to
explain this to his or her clients.

With respect to this, and to bring the question a bit closer to
Postfix:

> The OP's question could probably have been better phrased as "What
> solution would best approximate the contractor's requirements while
> still allowing the system to function correctly?"

In cases like the truck delivery company, solving it by MTA-only means
seems like "I only have a hammer and I only know how to use a hammer,
so I'll treat every problem as a nail". Isn't it much simpler, better
and much less a burden to maintain, rather than geo-IP or language
filtering, to work _outside_ the MTA, ie to simply:

put online a form with an (accessible) captcha as the *only* contact
point. A form which says clearly that only the email address you put
in there will be usable for further contacts (even if it's a .ru
address, why should you refuse customers who just relocated from
Moscow or Shangai to your neighborhood?), a form that before any other
data asks your ZIP code and refuses the request period if it isn't a
"serviceable" one;

whenever a company employee writes anything to any email address
(whether it came from that form or not is irrelevant at this point),
automatically add it to a whitelist

refuse period any email that does not come from a whitelisted address
(yes, it's useless if your customers computers become spambots, but
that's another topic, isn't it?)

I'll now leave to real postfix gurus the task to explain how to
automatically create the whitelist above, I'm interested to know the
most efficient way.

The bottom line remains that if what the requirements say is to
discriminate on the basis of sender's PHYSICAL LOCATION, they are not
saying AT ALL to discriminate by looking at his or her email address,
or that there is any relation whatever between address and
location. Therefore it makes infinitely more sense to check the
customer's location with non-email means and then accept whatever
address he or she provides.

                                Marco
--
Your own civil rights and the quality of your life heavily depend on how
software is used *around* you:            http://digifreedom.net/node/84
Reply | Threaded
Open this post in threaded view
|

Re: Rejecting International Email

Terry Carmen
In reply to this post by M. Fioretti-2
M. Fioretti wrote:
> Exactly. From all Terry has said so far, it really sounds like:
>
> a) he continues to believe, or at least market to his client, the idea
>    that "only accepting connections from **systems** inside the
>    geographic area they service" is **equal** to "only accepting
>    connections from **REAL** customers or potential customers who LIVE
>    AND WORK "inside the geographic area they service""
>  

OK. I was trying to be brief, but you guys just won't let this drop.

The customer, not me, selects which countries they wish to accept mail
from. I explained that the IP allocation lists were close, but not
perfect and that they will need to whitelist and blacklist various CIDR
blocks that are incorrectly allocated.

And I gave them a configuration application to do it from.

The countries that are blocked are by their choice, not mine and if
they're unhappy with the results, it only takes a click of a button to
turn it back on.
> Final note to Terry: if your clients are happy, OK. Although you're
> making them a disservice if you haven't clearly explained that the
> assumption in point a) above, upon which you are building the service
> you sell them, has very shaky basis in reality: *they* may still be
> one of the exceptions, I'm not in a position to deny it or to care at
> all, but they should be sure to be one of such exceptions.
>  
I allow any IPs that have been manually whitelisted, which includes
Yahoo, GMail and customers/vendors that use IPs that are inside blocked
countries, blacklists, etc.
        check_client_access cidr:/etc/postfix/OK.cidr,

I deny all IPs that have an RDNS that matches a number of Dynamic-IP
regular expressions
        check_client_access regexp:/etc/postfix/spam_ip_regex,

I deny all IPs except those for those that are geo-located inside areas
they have allowed.
        check_client_access cidr:/etc/postfix/ok_countries.cidr,

I deny any IPs that are found on several RBLs

All denied connections get a reject message containing a toll-free phone
number. A phone call will result in being whitelisted immediately.

Mail that is accepted is scanned for spamminess and attachments. If it's
very spammy or contains anything but text or images, it's held for
manual inspection, then released or deleted.

This is all completely in their control, and they get a number of
reports every day showing which connections were blocked or allowed and
why, what was passed or deleted, as well as the normal postfix queue and
I/O stats, so if they're not happy with the results, all it takes is a
click of a button.

I did not sell a "magic box" that claims to "Eliminate Spam". I built a
system that met their requirements to block mail that originated outside
the areas they want to talk to, and gave them full control over it.

> The _only_ thing which made unhappy "a few people on this list" (or
> me, at least), is simply the idea to endorse, even by just being
> silent, the assumption in point a) as a general criterion valid in
> anything but very few, fery special situations.
>
>  
While rejecting mail via geographic IP matching and regular expressions
is not perfect and may be "politically incorrect", it is quite
effective. It's use is a *business decision* not a technology decision.

The only thing that matters is that it complies with the RFCs and meets
the customer's requirements.

Terry


Reply | Threaded
Open this post in threaded view
|

Re: Rejecting International Email

M. Fioretti-2
On Thu, May 01, 2008 09:37:48 AM -0400, Terry Carmen wrote:

> The customer, not me, selects which countries they wish to accept mail
> from. I explained that the IP allocation lists were close, but not perfect
> and that they will need to whitelist and blacklist various CIDR blocks that
> are incorrectly allocated.
>
> And I gave them a configuration application to do it from.

OK, great. This wasn't clear, not to me at least, from your original assertion.

Marco

--
Your own civil rights and the quality of your life heavily depend on how
software is used *around* you:            http://digifreedom.net/node/84
123