Microsoft silently discarding emails after recepit

classic Classic list List threaded Threaded
35 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Re: Microsoft silently discarding emails after recepit

Richard Damon
On 1/7/18 6:30 PM, Stephen Satchell wrote:

> On 01/07/2018 03:09 PM, D'Arcy Cain wrote:
>> On 01/07/2018 01:15 PM, Yuval Levy wrote:
>>> would detract from the main issue which is "silently discarded emails,"
>>> I behavior that in my view is plain wrong and threatens the usefulness
>>> of email more than a few false positive spam messages.
>>
>> Absolutely.  There are only two things that an MTA should do with email,
>> deliver it or bounce it.  Silently dropping is plain wrong.
>
> But the best time to bounce the mail is during initial delivery.
> Microsoft makes it clear that their Smart-whatever is done *after* the
> SMTP server has accepted the mail.  Rejecting the mail afterwards has
> the risk of sending the bounce not to the sender, but to another party
> whose name just happened to be in the From: field.
>
> Most of the spam I've received over the years doesn't have a live
> account as specified in the From: field.  So the problem is, where to
> send the bounce notification if you already said "I got it"?
>
> You guessed:  you don't.  You drop the mail, and that's that.  No
> side-channel spaming.
>
> When I smart-hosted a bunch of Plesk and CPanel systems with edge
> PostFix servers, I had to be very careful not to run into this very
> same situation with quotas.  It was tedious and complicated and ugly,
> but I made it work.  I also had those same servers accept mail from
> the Web boxes and did anti-spam, dropping "bad" e-mails after sending
> a note to my admin logs about the dirty delivery.
>
> (I was able to find the small number of spammers in a community of
> thousands of accounts this way, and closed them due to violations of
> the terms of service.)
>
Which is why a QUALITY MTA will do all the test that decide IF a message
will be delivered before accepting the message. Test for where to
deliver (normal inbox/spam box) can be done later. This means you need
to validate recipient & quotas etc before acceptance. If quota tests
fall behind and an over quote message gets accepted, you still should
deliver and consider the quota limit 'soft'.

Of course, the issue now becomes that most of the 'free email' systems
aren't quality system, so the above promise isn't kept, and some stuff
is just dropped.

--
Richard Damon

Reply | Threaded
Open this post in threaded view
|

Re: Microsoft silently discarding emails after recepit

stroller-6
In reply to this post by Yuval Levy

> On 7 Jan 2018, at 19:15, Yuval Levy <[hidden email]> wrote:
> …
> My understanding of Microsoft's reply is that there is nothing wrong
> with the server / IP / reputation.  Their reply point to a content
> filter at work.

Because they say it was determined by SmartScreen™?

Is the server's IP / reputation assessed separately? How do you know?

Stroller.

Reply | Threaded
Open this post in threaded view
|

Re: Microsoft silently discarding emails after recepit

soumitri@iitk.ac.in
In reply to this post by Yuval Levy
Thanks to this list and its good members, I got support contact easily,along with lots of helpful information.
It seems my issue is solved for now. OK MS support.
I have few users, who are paid customers of MS. Possibly, they also lodged the issue with MS.
But it is still bad for a reputed organization to silently discard the messages. I guess there is a problem of tuning with there mail firewall.

Following is the reply from MS, which explains noting to me technically. Any way my relay is a low traffic sender.
-------------------
Our investigation has determined that there are no active blocks against these IP(s); however, some messages were filtered. We have confirmed that these IP(s) are eligible for conditional mitigation, but may be subject to low daily email limits until they have established a good reputation. Please note that this mitigation does not guarantee that your mail will be delivered to a user's inbox.
---------------------------

Sincerely,
Soumitri Mishra, IITK

On 07/01/18 4:12 AM, Yuval Levy wrote:
> TL;DR: IF YOU ARE STILL EXPERIENCING DELIVERABILITY ISSUES, CONTACT
> OUTLOOK.COM DELIVERABILITY SUPPORT:
> <http://go.microsoft.com/fwlink/?LinkID=614866>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Microsoft silently discarding emails after recepit

Yuval Levy
In reply to this post by Peter Ajamian
On 2018-01-07 04:32 AM, Peter wrote:
> So to put it simply, they're basically saying that their black box
> thinks that your IP(s) are sending SPAM.

That's not how I read my conversation with them.  My understanding based
on their explanations and their advice on how to fix the problem is that
their black box thinks that the content of the message sent is spam and
has very little to do with the IP(s).  Confirmed by their SNDS tools.


> I can speculate that this might be a case of your emails somewhat
> resembling emails they have seen from scammers that claim to be legal
> representatives in order to further some scam or another.  This is only
> a guess though, I could be completely wrong here as I really don't know
> anything more about Microsoft's black box than you do.

Indeed we can only speculate about what triggers the blackbox.  My email
was just a set of boring corporate decisions in a language that I have
never seen used in any scam/spam.  They are neatly formatted into a PDF
saved directly out of LibreOffice with a short text body, composed in
Thunderbird.  The only minor objectionable issue I find with my email is
the GPG signatures: I sign with a key that is not associated with any
email address, contrary to RFC_dont_remember_which, and I sign with an
expired key (my bad: good enough for my purpose and therefore very low
priority to fix).  I doubt Microsoft pulls in GPG keys and verifies
content signatures anyway.  There is a banking resolution, so there is
text in the PDF saying that the corporation has decided to open a bank
account with bank X.  If that triggers the content filter, it is a
really picky one and whoever programmed it should be sentenced to
manually read out loud Nigerian spam for eight hours straight.


>> HTML format?
>
> Highly unlikely.

Then why Microsoft's advice to "brand" and "properly format" the
message?  Coming from the company that has gifted the world the scourge
that is HTML email, and whose view of what represent a properly format
has probably been over-represented in the training sample that was fed
to the neural network (just guessing) powering the filter?


>> FLOSS client software and O/S?
>
> Also highly unlikely, I do the same and don't have issues.  Also it's
> all about following the SMTP protocol correctly, which has nothing to do
> with whether the software used is FLOSS or proprietary or anything
> in-between.

On this one I tend to agree with you, for a different reason.  I do not
think it is about following the SMTP protocol correctly because my MTA
received a 250 from their MTA on the messages and because of primary
factors mentioned in Microsoft's vague description of SmartScreen's
outcome being "primarily due to mail content and recipient interaction
with that mail."

My very wild guess is that their system mixes up correlation with
causality or is skewed against exotic / seldom seen combination of the
User-Agent header just because it does not see it so often, but that
would be stupid given the overwhelmingly large volume of spam vs
ordinary mail.


>> sending MTA is dynamic/residential>
> This shouldn't be the case, but if your submission server resides on a
> dynamic IP address then this could very well be the case.  I say
> "shouldn't", though, because there have been known cases of anti-spam
> appliances in the past that do deep inspection of Received headers and
> compare IP addresses found in them against policy blacklists.  These
> types of blacklists are designed only to be used against the IP address
> of the connecting server, not IPs found in headers.  That said, I don't
> think that Microsoft is doing that.

Your argument on this one is convincing.  I will probably never know the
answer, at least not in my setup.  I realized this was a weak spot and
fixed it with:

/etc/postfix/main.cf

  mime_header_checks = regexp:/etc/postfix/headers_checks
  header_checks = regexp:/etc/postfix/headers_checks

/etc/postfix/headers_checks

  /^Received:/ IGNORE
  /^X-Originating-IP:/ IGNORE
  /^X-Mailer:/ IGNORE
  /^Mime-Version 1.0.*:/ REPLACE Mime-Version: 1.0


> I really would not put any malicious intent here.

Agree.  Negligence is easier to prove and in both cases the outcome is
the same and so is the practical fix.  Just the punishment is different,
but I am not out to punish Microsoft, I just wish it behaved more
collaboratively.


>> Bottom line, I think the problem is more ethical than technical.
>
> I certainly think it's technical on their part, but if by ethical you
> mean that Microsoft just doesn't care enough about you to want to solve
> your problem then you're probably right.

By ethical I mean their way of conducting business, which results in
their technical decisions.  I think it is unethical to make the
statement "My MTA received an email from your MTA and it is queued for
delivery" when in fact it should be "My MTA has received an email from
your MTA and has dropped it into a black hole," or even "My MTA has
received an email from your MTA and what will happen next I won't tell
you."  Even assuming no malicious intent, the first statement is a
misrepresentation with serious consequences.  The last statement makes
email worthless.


> I would suggest, as others have, that if you cannot resolve this
> directly then you use a relayhost for messages that go out to Microsoft
> clients, then you should at least be able to get your mail through.

A simpler way for me is to open and operate a mailbox on Microsoft's
system and communicate with Microsoft clients through that mailbox.
Both solutions are not good enough: who guarantees that Microsoft is not
going to drop silently a message coming from their own system?

This is why I am still of the opinion that the problem is more ethical
than technical.  It is akin to a rental car company renting a car that
will loose a wheel while driving.  Sure the immediate solution is
technical, but the overarching solution is ethical:  a standard that
distinguish between good behavior, where the accidental loss of a wheel
in circumstances beyond the control of the service provider vs bad
behavior where the accidental loss of a wheel is caused by the
negligence of the service provider and could be avoided with a proper
standard of care (which in this case would be to put the message
identified as spammy in the junk folder; or to let the sending MTA know
that the message was dropped).

Yuv
Reply | Threaded
Open this post in threaded view
|

Re: Microsoft silently discarding emails after recepit

Yuval Levy
In reply to this post by soumitri@iitk.ac.in
On 2018-01-08 12:48 AM, [hidden email] wrote:
> It seems my issue is solved for now. OK MS support.

Happy to read your issue is solved, and thank you for sharing the reply
from Microsoft.


> -------------------
> Our investigation has determined that there are no active blocks against
> these IP(s); however, some messages were filtered. We have confirmed
> that these IP(s) are eligible for conditional mitigation, but may be
> subject to low daily email limits until they have established a good
> reputation. Please note that this mitigation does not guarantee that
> your mail will be delivered to a user's inbox.
> ---------------------------

Interestingly different from what I received.  The above looks to me
like an IP(s) issue, vs. the content issue that I believe has resulted
in my loss.

Yuv
Reply | Threaded
Open this post in threaded view
|

Re: Microsoft silently discarding emails after recepit

Yuval Levy
In reply to this post by Stephen Satchell
On 2018-01-07 09:32 AM, Stephen Satchell wrote:
> After reading all the responses, and reading the reference links about
> Microsoft Smartscreen, I have a really stupid question:

Not stupid at all.  I'll limit my answer because of the off-topic nature
of the question.  Let me know if you are interested to know more, off-list.


> Have you considered encrypting your e-mail traffic?

Yes.  Clients do not buy in.


> You are a lawyer.  The contents of your mail could be considered
> sensitive, especially if exposure of the mail could materially affect
> the legal status of your clients.

You are romanticizing the profession, or at least holding me for more
than I am.  I don't represent political activists or do other high stake
legal work.  I do transactions, corporations, taxes, and most
information has a very short half-life.


>  By encrypting your mail, you
> instantly remove the idea that the message you are sending is "bulk" in
> any way, because the methods used to encrypt email involve key-pairs
> specific to you and your client.  Using public keys to encrypt, only the
> secret private key can be used to decrypt -- and Microsoft wouldn't have
> access to the secret keys.

I am familiar with encryption and I see all the benefits you are
touting.  Your comment made me wonder whether something simpler, such as
encoding the email in base64, would be enough to deter Microsoft from
deep inspection?  Does anybody have experience with the treatment of
base64 encoded emails?


> Hmmm....need to experiment.  Could I write a milter for PostFix that
> would (1) detect the message body is in plaintext, (2) the recipient
> address has a public key listed in the key servers, and (3) encrypt the
> body of the message.

Neat idea.  not sure if the recipient will like that, though.  I am as
guilty as my clients on the convenience vs confidentiality tradeoff.
Last time I received encrypted emails I was annoyed at the delay for the
decrypted message to appear in Thunderbird's preview pane every time I
was moving up/down the message list.  I understand the design decision
not to store the unencrypted message at destination, but I still do not
like it, also because of the risk of not being able to decrypt it in the
future.  I'd rather decrypt incoming messages once, then encrypt all of
the storage with my own keys, but I have not found anything that works
like that.

Yuv
Reply | Threaded
Open this post in threaded view
|

Re: Microsoft silently discarding emails after recepit

Viktor Dukhovni
In reply to this post by Yuval Levy


> On Jan 8, 2018, at 1:56 AM, Yuval Levy <[hidden email]> wrote:
>
> /etc/postfix/headers_checks
>
>  /^Received:/ IGNORE

Complete absence of Received: headers is likely to increase your
spam score.

--
        Viktor.

Reply | Threaded
Open this post in threaded view
|

Re: Microsoft silently discarding emails after recepit

Yuval Levy
On 2018-01-08 02:20 AM, Viktor Dukhovni wrote:
> Complete absence of Received: headers is likely to increase your
> spam score.

What would you suggest instead?

Reply | Threaded
Open this post in threaded view
|

Re: Microsoft silently discarding emails after recepit

Viktor Dukhovni


> On Jan 8, 2018, at 2:22 AM, Yuval Levy <[hidden email]> wrote:
>
>> Complete absence of Received: headers is likely to increase your
>> spam score.
>
> What would you suggest instead?

You can mask the address with 127.0.0.1, and otherwise keep the header
as-is.  Mind you, I would not expect Microsoft to pay much attention
to addresses in Received headers except when the connecting SMTP client
is "trusted" in some manner.  Few users would be able to use submission
services from their broadband IP in the face of such filters.

Not all the advice you'll get on this list is sound.  Generally, doing
less (be as typical as possible) is safer than doing more.

--
        Viktor.

Reply | Threaded
Open this post in threaded view
|

Re: Microsoft silently discarding emails after recepit

Peter Ajamian
In reply to this post by Yuval Levy
On 08/01/18 19:56, Yuval Levy wrote:
> On 2018-01-07 04:32 AM, Peter wrote:
>> So to put it simply, they're basically saying that their black box
>> thinks that your IP(s) are sending SPAM.
>
> That's not how I read my conversation with them.  My understanding based
> on their explanations and their advice on how to fix the problem is that
> their black box thinks that the content of the message sent is spam and
> has very little to do with the IP(s).

I may have misunderstood, anyways, either way it is based, at least at
some point, on the content.

> Indeed we can only speculate about what triggers the blackbox.

Yep, so it probably doesn't pay to speculate down this line of thinking
any further.

>>> HTML format?
>>
>> Highly unlikely.
>
> Then why Microsoft's advice to "brand" and "properly format" the
> message?  Coming from the company that has gifted the world the scourge
> that is HTML email, and whose view of what represent a properly format
> has probably been over-represented in the training sample that was fed
> to the neural network (just guessing) powering the filter?

Well, this at least is easily tested.  Send the same email in both plain
text and HTML/Plain multi-part and see if the HTML one goes through and
the plain is blocked.

>>> FLOSS client software and O/S?
>>
>> Also highly unlikely, I do the same and don't have issues.  Also it's
>> all about following the SMTP protocol correctly, which has nothing to do
>> with whether the software used is FLOSS or proprietary or anything
>> in-between.
>
> On this one I tend to agree with you, for a different reason.  I do not
> think it is about following the SMTP protocol correctly because my MTA
> received a 250 from their MTA on the messages and because of primary
> factors mentioned in Microsoft's vague description of SmartScreen's
> outcome being "primarily due to mail content and recipient interaction
> with that mail."
>
> My very wild guess is that their system mixes up correlation with
> causality or is skewed against exotic / seldom seen combination of the
> User-Agent header just because it does not see it so often, but that
> would be stupid given the overwhelmingly large volume of spam vs
> ordinary mail.

This as well is test-able.  Try the same message in different
user-agents.  Borrow a laptop with windows, configure outlook to send
through your server and send a message identical to one that was dropped
from the same network.

You also mentioned that you're GPG signing the mail and using PDFs.  If
the PDF is identified as a possible source of a virus (and PDFs can
contain viruses) then this could be a legitimate reason for them to drop
your email.  At any rate the GPG and PDF scenarios are also easy to test
by sending a message that is not signed and one with the content that is
just plain text (not in a PDF) and seeing if either of these tests go
through.

One more thing I should note here is that if MS black box is anything
like other, open source anti-spam solutions then it will use a scoring
system, so an expired cert might give you a half point, no HTML content
might add one point, PDF might be another point.  Some word or phrase in
the email might add a bit more and if your SPAM score gets above a
certain threshold it is dropped, so in that respect it is not any one
thing that is causing your mail to get dropped but a number of things
combined.

>>> sending MTA is dynamic/residential>
>> This shouldn't be the case, but if your submission server resides on a
>> dynamic IP address then this could very well be the case.  I say
>> "shouldn't", though, because there have been known cases of anti-spam
>> appliances in the past that do deep inspection of Received headers and
>> compare IP addresses found in them against policy blacklists.  These
>> types of blacklists are designed only to be used against the IP address
>> of the connecting server, not IPs found in headers.  That said, I don't
>> think that Microsoft is doing that.
>
> Your argument on this one is convincing.  I will probably never know the
> answer, at least not in my setup.  I realized this was a weak spot and
> fixed it with:
>
> /etc/postfix/main.cf
>
>   mime_header_checks = regexp:/etc/postfix/headers_checks
>   header_checks = regexp:/etc/postfix/headers_checks
>
> /etc/postfix/headers_checks
>
>   /^Received:/ IGNORE
>   /^X-Originating-IP:/ IGNORE
>   /^X-Mailer:/ IGNORE
>   /^Mime-Version 1.0.*:/ REPLACE Mime-Version: 1.0

...and did any of this help?

>> I really would not put any malicious intent here.
>
> Agree.  Negligence is easier to prove and in both cases the outcome is
> the same and so is the practical fix.  Just the punishment is different,
> but I am not out to punish Microsoft, I just wish it behaved more
> collaboratively.

On this we can agree.

>>> Bottom line, I think the problem is more ethical than technical.
>>
>> I certainly think it's technical on their part, but if by ethical you
>> mean that Microsoft just doesn't care enough about you to want to solve
>> your problem then you're probably right.
>
> By ethical I mean their way of conducting business, which results in
> their technical decisions.  I think it is unethical to make the
> statement "My MTA received an email from your MTA and it is queued for
> delivery" when in fact it should be "My MTA has received an email from
> your MTA and has dropped it into a black hole," or even "My MTA has
> received an email from your MTA and what will happen next I won't tell
> you."  Even assuming no malicious intent, the first statement is a
> misrepresentation with serious consequences.  The last statement makes
> email worthless.

Well I think at least here I might be able to give some explanation that
might help.  When an MTA receives a message it has a number of different
choices of what it can do with it:

* It can accept the message and deliver it (delivering it to SPAM is
still considered delivering it here).

* It can accept the message and then drop it (which is what they are
doing here).  This is what they are doing and it's bad.

* It can accept the message, send a DSN (bounce email) to the envelope
sender address and then drop the message. Look up "backscatter" to see
why this is also bad.

* It can reject the message with a 5XX code (or defer it with 4XX).

What is happening here is that at the time that Microsoft accepts the
message they have not processed it through their back box yet, so they
can't tell in time to reject it if the message looks like SPAM to them
or not.  They don't want to send out a DSN because it ends up causing
them to become a source of backscatter and so they drop the message,
which is also bad for reasons that you have already pointed out.

What they should be doing is rejecting it straight away if they possibly
can or delivering it into the user's SPAM folder.  The only time (as
someone else already pointed out) that it is generally considered
appropriate to drop mail is if it contains dangerous content such as a
virus.  In that case delivering it to SPAM is not a good idea because
someone may still open the virus to look at the message.

> This is why I am still of the opinion that the problem is more ethical
> than technical.  It is akin to a rental car company renting a car that
> will loose a wheel while driving.  Sure the immediate solution is
> technical, but the overarching solution is ethical:  a standard that
> distinguish between good behavior, where the accidental loss of a wheel
> in circumstances beyond the control of the service provider vs bad
> behavior where the accidental loss of a wheel is caused by the
> negligence of the service provider and could be avoided with a proper
> standard of care (which in this case would be to put the message
> identified as spammy in the junk folder; or to let the sending MTA know
> that the message was dropped).

The analogy of the rental car fails when you realize that neither you,
nor the recipient of your messages (who likely do not pay MS for the
usage of their hotmail account) are considered by MS to be "customers".
They are more accurately described as the "product" and the MS customers
are actually advertisers that pay to display ads on the webmail
interface.  Now you probably know the full legal issues way better than
I ever could but it does often times pay to see how a company like MS
views it.

Anyways, I hope this continues to help you shed light on the situation.


Peter
Reply | Threaded
Open this post in threaded view
|

Re: Microsoft silently discarding emails after recepit

D'Arcy Cain
In reply to this post by Viktor Dukhovni
On 01/07/2018 05:25 PM, Viktor Dukhovni wrote:
>> On Jan 7, 2018, at 6:09 PM, D'Arcy Cain <[hidden email]> wrote:
>>
>> Absolutely.  There are only two things that an MTA should do with email,
>> deliver it or bounce it.  Silently dropping is plain wrong.
>
> There's a reasonable exception for malware.  Detection of malware has
> a much lower FP rate than detection of spam.  There's little benefit

"Hello.  I have attached that malware that we discussed for your
inspection so that you can take steps to mitigate it.  Please don't open
the attachment."

There are always exceptions to the exceptions.

--
D'Arcy J.M. Cain
System Administrator, Vex.Net
http://www.Vex.Net/ IM:[hidden email]
VoIP: sip:[hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Microsoft silently discarding emails after recepit

Kris Deugau
In reply to this post by Dominic Raferd
Dominic Raferd wrote:
> Otherwise, the formatting of your DKIM record in DNS seems weird (try:
> dig +short 201605sfinacom._domainkey.sfina.com TXT); even if technically
> valid the intermediate quotes may be influencing 'SmartScreen'.

Strictly speaking that result does not in fact contain quotes at all;
it's a byproduct of TXT record chunking.

I ran into a similar issue with a somewhat snarky DNS validation tool;
tinydns automatically splits TXT records into 128-byte chunks by default
(not 256, or "up to 256 on whitespace" as BIND does), which upset the
validation tool.  I had to extend my DNS management tool to publish the
TXT records in such a way that tinydns did not automatically split them
at 128 bytes.

That said, it wouldn't surprise me if that's contributing to the problem...

-kgd
Reply | Threaded
Open this post in threaded view
|

Re: Microsoft silently discarding emails after recepit

Bill Cole-3
In reply to this post by Yuval Levy
A better place for this discussion would be the MailOps list, where a
broader variety of mail admins *INCLUDING MS EMPLOYEES* take part, and
this problem class has been discussed multiple times. See
https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop

With that said, a couple of paragraphs stand out as demanding response.
They literally made me choke on my coffee, repeatedly:

On 8 Jan 2018, at 1:56 (-0500), Yuval Levy wrote:

[...]
> Indeed we can only speculate about what triggers the blackbox.  My
> email
> was just a set of boring corporate decisions in a language that I have
> never seen used in any scam/spam.  They are neatly formatted into a
> PDF
> saved directly out of LibreOffice with a short text body, composed in
> Thunderbird.

This sort of mail -- a PDF attached to a brief text message -- describes
 >90% of the spam that has made it through my spam filters in the past 6
months, much of it "spearphishing" mail aimed at tricking me into doing
risky things involving money for people who claim a false identity.
Unfortunately, it seems to also be a preferred form for high-value
document exchange (i.e. legal papers...) so it is viciously hard to
filter out safely. This is why operations like DocuSign & Greenvelope
exist. Email is a terrible medium for exchange of legal docs, but that's
a tough sale to lawyers...

> The only minor objectionable issue I find with my email is
> the GPG signatures: I sign with a key that is not associated with any
> email address, contrary to RFC_dont_remember_which,

And totally against the whole design of the OpenPGP signing protocol.
You might as well use a key claiming to be from someone else. This is
WORSE than not signing at all. It screams "THIS EMAIL IS FRAUDULENT!!!"

> and I sign with an
> expired key

So it looks like you're using a key that you have in the past said that
you wouldn't use at this date. This is practically begging to be
distrusted.

> (my bad: good enough for my purpose and therefore very low
> priority to fix).

This free-floating assertion seems at odds with the fact that your email
to Microsoft customers is being treated in a manner that only makes
sense for phishing or malware email.

Have you tried taking the 5 minutes required to set up a correct GPG key
and use that instead?

> I doubt Microsoft pulls in GPG keys and verifies
> content signatures anyway.

I do not. It is certainly easier and more sensible than some of the
other behaviors you suggested that MS might be engaging in. If you're
lucky, they cannot find your key; it's best if they can't tell how bogus
your signatures are.

> There is a banking resolution, so there is
> text in the PDF saying that the corporation has decided to open a bank
> account with bank X.  If that triggers the content filter, it is a
> really picky one and whoever programmed it should be sentenced to
> manually read out loud Nigerian spam for eight hours straight.

It is not a bad idea to reject email with a general format that is
widely used by scammers, discussing activities often discussed in such
scams, bearing a doubly bogus "signature."

In fact, that strikes me as a very GOOD idea.

>>> HTML format?
>>
>> Highly unlikely.
>
> Then why Microsoft's advice to "brand" and "properly format" the
> message?

Boilerplate. They cite the most common broad issues, not weird niche
problems like bogus PGP signatures on phishy messages.

SmartScreen is mainly targeted at phishing and malware mail, so the most
common sorts of FPs are cases of mail not having the form and content of
typical business (or personal) one-to-one email. Like it or not, "rich"
content in email is pervasive these days so doing it in odd ways like
embedded PDFs with  short text parts looks suspect. Also, using a stale
key without an email address for "signing" a message as someone your
mail is not from could be argued to be both a format error and a
branding error.

> Coming from the company that has gifted the world the scourge
> that is HTML email,

Niggle: Outlook Express added HTML email in response to that
"innovation" in Netscape Navigator Gold 3.0.

--
Bill Cole
[hidden email] or [hidden email]
(AKA @grumpybozo and many *@billmail.scconsult.com addresses)
Currently Seeking Steady Work: https://linkedin.com/in/billcole
Reply | Threaded
Open this post in threaded view
|

Re: Microsoft silently discarding emails after recepit

stroller-6

> On 9 Jan 2018, at 15:25, Bill Cole <[hidden email]> wrote:
>
> This sort of mail -- a PDF attached to a brief text message -- describes >90% of the spam that has made it through my spam filters in the past 6 months, …
>
> … the most common sorts of FPs are cases of mail not having the form and content of typical business (or personal) one-to-one email. Like it or not, "rich" content in email is pervasive these days so doing it in odd ways like embedded PDFs with  short text parts looks suspect.

I accept what you say about receiving a lot of spam of this format, but I speculate that spammers use it because it's so commonly used among normal people.

You're right, in that I often get PDFs from my solicitor with only a paragraph or so of preamble, "please see attached", in the body of the email.

But I also often print to PDF in my browser when I buy something online (from an unfamiliar supplier) - my mum emails me PDFs of her theatre ticket bookings to print out for her, last week I bought something on eBay and emailed the seller a PDF of the booking confirmation from the courier company I sent for it.

PDFs are a convenient format because they'll always display the same on any system. On a Mac they're easy to create from any application that prints, and display more quickly than a Word document - completely natively as part of the message in Mail.app.

Stroller.
Reply | Threaded
Open this post in threaded view
|

Re: Microsoft silently discarding emails after recepit

Yuval Levy
In reply to this post by Bill Cole-3
On 2018-01-09 10:25 AM, Bill Cole wrote:
> A better place for this discussion would be the MailOps list, where a
> broader variety of mail admins *INCLUDING MS EMPLOYEES* take part, and
> this problem class has been discussed multiple times. See
> https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop

Thanks for the pointer.  I may join when I will have time to engage.


> With that said, a couple of paragraphs stand out as demanding response.
> They literally made me choke on my coffee, repeatedly:

Sorry for the coffee, I hope you did not get scolded, stained, or
keyboard spilled.  Happy to offer you a replacement coffee over a
reasoned discussion of the issues.


> This is why operations like DocuSign & Greenvelope
> exist. Email is a terrible medium for exchange of legal docs, but that's
> a tough sale to lawyers...

I am a lawyer and I am all ears.  I am listening to my clients and using
the medium they choose to communicate.  I've seen some DocuSign use here
and there.  Not sure how is that better than email since after all it is
an email with a link to click, so the whole issue of legitimacy,
authenticity, etc. is just displaced.  In the end, all we need is a
single-purpose tool to reliably send a sequence of ones and zeroes
between the different parties.  Wether that tool is HTTPS or SMTP+TLS or
some other form of electronic signaling, I am agnostic.


>> The only minor objectionable issue I find with my email is
>> the GPG signatures: I sign with a key that is not associated with any
>> email address, contrary to RFC_dont_remember_which,
>
> And totally against the whole design of the OpenPGP signing protocol.
> You might as well use a key claiming to be from someone else. This is
> WORSE than not signing at all. It screams "THIS EMAIL IS FRAUDULENT!!!"

The OpenPGP signing protocol that has miserably failed achieving any
level of significance outside of specialized IT engineering circles for
twenty years?  The protocol that mixes up identity with address with
authentication with authorization?  Happy to continue this discussion in
the appropriate forum.  Open to be persuaded that for a decade RFC4880
has been hiding the solution to the faults of RFC1991 and RFC2440.  I
remember reading them back then, but I do not have my notes of the time.
 Bottom line: I live at an address, I am not that address.  I can live
at multiple addresses, with multiple people, and the way the OpenPGP
signing protocol connects keys to email addresses is a bad abstraction
with many bad consequences.


>> and I sign with an expired key
>
> So it looks like you're using a key that you have in the past said that
> you wouldn't use at this date. This is practically begging to be
> distrusted.

Separate content from transport.  In analogy to snail mail: separate
letter from envelope.  To trust or distrust the content is a job for the
recipient or its post-delivery filter, not for the MTA.  Input from the
post-delivery filter or post-delivery user interaction may feed back and
inform the receiving MTA's future rejections (i.e. the reputation of the
sending MTA).  An MTA should only accept or reject.  If it accepts, it
must deliver.

Once delivered, open the envelope and analyze the letter.  That's where
you can decide to trust or distrust.  Sort into inbox or junkbox.  If
the recipient wants to risk of losing messages, feel free to drop them.
However, not silently drop them.  Leave a message, a textual notice to
the recipient with the empty envelope.

Fax machines are still wildly popular because there is a proper protocol
in place and when the sender receives an OK, it can rely on the
certainty that the message has been received.  If the recipient's dog
eats the paper on the other end of the transmission, it is a liability
for the recipient, not for the sender.


>> (my bad: good enough for my purpose and therefore very low
>> priority to fix).
>
> This free-floating assertion seems at odds with the fact that your email
> to Microsoft customers is being treated in a manner that only makes
> sense for phishing or malware email.

I can explain the appearance if you are interested.  No contradiction.


> Have you tried taking the 5 minutes required to set up a correct GPG key
> and use that instead?

As you can see at
<http://pgp.mit.edu/pks/lookup?search=Yuval+Levy&op=index>, not in a
long time.

Because if you look at
<http://pgp.mit.edu/pks/lookup?search=Bill+Cole&op=index>, you see how
spambots can harvest email addresses from key servers.  Yay, who needs
namespace mining?  Plus, the resulting list is very targeted: highly
intelligent people who take the time to set up GPG keys.  A spammer's
dream.  Some other interesting information can be gleaned that may not
be intended to be public.  I live at an address, I have multiple
addresses, I do not need the public to know any of them nor to link all
of them with my person/identity/activity.

The PGP protocol is almost as bad as using a fingerprint sensor, or
facial recognition, to unlock your phone.


> It is not a bad idea to reject email with a general format that is
> widely used by scammers, discussing activities often discussed in such
> scams, bearing a doubly bogus "signature."
>
> In fact, that strikes me as a very GOOD idea.

Continuing your line of thoughts, maybe it is an equally good idea to
reject SMTP because it is so widely used by scammers?  If I look at my
server's statistics, over 90% of messages are rejected (thanks
Spamhaus).  Let's apply Pareto-efficiency and get rid of the remaining 10%.

Scammers will inevitably mimic the legitimate protocols and the
legitimate formats.  The way to deal with them is not make the
legitimate protocols and formats unusable.  The way to deal with them is
to increase their cost of participation to the federated protocols.
Easier said than done, I admit.


>> Coming from the company that has gifted the world the scourge
>> that is HTML email,
>
> Niggle: Outlook Express added HTML email in response to that
> "innovation" in Netscape Navigator Gold 3.0.

Have you ever observed the adoption of new ideas?  Think committees at
work.  The first mover takes a risk, and it depends on whether the move
is seconded or not that the first mover is then seen as the fool that
came up with the rejected idea, or the visionary that came up with the
next big thing.  Similar in everything.  The mouse may have been
invented at Xerox, but its adoption was driven by others -- and even
though it was seconded by Apple, I'd argue that it is Microsoft that
made it truly popular.

Yuv

12