empty message-ID

[natan]

Hi
I have probably to trivial questions about message-ID

Why sometimes, some user have empty message-id=<>

example:
Nov 23 13:13:53 smtp1 postfix/submission/smtpd[29867]: 4CfmKF1CSDz5MwK:
xxxx.domain.ltd[xxx.xxx.xxx.xxx], sasl_method=login,
sasl_username=[hidden email]
Nov 23 13:13:53 smtp1 postfix/cleanup[46909]: 4CfmKF1CSDz5MwK: message-id=<>
Nov 23 13:13:53 smtp1 postfix/qmgr[25287]: 4CfmKF1CSDz5MwK:
from=<[hidden email]>, size=94874, nrcpt=3 (queue active)
.....



--

[Matus UHLAR - fantomas]

On 23.11.20 14:35, natan wrote:

>I have probably to trivial questions about message-ID
>
>Why sometimes, some user have empty message-id=<>
>
>example:
>Nov 23 13:13:53 smtp1 postfix/submission/smtpd[29867]: 4CfmKF1CSDz5MwK:
>xxxx.domain.ltd[xxx.xxx.xxx.xxx], sasl_method=login,
>sasl_username=[hidden email]
>Nov 23 13:13:53 smtp1 postfix/cleanup[46909]: 4CfmKF1CSDz5MwK: message-id=<>
>Nov 23 13:13:53 smtp1 postfix/qmgr[25287]: 4CfmKF1CSDz5MwK:
>from=<[hidden email]>, size=94874, nrcpt=3 (queue active)
>.....

client's MUA apparently does not generate Message-Id: header.

--
Matus UHLAR - fantomas, [hidden email] ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Boost your system's speed by 500% - DEL C:\WINDOWS\*.*

[mami64]

Hi
Thanks for replay I found "RFC 822 Message-ID is not required"
Probably "problem" is in configurations in some clients.

On 23.11.2020 14:39, Matus UHLAR - fantomas wrote:

> On 23.11.20 14:35, natan wrote:
>> I have probably to trivial questions about message-ID
>>
>> Why sometimes, some user have empty message-id=<>
>>
>> example:
>> Nov 23 13:13:53 smtp1 postfix/submission/smtpd[29867]: 4CfmKF1CSDz5MwK:
>> xxxx.domain.ltd[xxx.xxx.xxx.xxx], sasl_method=login,
>> sasl_username=[hidden email]
>> Nov 23 13:13:53 smtp1 postfix/cleanup[46909]: 4CfmKF1CSDz5MwK:
>> message-id=<>
>> Nov 23 13:13:53 smtp1 postfix/qmgr[25287]: 4CfmKF1CSDz5MwK:
>> from=<[hidden email]>, size=94874, nrcpt=3 (queue active)
>> .....
>
> client's MUA apparently does not generate Message-Id: header.
>

[D'Arcy Cain]

On 11/23/20 9:49 AM, maciejm wrote:
> Hi
> Thanks for replay I found "RFC 822 Message-ID is not required"
> Probably "problem" is in configurations in some clients.

I used to have a client who was not getting emails from one of his friends.
  Turned out that the friend's client/MUA was not adding the message ID.
After the first message was accepted all of the rest were silently dropped
as duplicates due to a very standard procmail recipe:

:0 Wh: msgid.lock
| formail -D 65536 $HOME/.msgid.cache

In other words, the message ID "" was considered a duplicate after the first
one.

--
D'Arcy J.M. Cain
System Administrator, Vex.Net
http://www.Vex.Net/ IM:[hidden email]
VoIP: sip:[hidden email]

[Jaroslaw Rafa]

Dnia 23.11.2020 o godz. 10:18:39 D'Arcy Cain pisze:
>
> I used to have a client who was not getting emails from one of his
> friends.  Turned out that the friend's client/MUA was not adding the
> message ID.

Doesn't Postfix automatically add Message-Id: header upon sending a message
if none is present?

> After the first message was accepted all of the rest
> were silently dropped as duplicates due to a very standard procmail
> recipe:
>
> :0 Wh: msgid.lock
> | formail -D 65536 $HOME/.msgid.cache

Who uses that? It's not normal to get email duplicates and it usually
means that mail system is not functioning properly. They should find the
cause of the duplicates and eliminate it instead of hiding symptoms...
--
Regards,
   Jaroslaw Rafa
   [hidden email]
--
"In a million years, when kids go to school, they're gonna know: once there
was a Hushpuppy, and she lived with her daddy in the Bathtub."

[Phil Stracchino]

On 11/23/20 9:18 AM, D'Arcy Cain wrote:

> On 11/23/20 9:49 AM, maciejm wrote:
>> Hi
>> Thanks for replay I found "RFC 822 Message-ID is not required"
>> Probably "problem" is in configurations in some clients.
>
> I used to have a client who was not getting emails from one of his friends.
>   Turned out that the friend's client/MUA was not adding the message ID.
> After the first message was accepted all of the rest were silently dropped
> as duplicates due to a very standard procmail recipe:
>
> :0 Wh: msgid.lock
> | formail -D 65536 $HOME/.msgid.cache
>
> In other words, the message ID "" was considered a duplicate after the first
> one.

The lesson from which is, don't make message delivery dependent upon an
optional header.


--
  Phil Stracchino
  Babylon Communications
  [hidden email]
  [hidden email]
  Landline: +1.603.293.8485
  Mobile:   [phone]

[Wietse Venema]

Jaroslaw Rafa:
> Dnia 23.11.2020 o godz. 10:18:39 D'Arcy Cain pisze:
> >
> > I used to have a client who was not getting emails from one of his
> > friends.  Turned out that the friend's client/MUA was not adding the
> > message ID.
>
> Doesn't Postfix automatically add Message-Id: header upon sending a message
> if none is present?

For the last 17 years, Message-ID is added to "local" email only.

http://www.postfix.org/postconf.5.html#always_add_missing_headers

> > After the first message was accepted all of the rest
> > were silently dropped as duplicates due to a very standard procmail
> > recipe:
> >
> > :0 Wh: msgid.lock
> > | formail -D 65536 $HOME/.msgid.cache

They should skip that when email has no Message-Id: header.

        Wietse

[D'Arcy Cain]

On 11/23/20 10:44 AM, Jaroslaw Rafa wrote:
> Dnia 23.11.2020 o godz. 10:18:39 D'Arcy Cain pisze:
>>
>> I used to have a client who was not getting emails from one of his
>> friends.  Turned out that the friend's client/MUA was not adding the
>> message ID.
>
> Doesn't Postfix automatically add Message-Id: header upon sending a message
> if none is present?

I guess they weren't using Postfix.

>> After the first message was accepted all of the rest
>> were silently dropped as duplicates due to a very standard procmail
>> recipe:
>>
>> :0 Wh: msgid.lock
>> | formail -D 65536 $HOME/.msgid.cache
>
> Who uses that? It's not normal to get email duplicates and it usually
> means that mail system is not functioning properly. They should find the
> cause of the duplicates and eliminate it instead of hiding symptoms...

If someone replies to a mailing list and copies the sender then that person
gets two copies.  The above recipe avoids that.

People also send to every alias that someone has.  Example, billing@,
admin@, support@, joe@, etc.

--
D'Arcy J.M. Cain
System Administrator, Vex.Net
http://www.Vex.Net/ IM:[hidden email]
VoIP: sip:[hidden email]

[Benny Pedersen-2]

D'Arcy Cain skrev den 2020-11-23 15:18:

> :0 Wh: msgid.lock
> | formail -D 65536 $HOME/.msgid.cache
>
> In other words, the message ID "" was considered a duplicate after the
> first one.

if you use postfix there would be uniq msgid always, eq postfix ensures
there is always fqdn in msgid aswell, many mua's does not ensure the
fqdn part

[Wietse Venema]

Benny Pedersen:

> D'Arcy Cain skrev den 2020-11-23 15:18:
>
> > :0 Wh: msgid.lock
> > | formail -D 65536 $HOME/.msgid.cache
> >
> > In other words, the message ID "" was considered a duplicate after the
> > first one.
>
> if you use postfix there would be uniq msgid always, eq postfix ensures
> there is always fqdn in msgid aswell, many mua's does not ensure the
> fqdn part

Postfix 2.6 and later don't add a Message-ID header, unless the
message comes from a "local" source. That header is a combination
of a time stamp and the Postfix $myhostname value, so it is unique
as long as both values are unique.

        Wietse

[Benny Pedersen-2]

Wietse Venema skrev den 2020-11-23 17:10:

> Postfix 2.6 and later don't add a Message-ID header, unless the
> message comes from a "local" source. That header is a combination
> of a time stamp and the Postfix $myhostname value, so it is unique
> as long as both values are unique.

okay, what if msgid miss the @ charter ?

i remember postfix have configs for when @ is not part of msgid it can
add @invalid.example.org so its diffrent if not sent local or remote

[Wietse Venema]

Benny Pedersen:
> Wietse Venema skrev den 2020-11-23 17:10:
>
> > Postfix 2.6 and later don't add a Message-ID header, unless the
> > message comes from a "local" source. That header is a combination
> > of a time stamp and the Postfix $myhostname value, so it is unique
> > as long as both values are unique.
>
> okay, what if msgid miss the @ charter ?

A message-id is not an email address.

> i remember postfix have configs for when @ is not part of msgid it can
> add @invalid.example.org so its diffrent if not sent local or remote

Postfix can be configured to 'fix' an email address.

        Wietse

remote_header_rewrite_domain (default: empty)
       Don't rewrite message headers from remote  clients  at  all  when  this
       parameter  is  empty; otherwise, rewrite message headers and append the
       specified domain name to incomplete  addresses.   The  local_header_re-
       write_clients parameter controls what clients Postfix considers local.

       Examples:

       The   safe   setting:  append  "domain.invalid"  to  incomplete  header
       addresses from remote SMTP clients, so that those addresses  cannot  be
       confused with local addresses.

           remote_header_rewrite_domain = domain.invalid

       The default, purist, setting: don't rewrite headers from remote clients
       at all.

           remote_header_rewrite_domain =


        Wietse

On 23 Nov 2020, at 06:49, maciejm <[hidden email]> wrote:
> "RFC 822 Message-ID is not required"

RFC 822 has been obsoleted several times.

RFC 5322 states:

   Though listed as optional in the table in section 3.6, every message
   SHOULD have a "Message-ID:" field.  Furthermore, reply messages
   SHOULD have "In-Reply-To:" and "References:" fields as appropriate
   and as described below.

And:

RFC 2119
SHOULD This word, or the adjective "RECOMMENDED", mean that there
        may exist valid reasons in particular circumstances to ignore a
        particular item, but the full implications must be understood and
        carefully weighed before choosing a different course.

So SHOULD is much stronger than "it's a good idea" and much more like "You better have a really good reason for ignoring this".

I would feel comfortable rejecting messages without a Message-ID.

--
Imagine all the people Sharing all the world

On 23 Nov 2020, at 07:44, Jaroslaw Rafa <[hidden email]> wrote:
> Dnia 23.11.2020 o godz. 10:18:39 D'Arcy Cain pisze:
>>
>> :0 Wh: msgid.lock
>> | formail -D 65536 $HOME/.msgid.cache
>
> Who uses that?

Everyone who ever used procmail? Nearly everyone who ever used procmail?

It's even in the procmail man page.

>        If you are subscribed to several mailinglists and people cross-post to
>        some of them, you usually receive several duplicate mails (one from
>        every list).  The following simple recipe eliminates duplicate mails.
>        It tells formail to keep an 8KB cache file in which it will store the
>        Message-IDs of the most recent mails you received.  Since Message-IDs
>        are guaranteed to be unique for every new mail, they are ideally suited
>        to weed out duplicate mails.  Simply put the following recipe at the
>        top of your rcfile, and no duplicate mail will get past it.
>
>               :0 Wh: msgid.lock
>               | formail -D 8192 msgid.cache

--
All our loves are first loves

[Bob Proulx]

Jaroslaw Rafa wrote:

> Dnia 23.11.2020 o godz. 10:18:39 D'Arcy Cain pisze:
> > After the first message was accepted all of the rest
> > were silently dropped as duplicates due to a very standard procmail
> > recipe:
> >
> > :0 Wh: msgid.lock
> > | formail -D 65536 $HOME/.msgid.cache
>
> Who uses that? It's not normal to get email duplicates and it usually
> means that mail system is not functioning properly. They should find the
> cause of the duplicates and eliminate it instead of hiding symptoms...

Although I have been using procmail since the inception of it I have
always found this type rule problematic.  Because for me it keeps the
wrong message.  If I am sent a direct copy and a mailing list copy
then the copy I want is the mailing list copy.

But so many people use Gmail these days that they have gotten used to
the way Gmail does things.  And Gmail de-duplicates and saves the
first message with any particular message-id that arrives.  And then
displays a "mailbox" showing a view of the current tag being
displayed.  It's a very different paradigm from having separate
mailbox folders for different topics.  Gmail has one mailbox for
everything and multiple tags are possible on each message and only
displays the current display tag view of the mailbox.  And since it is
one mailbox it de-duplicates by only showing the first message-id.
And people have gotten used to that paradigm.  But it does cause some
odd behavior when dealing with mailing lists.

Bob

[Erwan David]

Le 23/11/2020 à 20:16, @lbutlr a écrit :

> On 23 Nov 2020, at 06:49, maciejm <[hidden email]> wrote:
>> "RFC 822 Message-ID is not required"
> RFC 822 has been obsoleted several times.
>
> RFC 5322 states:
>
>    Though listed as optional in the table in section 3.6, every message
>    SHOULD have a "Message-ID:" field.  Furthermore, reply messages
>    SHOULD have "In-Reply-To:" and "References:" fields as appropriate
>    and as described below.
>
> And:
>
> RFC 2119
> SHOULD This word, or the adjective "RECOMMENDED", mean that there
> may exist valid reasons in particular circumstances to ignore a
> particular item, but the full implications must be understood and
> carefully weighed before choosing a different course.
>
> So SHOULD is much stronger than "it's a good idea" and much more like "You better have a really good reason for ignoring this".
>
> I would feel comfortable rejecting messages without a Message-ID.
>

Maybe on smtp, but not on submission. FOr me policy there is completeley
different

[Richard Damon]

On 11/23/20 3:34 PM, Erwan David wrote:

> Le 23/11/2020 à 20:16, @lbutlr a écrit :
>> On 23 Nov 2020, at 06:49, maciejm <[hidden email]> wrote:
>>> "RFC 822 Message-ID is not required"
>> RFC 822 has been obsoleted several times.
>>
>> RFC 5322 states:
>>
>>    Though listed as optional in the table in section 3.6, every message
>>    SHOULD have a "Message-ID:" field.  Furthermore, reply messages
>>    SHOULD have "In-Reply-To:" and "References:" fields as appropriate
>>    and as described below.
>>
>> And:
>>
>> RFC 2119
>> SHOULD This word, or the adjective "RECOMMENDED", mean that there
>> may exist valid reasons in particular circumstances to ignore a
>> particular item, but the full implications must be understood and
>> carefully weighed before choosing a different course.
>>
>> So SHOULD is much stronger than "it's a good idea" and much more like "You better have a really good reason for ignoring this".
>>
>> I would feel comfortable rejecting messages without a Message-ID.
>>
> Maybe on smtp, but not on submission. FOr me policy there is completeley
> different

I thought one strategy to handle this was that submission would detect
lack of the message-id header and add one with a proper message-id.

--
Richard Damon

[Jaroslaw Rafa]

Dnia 23.11.2020 o godz. 11:49:39 D'Arcy Cain pisze:
>
> If someone replies to a mailing list and copies the sender then that
> person gets two copies.  The above recipe avoids that.

If someone gets two copies - a direct one and the mailing list one - then
he/she knows that the sender has replied both to author and to the list and
can instruct the sender not to do it. With the above recipe, the recipient
doesn't even know about that.

Moreover, it breaks the continuity of threads on mailing lists, because it's
unpredictable which copy will arrive first, and if only the direct copy is
left, the reply will go only to the sender and not to the mailing list. Thus
some messages are missing from lists.

> People also send to every alias that someone has.  Example,
> billing@, admin@, support@, joe@, etc.

But it's usually one message with multiple recipients, and if all these
recipients "resolve" to the same final destination, the receiving MTA
usually avoids creating duplicates. At least that was always the case for me
as the recipient, no matter if I was using sendmail, Exim or Postfix for my
mail.


Dnia 23.11.2020 o godz. 12:22:37 @lbutlr pisze:
>
> Everyone who ever used procmail? Nearly everyone who ever used procmail?
>
> It's even in the procmail man page.

Yes it is, but I never saw any real use case for this.


Dnia 23.11.2020 o godz. 13:24:13 Bob Proulx pisze:
>
> Although I have been using procmail since the inception of it I have
> always found this type rule problematic.  Because for me it keeps the
> wrong message.  If I am sent a direct copy and a mailing list copy
> then the copy I want is the mailing list copy.
>
> But so many people use Gmail these days that they have gotten used to
> the way Gmail does things.
[...]

+1 ;)
--
Regards,
   Jaroslaw Rafa
   [hidden email]
--
"In a million years, when kids go to school, they're gonna know: once there
was a Hushpuppy, and she lived with her daddy in the Bathtub."

[Richard Damon]

On 11/23/20 5:27 PM, Jaroslaw Rafa wrote:

> Dnia 23.11.2020 o godz. 11:49:39 D'Arcy Cain pisze:
>> If someone replies to a mailing list and copies the sender then that
>> person gets two copies.  The above recipe avoids that.
> If someone gets two copies - a direct one and the mailing list one - then
> he/she knows that the sender has replied both to author and to the list and
> can instruct the sender not to do it. With the above recipe, the recipient
> doesn't even know about that.
>
> Moreover, it breaks the continuity of threads on mailing lists, because it's
> unpredictable which copy will arrive first, and if only the direct copy is
> left, the reply will go only to the sender and not to the mailing list. Thus
> some messages are missing from lists.

You CAN still reply to the list from the private copy, you won't have a
'Reply-to-List' opiton, because of the lack of list headers, but
'Reply-All' will still work.

It just becomes a bit harder to reply back JUST to the list. Your need
Reply-All and then editing the list of recipients.



--
Richard Damon

On 23 Nov 2020, at 13:34, Erwan David <[hidden email]> wrote:
> Le 23/11/2020 à 20:16, @lbutlr a écrit :
>> I would feel comfortable rejecting messages without a Message-ID.

> Maybe on smtp, but not on submission. FOr me policy there is completeley
> different

On submission postfix adds the message ID as is proper if the MUA hasn't added it.

--
'Now what?' it said. IT'S UP TO YOU. IT'S ALWAYS UP TO YOU.
        --Maskerade