Message-ID

classic Classic list List threaded Threaded
7 messages Options
CP
Reply | Threaded
Open this post in threaded view
|

Message-ID

CP
Hi list,

I'm trying to eliminate a problem with duplicate emails in alias expansion
and I have seen that some (local generated and SMTP also) messages don't
have the
Message-ID header . I'm reading in  in postconf that there is a
configuration
option  always_add_missing_headers that does exactly that.  If I enable
this
option do I get in any risk of braking anything in my existing setup ?
The manual refers  that " this may break DKIM signatures that cover
non-existent headers"
but I'm not sure if that refers to local generated messages or
everything received my SMTP.
I'm not currently using DKIM in my setup.

Thanks for any info

George




Reply | Threaded
Open this post in threaded view
|

Re: Message-ID

Matus UHLAR - fantomas
On 27.01.18 11:16, CP wrote:

>I'm trying to eliminate a problem with duplicate emails in alias expansion
>and I have seen that some (local generated and SMTP also) messages
>don't have the
>Message-ID header . I'm reading in  in postconf that there is a
>configuration
>option  always_add_missing_headers that does exactly that.  If I
>enable this
>option do I get in any risk of braking anything in my existing setup ?
>The manual refers  that " this may break DKIM signatures that cover
>non-existent headers"
>but I'm not sure if that refers to local generated messages or
>everything received my SMTP.
>I'm not currently using DKIM in my setup.

If you receive dkim-signed mail without message-id, but the dkim signature
will claim it contains message-id too (does this happen?), and when you add
it, the message will not pass the check (including your message-id changes
the hash).

this should not be a problem if you aren't resending the mail to outside,
and can be avoided by checking the DKIM before message-id is added and never
later.


--
Matus UHLAR - fantomas, [hidden email] ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Eagles may soar, but weasels don't get sucked into jet engines.
CP
Reply | Threaded
Open this post in threaded view
|

Re: Message-ID

CP
On 01/27/2018 12:09 PM, Matus UHLAR - fantomas wrote:

> On 27.01.18 11:16, CP wrote:
>> I'm trying to eliminate a problem with duplicate emails in alias
>> expansion
>> and I have seen that some (local generated and SMTP also) messages
>> don't have the
>> Message-ID header . I'm reading in  in postconf that there is a
>> configuration
>> option  always_add_missing_headers that does exactly that.  If I
>> enable this
>> option do I get in any risk of braking anything in my existing setup ?
>> The manual refers  that " this may break DKIM signatures that cover
>> non-existent headers"
>> but I'm not sure if that refers to local generated messages or
>> everything received my SMTP.
>> I'm not currently using DKIM in my setup.
>
> If you receive dkim-signed mail without message-id, but the dkim
> signature
> will claim it contains message-id too (does this happen?), and when
> you add
> it, the message will not pass the check (including your message-id
> changes
> the hash).

So far I have only seen messages without the message-id header , is
there some way to
check for what you're describing above in received message headers ?


>
> this should not be a problem if you aren't resending the mail to outside,
> and can be avoided by checking the DKIM before message-id is added and
> never
> later.
>
>

Usually I don't resend the message but users might forward  the message  to
an outside recipient, this is not the case I hope ?
Checking DKIM  is done by postfix ?  any further info or reading about
this ?


Thanks In advance

Reply | Threaded
Open this post in threaded view
|

Re: Message-ID

Matus UHLAR - fantomas
>>On 27.01.18 11:16, CP wrote:
>>>I'm trying to eliminate a problem with duplicate emails in alias
>>>expansion and I have seen that some (local generated and SMTP also)
>>>messages don't have the Message-ID header .

I should add that such duplicity should not happen when using aliases.
Also, it may happen that two people get different copies of the same mail
(e.g. from this list).

Are you sure your duplicities can't be fixed other way?
 
>>>  I'm reading in  in postconf
>>>that there is a configuration option  always_add_missing_headers that
>>>does exactly that.  If I enable this option do I get in any risk of
>>>braking anything in my existing setup ?  The manual refers  that " this
>>>may break DKIM signatures that cover non-existent headers" but I'm not
>>>sure if that refers to local generated messages or everything received my
>>>SMTP.  I'm not currently using DKIM in my setup.

>On 01/27/2018 12:09 PM, Matus UHLAR - fantomas wrote:
>>If you receive dkim-signed mail without message-id, but the dkim signature
>>will claim it contains message-id too (does this happen?), and when you
>>add it, the message will not pass the check (including your message-id
>>changes the hash).

On 27.01.18 13:21, CP wrote:
>So far I have only seen messages without the message-id header , is there
>some way to check for what you're describing above in received message
>headers ?

this is from your mail:

DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
         d=gmail.com; s=20161025;
         h=subject:to:references:from:message-id:date:user-agent:mime-version
         :in-reply-to:content-transfer-encoding:content-language;

you see message-id there.
if such DKIM header includes Message-Id: but message has none, DKIM breakage
may happen when you add it.

>>this should not be a problem if you aren't resending the mail to outside,
>>and can be avoided by checking the DKIM before message-id is added
>>and never
>>later.

>Usually I don't resend the message but users might forward  the message  to
>an outside recipient, this is not the case I hope ?

no.

>Checking DKIM  is done by postfix ?  any further info or reading
>about this ?

as you configure, usually spam filter checks for it.

if you are going to add message-id as postfix level you apparently need
implement before that, DKIM milter probably.

--
Matus UHLAR - fantomas, [hidden email] ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
If Barbie is so popular, why do you have to buy her friends?
CP
Reply | Threaded
Open this post in threaded view
|

Re: Message-ID

CP
On 01/27/2018 02:05 PM, Matus UHLAR - fantomas wrote:

>>> On 27.01.18 11:16, CP wrote:
>>>> I'm trying to eliminate a problem with duplicate emails in alias
>>>> expansion and I have seen that some (local generated and SMTP also)
>>>> messages don't have the Message-ID header .
>
> I should add that such duplicity should not happen when using aliases.
> Also, it may happen that two people get different copies of the same mail
> (e.g. from this list).
>
> Are you sure your duplicities can't be fixed other way?

In what way ? check alias expansion ?

The thing is  I have some groups say group1 group2 and group3

there may be users  that are part of only one group but there are others
that maybe in all groups .
Until I implemented lmtp and duplicate supression from it (dovecot) a
user who was in all 3 groups
would receive the message 3 times (and if send to his own email address
4 !)  The problem persists
even with lmtp and messages that don't have Message-ID in header  ( I
guess  this is the criteria for supressing
duplicates) .

How can I  track down this ?


Thanks  for your time

Reply | Threaded
Open this post in threaded view
|

Re: Message-ID

Karol Augustin
On 2018-01-27 17:24, CP wrote:

> On 01/27/2018 02:05 PM, Matus UHLAR - fantomas wrote:
>>>> On 27.01.18 11:16, CP wrote:
>>>>> I'm trying to eliminate a problem with duplicate emails in alias
>>>>> expansion and I have seen that some (local generated and SMTP also)
>>>>> messages don't have the Message-ID header .
>>
>> I should add that such duplicity should not happen when using aliases. Also, it may happen that two people get different copies of the same mail
>> (e.g. from this list).
>>
>> Are you sure your duplicities can't be fixed other way?
>
> In what way ? check alias expansion ?
>
> The thing is  I have some groups say group1 group2 and group3
>
> there may be users  that are part of only one group but there are
> others that maybe in all groups .
> Until I implemented lmtp and duplicate supression from it (dovecot) a
> user who was in all 3 groups
> would receive the message 3 times (and if send to his own email
> address 4 !)  The problem persists
> even with lmtp and messages that don't have Message-ID in header  ( I
> guess  this is the criteria for supressing
> duplicates) .
>
> How can I  track down this ?
>
>
> Thanks  for your time

Dovecot documentation says: "For more complex applications, the
duplicate test can also use the content of a specific header field or
other parts of the message."
https://wiki.dovecot.org/Pigeonhole/Sieve/Extensions/Duplicate

I couldn't find specific example but maybe dovecot list is a better
place to ask. It might be possible to detect duplicates using different
headers. Anyway just Message-ID might be not a perfect solution.

If I understand your issue correctly I think that I have the same
problem that the user is a part of an alias office@ and he gets
duplicates of all mails send to him and copied to office@.

So @office -> guy1@, @guy2. If the e-mail is addressed to guy1@, office@
then Guy1 gets two copies and Guy2 only one. Often this e-mail is sent
in two different SMTP sessions so I don't see any possibility that
Postfix can catch that.

It's not a big deal so I have never looked into that but now I might.
The problem with using solely Message-ID is that off-list replies will
be matched with list-replies and I don't want that. Don't know if there
are any other possible side-effects of basically discarding same
Message-ID going to the same user in some finite timeframe.

Karol



--
Karol Augustin
[hidden email]
http://karolaugustin.pl/
+353 85 775 5312
Reply | Threaded
Open this post in threaded view
|

Re: Message-ID

Karol Augustin
On 2018-01-27 19:30, Karol Augustin wrote:

> On 2018-01-27 17:24, CP wrote:
>> On 01/27/2018 02:05 PM, Matus UHLAR - fantomas wrote:
>>>>> On 27.01.18 11:16, CP wrote:
>>>>>> I'm trying to eliminate a problem with duplicate emails in alias
>>>>>> expansion and I have seen that some (local generated and SMTP also)
>>>>>> messages don't have the Message-ID header .
>>>
>>> I should add that such duplicity should not happen when using aliases. Also, it may happen that two people get different copies of the same mail
>>> (e.g. from this list).
>>>
>>> Are you sure your duplicities can't be fixed other way?
>>
>> In what way ? check alias expansion ?
>>
>> The thing is  I have some groups say group1 group2 and group3
>>
>> there may be users  that are part of only one group but there are
>> others that maybe in all groups .
>> Until I implemented lmtp and duplicate supression from it (dovecot) a
>> user who was in all 3 groups
>> would receive the message 3 times (and if send to his own email
>> address 4 !)  The problem persists
>> even with lmtp and messages that don't have Message-ID in header  ( I
>> guess  this is the criteria for supressing
>> duplicates) .
>>
>> How can I  track down this ?
>>
>>
>> Thanks  for your time
>
> Dovecot documentation says: "For more complex applications, the
> duplicate test can also use the content of a specific header field or
> other parts of the message."
> https://wiki.dovecot.org/Pigeonhole/Sieve/Extensions/Duplicate
>
> I couldn't find specific example but maybe dovecot list is a better
> place to ask. It might be possible to detect duplicates using different
> headers. Anyway just Message-ID might be not a perfect solution.
>
> If I understand your issue correctly I think that I have the same
> problem that the user is a part of an alias office@ and he gets
> duplicates of all mails send to him and copied to office@.
>
> So @office -> guy1@, @guy2. If the e-mail is addressed to guy1@, office@
> then Guy1 gets two copies and Guy2 only one. Often this e-mail is sent
> in two different SMTP sessions so I don't see any possibility that
> Postfix can catch that.
>
> It's not a big deal so I have never looked into that but now I might.
> The problem with using solely Message-ID is that off-list replies will
> be matched with list-replies and I don't want that. Don't know if there
> are any other possible side-effects of basically discarding same
> Message-ID going to the same user in some finite timeframe.
>
Also in the scenario I described adding Message-ID by postfix won't help
you as I think it will add different message-ids if e-mail arrives in
two SMTP sessions (like from gmail for example).

--
Karol Augustin
[hidden email]
http://karolaugustin.pl/
+353 85 775 5312