Milter: Removing multiple headers with the same name

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Milter: Removing multiple headers with the same name

Vsevolod Stakhov
Hello,

I'm trying to debug one issue with the milter interface in Postfix. I
want to remove multiple headers with the same name, for example:

X-Spam-Flag: No
X-Spam-Flag: No
X-Spam-Flag: Not really
X-Spam-Flag: No

To do that, I track headers on header phase and track numbers for each
header. For this particular sample, I will have some map like
`X-Spam-Flag = 4`.

After end-of-data stage, I send multiple `CHGHEADER` ('m') commands with
header value equal to a empty string:

change/insert header command pos = 1 - "X-Spam-Flag"=""
change/insert header command pos = 2 - "X-Spam-Flag"=""
change/insert header command pos = 3 - "X-Spam-Flag"=""
change/insert header command pos = 4 - "X-Spam-Flag"=""

Literally, my code (Rspamd) sends CHGHEADER command 4 times. After
message being processed, I can see that Postfix removes 2 of 4
`X-Spam-Flag` headers.

I have tested something else after that. I have tried to add more
headers where names are equal and non equal. It seems that all odd
headers are not removed.

For example:

Input:

X-Spam-Flag: Test1
X-Spam-Flag: Test2
X-Spam-Flag: Test3
X-Spam-Flag: Test4

Output:

X-Spam-Flag: Test2
X-Spam-Flag: Test4

Input:

X-Spam-Flag: No
X-Spam-Flag: No
X-Spam-Flag: No
X-Spam-Flag: No
X-Spam-Flag: No
X-Spam-Flag: No

Output:

X-Spam-Flag: No
X-Spam-Flag: No
X-Spam-Flag: No

Am I doing something wrong from my side or it is some issue in the
Postfix? I'm using Postfix 3.1.0 but I can try other MTA or other
Postfix version if there is no visible issue from my side.
Reply | Threaded
Open this post in threaded view
|

Re: Milter: Removing multiple headers with the same name

Bill Cole-3
On 4 Mar 2019, at 6:58, Vsevolod Stakhov wrote:

> Hello,
>
> I'm trying to debug one issue with the milter interface in Postfix. I
> want to remove multiple headers with the same name, for example:
>
> X-Spam-Flag: No
> X-Spam-Flag: No
> X-Spam-Flag: Not really
> X-Spam-Flag: No
>
> To do that, I track headers on header phase and track numbers for each
> header. For this particular sample, I will have some map like
> `X-Spam-Flag = 4`.
>
> After end-of-data stage, I send multiple `CHGHEADER` ('m') commands
> with
> header value equal to a empty string:
>
> change/insert header command pos = 1 - "X-Spam-Flag"=""
> change/insert header command pos = 2 - "X-Spam-Flag"=""
> change/insert header command pos = 3 - "X-Spam-Flag"=""
> change/insert header command pos = 4 - "X-Spam-Flag"=""
>
> Literally, my code (Rspamd) sends CHGHEADER command 4 times. After
> message being processed, I can see that Postfix removes 2 of 4
> `X-Spam-Flag` headers.
>
> I have tested something else after that. I have tried to add more
> headers where names are equal and non equal. It seems that all odd
> headers are not removed.

I have not tested this, but your description is consistent with the
Milter interface acting immediately on your deletion of the indexed
header instance and collapsing the array, modifying which index points
to which data. So, after you delete the 1st "X-Spam-Flag" header, you
now have 3 instances remaining, with the new 1st instance being the
former 2nd instance. This way, if you delete the 2nd instance, you are
deleting what WAS the 3rd.

In any event, the results described also imply that you are not
bothering to check for errors when you call a library routine that CAN
return an error. Again, I have not tested this but I know that chgheader
CAN return an error and would expect it to do so when passed an
out-of-range index.

Return values exist for good reasons. One of them is to help novice
coders avoid public embarrassment.

--
Bill Cole
[hidden email] or [hidden email]
(AKA @grumpybozo and many *@billmail.scconsult.com addresses)
Available For Hire: https://linkedin.com/in/billcole
Reply | Threaded
Open this post in threaded view
|

Re: Milter: Removing multiple headers with the same name

Claus Assmann-22
On Mon, Mar 04, 2019, Bill Cole wrote:

> Again, I have not tested this but I know that chgheader CAN return an error
> and would expect it to do so when passed an out-of-range index.

Did you check the documentation?

> Return values exist for good reasons. One of them is to help novice coders
> avoid public embarrassment.

Documentation exists for a reason too :-)
Reply | Threaded
Open this post in threaded view
|

Re: Milter: Removing multiple headers with the same name

Vsevolod Stakhov
In reply to this post by Bill Cole-3
Bill,

On 04/03/2019 15:11, Bill Cole wrote:

> On 4 Mar 2019, at 6:58, Vsevolod Stakhov wrote:
>
>> Hello,
>>
>> I'm trying to debug one issue with the milter interface in Postfix. I
>> want to remove multiple headers with the same name, for example:
>>
>> X-Spam-Flag: No
>> X-Spam-Flag: No
>> X-Spam-Flag: Not really
>> X-Spam-Flag: No
>>
>> To do that, I track headers on header phase and track numbers for each
>> header. For this particular sample, I will have some map like
>> `X-Spam-Flag = 4`.
>>
>> After end-of-data stage, I send multiple `CHGHEADER` ('m') commands with
>> header value equal to a empty string:
>>
>> change/insert header command pos = 1 - "X-Spam-Flag"=""
>> change/insert header command pos = 2 - "X-Spam-Flag"=""
>> change/insert header command pos = 3 - "X-Spam-Flag"=""
>> change/insert header command pos = 4 - "X-Spam-Flag"=""
>>
>> Literally, my code (Rspamd) sends CHGHEADER command 4 times. After
>> message being processed, I can see that Postfix removes 2 of 4
>> `X-Spam-Flag` headers.
>>
>> I have tested something else after that. I have tried to add more
>> headers where names are equal and non equal. It seems that all odd
>> headers are not removed.
>
> I have not tested this, but your description is consistent with the
> Milter interface acting immediately on your deletion of the indexed
> header instance and collapsing the array, modifying which index points
> to which data. So, after you delete the 1st "X-Spam-Flag" header, you
> now have 3 instances remaining, with the new 1st instance being the
> former 2nd instance. This way, if you delete the 2nd instance, you are
> deleting what WAS the 3rd.

It is not consistent with the Sendmail behaviour:

https://groups.google.com/forum/#!msg/comp.mail.sendmail/Xgl1DE3lTUg/_uaAEmTM8EYJ

For me, it doesn't make any difference to send `1` 4 times instead of
sending `1, 2, 3, 4` sequence. But I'm worried not to break other MTAs
that use milter protocol.

> In any event, the results described also imply that you are not
> bothering to check for errors when you call a library routine that CAN
> return an error.

I don't call any library, I have my own implementation of the milter
protocol. It checks for any errors that could arise but there are no
errors of course, or I would have not asked this question.

> Again, I have not tested this but I know that chgheader
> CAN return an error and would expect it to do so when passed an
> out-of-range index.
>
> Return values exist for good reasons. One of them is to help novice
> coders avoid public embarrassment.

JFYI, calling other people novice coders without knowing the real
background could also lead to a public embarrassment.

Reply | Threaded
Open this post in threaded view
|

Re: Milter: Removing multiple headers with the same name

Vsevolod Stakhov
On 04/03/2019 15:23, Vsevolod Stakhov wrote:

> Bill,
>
> On 04/03/2019 15:11, Bill Cole wrote:
>> On 4 Mar 2019, at 6:58, Vsevolod Stakhov wrote:
>>
>>> Hello,
>>>
>>> I'm trying to debug one issue with the milter interface in Postfix. I
>>> want to remove multiple headers with the same name, for example:
>>>
>>> X-Spam-Flag: No
>>> X-Spam-Flag: No
>>> X-Spam-Flag: Not really
>>> X-Spam-Flag: No
>>>
>>
>> I have not tested this, but your description is consistent with the
>> Milter interface acting immediately on your deletion of the indexed
>> header instance and collapsing the array, modifying which index points
>> to which data. So, after you delete the 1st "X-Spam-Flag" header, you
>> now have 3 instances remaining, with the new 1st instance being the
>> former 2nd instance. This way, if you delete the 2nd instance, you are
>> deleting what WAS the 3rd.
>
> It is not consistent with the Sendmail behaviour:
>
> https://groups.google.com/forum/#!msg/comp.mail.sendmail/Xgl1DE3lTUg/_uaAEmTM8EYJ
>
> For me, it doesn't make any difference to send `1` 4 times instead of
> sending `1, 2, 3, 4` sequence. But I'm worried not to break other MTAs
> that use milter protocol.
>


Ok, I've just realized that the obviously correct solution of this
problem is to delete headers in the reversed order. We can close this
question I suppose.
Reply | Threaded
Open this post in threaded view
|

Re: Milter: Removing multiple headers with the same name

Wietse Venema
In reply to this post by Vsevolod Stakhov
Vsevolod Stakhov:
> I don't call any library, I have my own implementation of the milter
> protocol. It checks for any errors that could arise but there are no
> errors of course, or I would have not asked this question.

Could that be a problem? Before I sink any time into this, does the
problem also exist with implementations based on libmilter (which
I suppose is what almist everything else is using)?

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: Milter: Removing multiple headers with the same name

Vsevolod Stakhov
Wietse,

On 04/03/2019 15:34, Wietse Venema wrote:

> Vsevolod Stakhov:
>> I don't call any library, I have my own implementation of the milter
>> protocol. It checks for any errors that could arise but there are no
>> errors of course, or I would have not asked this question.
>
> Could that be a problem? Before I sink any time into this, does the
> problem also exist with implementations based on libmilter (which
> I suppose is what almist everything else is using)?
>
> Wietse
>


I don't really see that libmilter does anything in a different way when
calling `smfi_chgheader`. The protocol does not expect any errors in
reply to this command and there are lots of undefined pieces indeed. The
only thing it states is 'If hdridx is greater than the number of times
headerf appears, a new copy of headerf is added.' Of course, it leads to
ambiguity when it comes to headers deletion.

In my case, I have found that removing headers in reversed order will
work with whatever milter implementation.

I'm not quite sure about the performance of these approaches: from
reading the code, it seems that all headers modifications are quite
expensive operations whatever order is selected. But I've not dig too
deeply into it I'm afraid.