Possible inconsistencies in the parsing of lookup table names and other oddities

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Possible inconsistencies in the parsing of lookup table names and other oddities

Fulvio Scapin
Hello to everyone.

I've been trying to write a small parser to parse Postfix lookup table calls for a piece of code I am developing.

I have taken a look at the source code and then resorted to postmap -q invocations to empirically test the descriptions at http://www.postfix.org/DATABASE_README.html .

I am writing since testing out possible table values for the inline table has left me a bit stumped, since the results I get do no seem to match the description I read.
Or, at least in a few cases, the description seems to mislead about what's actually accepted.

Firstly, I haven't found around the Postfix documentation (and here I hope that somebody could point me in the right direction) how to use comma «,» characters in keys or values; I assume it's not allowed but I haven't located any document where that's stated clearly, except for a few peculiar places such as the definition for inline tables.

The same goes for whitespace, for instance if one wanted to use a filename with ASCII spaces in it as a table name for, for example, pcre tables.

But, as I mentioned at the beginning, the inline table is the one leaving me slightly confused.

The definition for the inline table reads:

«
  A non-shared, in-memory lookup table.
  Example: "inline:{ key=value, { key = text with whitespace or comma }}".
  Key-value pairs are separated by whitespace or comma; with a key-value pair inside "{}", whitespace is ignored after the opening "{", around the "=" between key and value, and before the closing "}".
  Inline tables eliminate the need to create a database file for just a few fixed elements. See also the static: map type
»

Below a few examples generated through postmap & tee & bash.

--

# echo key | postmap -q - $'inline:{ key=value\,   }' | tee >( cat --show-all )
key     value\
key^Ivalue\$

So, commas and whitespace after the first «{» and before the last «}» are ignored, which wasn't specified in the description.

--

# echo key\{ | postmap -q - $'inline:{ key{=value\,   }' | tee >( cat --show-all )
postmap: fatal: bad syntax: "inline:{ key{=value\,   }"; need "inline:{name=value...}"

# echo key\} | postmap -q - $'inline:{ key}=value\,   }' | tee >( cat --show-all )
postmap: fatal: bad syntax: "inline:{ key}=value\,   }"; need "inline:{name=value...}"

# echo key\} | postmap -q - $'inline:{ key}=value}\,   }' | tee >( cat --show-all )
postmap: fatal: bad syntax: "inline:{ key}=value}\,   }"; need "inline:{name=value...}"


Ok, apparently no «{» or «}» characters in the key or value ?

# echo a\{\{akey | postmap -q - $'inline:{ a{{akey ={}}}   }' | tee >( cat --show-all )
a{{akey {}}}
a{{akey^I{}}}$

No, apparently it's fine if the braces are balanced???


# echo a\{\{akey | postmap -q - $'inline:{ aakey        ={}}}a }' | tee >( cat --show-all )
postmap: fatal: bad syntax: "inline:{ aakey        ={}}}a }"; need "inline:{name=value...}"

Ok that's consistent...

# echo a\{\{akey | postmap -q - $'inline:{ a{{akey        ={}}}a }' | tee >( cat --show-all )
a{{akey {}}}a
a{{akey^I{}}}a$

But why is this working?? Just because there are balanced braces at some point in the key and/or value?

I hope I haven't missed something obvious.


A last question, somewhat unrelated, is about this line from postconf(5)

«A logical line starts with non-whitespace text. A line that starts with whitespace continues a logical line.»

Does that mean that two lines in main.cf such as

2bounce_notice_recipient = post
 master


would be equivalent to

2bounce_notice_recipient = postmaster

ignoring the whitespace after the soft line break?


Thanks in advance for any answer or pointer.

Regards,
Fulvio Scapin
Reply | Threaded
Open this post in threaded view
|

Re: Possible inconsistencies in the parsing of lookup table names and other oddities

Wietse Venema
Fulvio Scapin:

> Hello to everyone.
>
> I've been trying to write a small parser to parse Postfix lookup table
> calls for a piece of code I am developing.
>
> I have taken a look at the source code and then resorted to postmap -q
> invocations to empirically test the descriptions at
> http://www.postfix.org/DATABASE_README.html .
>
> I am writing since testing out possible table values for the inline table
> has left me a bit stumped, since the results I get do no seem to match the
> description I read.
> Or, at least in a few cases, the description seems to mislead about what's
> actually accepted.
>
> Firstly, I haven't found around the Postfix documentation (and here I hope
> that somebody could point me in the right direction) how to use comma ?,?
> characters in keys or values; I assume it's not allowed but I haven't
> located any document where that's stated clearly, except for a few peculiar
> places such as the definition for inline tables.

You cannot use commas or spaces in (lookup table) pathnames where,
as documented, Postfix expects a comma-and-or-space separated list.

Apart from that, Postfix is written for UNIX systems. It would be
a mistake to use spaces (or other shell special characters) in
pathnames.

        Wietse

Reply | Threaded
Open this post in threaded view
|

Re: Possible inconsistencies in the parsing of lookup table names and other oddities

Viktor Dukhovni
In reply to this post by Fulvio Scapin
On Sat, Jul 27, 2019 at 03:13:01PM +0200, Fulvio Scapin wrote:

> But, as I mentioned at the beginning, the inline table is the one leaving
> me slightly confused.

The syntax is *simple* (minimal features, which is not always the
same as: intuitive or "easy to use", but generally not too far off
the mark).  Misunderstandings arise when one wishfully hopes for
or imagines features of the syntax that are not there.

>   A non-shared, in-memory lookup table.
>   Example: "inline:{ key=value, { key = text with whitespace or comma }}".
>
>   Key-value pairs are separated by whitespace or comma; with a key-value
> pair inside "{}", whitespace is ignored after the opening "{", around the
> "=" between key and value, and before the closing "}".
>   Inline tables eliminate the need to create a database file for just a few
> fixed elements. See also the static: map type

The only thing that's "missing" from the description is that the
braces have to match, this is because one can generally list multiple
tables:

        virtual_alias_maps = inline:{...}, cdb:..., inline:{...}

and for that to work, there needs to be a sensible algorithm for
finding theclosing brace of the first "inline".  But since there
is no support for literal "{" or "}" (i.e. no backslash or similar
"escape" characters are mentioned or implemented), and yet some
key-value pairs are enclosed in "{", "}", the braces must match.

> Below a few examples generated through postmap & tee & bash.

All the examples behave as expected given the general syntax.  Yes,
an escape-character would make it possible to use non-matching
braces in the key and value.  That's not presently possible.

--
        Viktor.
Reply | Threaded
Open this post in threaded view
|

Re: Possible inconsistencies in the parsing of lookup table names and other oddities

Fulvio Scapin
Thanks for the answer Viktor.

As I mentioned, mine was an attempt to parse/validate the syntax externally, so I used the documentation as a guideline and probed with postmap to see what it would actually accept or not.

I hope you will forgive me if I disagree on a specific case where I believe the parser not to behave properly.
This assuming we take the documentation as something which should match the actual behaviour.
It might be relatively pointless in a practical situation and/or nitpicking/hair splitting but anyway...

So, the description reads...
> Key-value pairs are separated by whitespace or comma; with a key-value
pair inside "{}", whitespace is ignored after the opening "{", around the
"=" between key and value, and before the closing "}".

Nothing here says that you cannot use braces as characters within a key or value.
So I could have "aa{bb" as a key and "cc}dd" as value. The braces are balanced so no problem there.

Now, according to the documentation whitespace is ignored around the «=» character only when the key-value pair is surrounded by braces.

So inline:{ key=value} is fine but inline:{ key = value } isn't.

So back to a slightly altered version of my example.

# echo key | postmap -q - $'inline:{ key        =value }' | tee >( cat --show-all )            
postmap: fatal: missing '=' after attribute name: "inline:{ key        =value }"; need "inline:{name=value...}"

This should be right since I can't have whitespace between «key» and «=a».

However when I throw braces inside «key» and «value»...

# echo ke\{y | postmap -q - $'inline:{ ke{y        =va}lue }' | tee >( cat --show-all )
ke{y    va}lue
ke{y^Iva}lue$

Now whitespace before «=» IS allowed, even though the key-value combination is not «inside {}» as the documentation says.
In fact «key{y» is accepted as key and «va}lue» is returned as matching.
The braces are just part of the key and of the value.
Am I missing something?

Again, I realize it's a pretty theoretical exercise but I am curious.
And also, though it may just be a personal preference, I prefer definitions explicitely stated rather than implied when a format is provided, even when the implied part is pretty obvious.
After all you said, unless I misunderstood, that there is no support for literal braces (by which I believe you mean that you cannot willfully escape them) and yet they are accepted as such if you place them at a specific place instead of another.

As I said at the beginning, I frankly find this to be a rather specific corner case of very little practical significance.
But I do believe the necessity of balanced braces and the bizarre situation with them when within keys and values, if not something to correct in the parser, could be documented (for instance for people translating from other tables to inline tables, where the braces in keys/values are just literal characters with no particular meaning).
It could prevent people from getting stumped for a time due to a trivial problem.

Regardless, as for me, I'll just balance the braces to find the end of inline table definitions and skip the validation of their contents for now in my little parser.

Again, thanks for the answer.

Fulvio
Reply | Threaded
Open this post in threaded view
|

Re: Possible inconsistencies in the parsing of lookup table names and other oddities

Wietse Venema
Fulvio Scapin:

> Thanks for the answer Viktor.
>
> As I mentioned, mine was an attempt to parse/validate the syntax
> externally, so I used the documentation as a guideline and probed with
> postmap to see what it would actually accept or not.
>
> I hope you will forgive me if I disagree on a specific case where I believe
> the parser not to behave properly.
> This assuming we take the documentation as something which should match the
> actual behaviour.

We weem to have a discrepancy of expectations.

Postfix documentation is not written up after some discovery process
of how the code works. It's the other way around.

Postfix documentation promises what the implementation will do. It
promises that if you use { and } in the documented manner, then the
implementation will behave as promised. Other uses of { and } are
not covered by the promise, and instead, daemons may fly out of
your nose.

        Wietse