Suggestion: make compilation fail if m4 is not installed

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
17 messages Options
Reply | Threaded
Open this post in threaded view
|

Suggestion: make compilation fail if m4 is not installed

John Fawcett
Hi

this is a strange error. I just had the chance to reinstall postfix from
latest snapshot source on a new centos 7 server.

When running postconf -n I was getting various errors relating to unused
parameters from mysql map configuration files. This is an example of the
message:

postconf: warning: mysql:/etc/postfix/mysql-domains.cf: unused
parameter: user=xxxxx

Such messages were repeated for various parameters. These same
configuration files work fine on another server (running the same
version of postfix). I realized it was something to do with the
compilation process. I have to admit that I never pay much attention to
the detailed output of the compilation as long as it ends in success,
but going back over the output I found several of these warnings:

extract_cfg.sh: line 74: m4: command not found

With this clue I noticed that extract_cfg.sh was not extracting all the
keywords that could be used in mysql map type configuration files due to
the fact I had forgotten to install m4.

I was wondering if it might be wise to make m4 a mandatory prerequisite,
without which compilation would fail. 

John

Reply | Threaded
Open this post in threaded view
|

Re: Suggestion: make compilation fail if m4 is not installed

Wietse Venema
John Fawcett:

> Hi
>
> this is a strange error. I just had the chance to reinstall postfix from
> latest snapshot source on a new centos 7 server.
>
> When running postconf -n I was getting various errors relating to unused
> parameters from mysql map configuration files. This is an example of the
> message:
>
> postconf: warning: mysql:/etc/postfix/mysql-domains.cf: unused
> parameter: user=xxxxx
>
> Such messages were repeated for various parameters. These same
> configuration files work fine on another server (running the same
> version of postfix). I realized it was something to do with the
> compilation process. I have to admit that I never pay much attention to
> the detailed output of the compilation as long as it ends in success,
> but going back over the output I found several of these warnings:
>
> extract_cfg.sh: line 74: m4: command not found
>
> With this clue I noticed that extract_cfg.sh was not extracting all the
> keywords that could be used in mysql map type configuration files due to
> the fact I had forgotten to install m4.
>
> I was wondering if it might be wise to make m4 a mandatory prerequisite,
> without which compilation would fail.?

We could add this at the top of the script:

    m4 </dev/null || exit 1

but that would be a point solution. For a dependency list someone
would have to write a tool that scrapes the scripts and enumerates
all their dependencies, and that can be (re)run with each release.
I am not inclined to curate the dependency list by hand.

I find it sad that systems have make but not m4. That tool has been
around as part of the UNIX toolkit for over 30 years, on all the
systems that I have played with.

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: Suggestion: make compilation fail if m4 is not installed

@lbutlr
In reply to this post by John Fawcett
On 2018-02-24 (09:44 MST), John Fawcett <[hidden email]> wrote:
>
> due to the fact I had forgotten to install m4.

Interesting. m4 is part of my base install. At least I think it is, it's not in /usr/local/... and i don't remember installing it.

--
Everywhere is walking distance if you have the time.

Reply | Threaded
Open this post in threaded view
|

Re: Suggestion: make compilation fail if m4 is not installed

Bastian Blank-3
In reply to this post by Wietse Venema
On Sat, Feb 24, 2018 at 07:18:32PM -0500, Wietse Venema wrote:
> We could add this at the top of the script:
>     m4 </dev/null || exit 1

Using "set -e" is much more effective and finds all errors.

> I find it sad that systems have make but not m4. That tool has been
> around as part of the UNIX toolkit for over 30 years, on all the
> systems that I have played with.

| $ dpkg -l m4
| dpkg-query: no packages found matching m4

Bastian

--
Hailing frequencies open, Captain.
Reply | Threaded
Open this post in threaded view
|

Re: Suggestion: make compilation fail if m4 is not installed

John Fawcett
In reply to this post by @lbutlr
On 25/02/18 03:09, @lbutlr wrote:
> On 2018-02-24 (09:44 MST), John Fawcett <[hidden email]> wrote:
>> due to the fact I had forgotten to install m4.
> Interesting. m4 is part of my base install. At least I think it is, it's not in /usr/local/... and i don't remember installing it.
>
In this case I didn't chose the initial package list, but typically I
install from the lastest minimal ISO (currently
CentOS-7-x86_64-Minimal-1708.iso) and that doesn't contain m4.

John

Reply | Threaded
Open this post in threaded view
|

Re: Suggestion: make compilation fail if m4 is not installed

John Fawcett
In reply to this post by Wietse Venema
On 25/02/18 01:18, Wietse Venema wrote:

> John Fawcett:
>> ....
>> I was wondering if it might be wise to make m4 a mandatory prerequisite,
>> without which compilation would fail.?
> We could add this at the top of the script:
>
>     m4 </dev/null || exit 1
>
> but that would be a point solution. For a dependency list someone
> would have to write a tool that scrapes the scripts and enumerates
> all their dependencies, and that can be (re)run with each release.
> I am not inclined to curate the dependency list by hand.

That would also be useful to assist in documenting dependencies.

>
> I find it sad that systems have make but not m4. That tool has been
> around as part of the UNIX toolkit for over 30 years, on all the
> systems that I have played with.
>
> Wietse

I normally start out with the minimal set of packages and only add
software that I need to those. While make was already there, the
compiler and m4 were missing. I installed the compiler but forgot m4, so
it was my fault really.

John

Reply | Threaded
Open this post in threaded view
|

Re: Suggestion: make compilation fail if m4 is not installed

@lbutlr
On 2018-02-25 (00:37 MST), John Fawcett <[hidden email]> wrote:

>
> On 25/02/18 01:18, Wietse Venema wrote:
>> John Fawcett:
>>> ....
>>> I was wondering if it might be wise to make m4 a mandatory prerequisite,
>>> without which compilation would fail.?
>> We could add this at the top of the script:
>>
>>    m4 </dev/null || exit 1
>>
>> but that would be a point solution. For a dependency list someone
>> would have to write a tool that scrapes the scripts and enumerates
>> all their dependencies, and that can be (re)run with each release.
>> I am not inclined to curate the dependency list by hand.
>
> That would also be useful to assist in documenting dependencies.
>
>>
>> I find it sad that systems have make but not m4. That tool has been
>> around as part of the UNIX toolkit for over 30 years, on all the
>> systems that I have played with.
>>
>> Wietse
>
> I normally start out with the minimal set of packages and only add
> software that I need to those. While make was already there, the
> compiler and m4 were missing. I installed the compiler but forgot m4, so
> it was my fault really.

I used to do the same back int eh 90s where trying to  slim down the OS was really worth the effort. Now though, a full base install is well under a handful of GB, or less than 1% of a small hard drive.

The package that don't matter are just taking a little bit of space, and it is hardly worth building a system by hand to save a tiny amount (percentage-wise) of space.

--
All he [Vimes] knew was that you couldn't hope to try for the big stuff,
like world peace and happiness, but you might just about be able to
achieve some tiny deed that'd make the world, in a small way, a better
place. Like shooting someone.

Reply | Threaded
Open this post in threaded view
|

Re: Suggestion: make compilation fail if m4 is not installed

Peter Ajamian
In reply to this post by Bastian Blank-3
On 25/02/18 19:27, Bastian Blank wrote:
> On Sat, Feb 24, 2018 at 07:18:32PM -0500, Wietse Venema wrote:
>> We could add this at the top of the script:
>>     m4 </dev/null || exit 1
>
> Using "set -e" is much more effective and finds all errors.

Please no.  set -e is an attempt to cover all error cases without
bailing on commands that return non-zero but aren't errors, therefore it
has all sorts of crazy rules about what is and isn't an error and much
of the time will not do what you think it does.  The safe way is to use
|| exit 1 as Wietse said above.  See:

http://mywiki.wooledge.org/BashFAQ/105


Peter
Reply | Threaded
Open this post in threaded view
|

Re: Suggestion: make compilation fail if m4 is not installed

John Fawcett
On 25/02/18 09:07, Peter wrote:

> On 25/02/18 19:27, Bastian Blank wrote:
>> On Sat, Feb 24, 2018 at 07:18:32PM -0500, Wietse Venema wrote:
>>> We could add this at the top of the script:
>>>     m4 </dev/null || exit 1
>> Using "set -e" is much more effective and finds all errors.
> Please no.  set -e is an attempt to cover all error cases without
> bailing on commands that return non-zero but aren't errors, therefore it
> has all sorts of crazy rules about what is and isn't an error and much
> of the time will not do what you think it does.  The safe way is to use
> || exit 1 as Wietse said above.  See:
>
> http://mywiki.wooledge.org/BashFAQ/105
>
>
> Peter

I tested both approaches on the case in hand.

set -e does not help, the compilation gives a warning message but
proceeds like the current situation.

Wietse's version stops the compilation in its tracks and so draws your
attention to something that should be addressed.

John

Reply | Threaded
Open this post in threaded view
|

Re: Suggestion: make compilation fail if m4 is not installed

John Fawcett
In reply to this post by @lbutlr
On 25/02/18 08:54, @lbutlr wrote:
> On 2018-02-25 (00:37 MST), John Fawcett <[hidden email]> wrote:
>> I normally start out with the minimal set of packages and only add
>> software that I need to those. While make was already there, the
>> compiler and m4 were missing. I installed the compiler but forgot m4, so
>> it was my fault really.
> I used to do the same back int eh 90s where trying to  slim down the OS was really worth the effort. Now though, a full base install is well under a handful of GB, or less than 1% of a small hard drive.
>
> The package that don't matter are just taking a little bit of space, and it is hardly worth building a system by hand to save a tiny amount (percentage-wise) of space.
>
It's now become a habit, but the main reason behind it in my case was
not to save space, but to increase security (admittedly probably only
marginally). The idea is that with less software there will be a smaller
surface of attack and potentially less vectors that can be used by a
malicious user or attacker. This is one of the reasons you might not
want to run a compile environment on the production system.

John

Reply | Threaded
Open this post in threaded view
|

Re: Suggestion: make compilation fail if m4 is not installed

John Fawcett
In reply to this post by Wietse Venema
On 25/02/18 01:18, Wietse Venema wrote:

> John Fawcett:
>> I was wondering if it might be wise to make m4 a mandatory prerequisite,
>> without which compilation would fail.?
> We could add this at the top of the script:
>
>     m4 </dev/null || exit 1
>
> but that would be a point solution. For a dependency list someone
> would have to write a tool that scrapes the scripts and enumerates
> all their dependencies, and that can be (re)run with each release.
> I am not inclined to curate the dependency list by hand.
>
Some thoughts on the possibilty to automate dependency tracking.

There are quite a few scripts in postfix and not all are used during the
build process or will be appropriate to every build. To extract the
dependencies to a common place and test for them could mean that more
dependencies are imposed than actually needed.

This is aside from the difficulty of extracting all the commands from
the files and then filtering out what are builtin shell functions not
external commands.

I'd go with one of these approaches:

1) the shell scripts used in the build process test for their external
dependencies and exit with an error if not satisfied. If that test is
done in a common way, then it can be grepped at least for documentation
purposes.

2) add the dependencies to the makefiles.

John

Reply | Threaded
Open this post in threaded view
|

Re: Suggestion: make compilation fail if m4 is not installed

Bill Cole-3
In reply to this post by @lbutlr
On 25 Feb 2018, at 2:54 (-0500), @lbutlr wrote:

> I used to do the same back int eh 90s where trying to  slim down the
> OS was really worth the effort. Now though, a full base install is
> well under a handful of GB, or less than 1% of a small hard drive.

There is also the independent issue of security. For many systems where
you have 'minimal' and 'full' options and maybe various in-betweens,
lots of that extra stuff you'll never use isn't just eating space, it
has a vanilla default configuration and enabled running services. This
means you have an attack surface whose scale and diversity adds to how
much you need to do to keep it safe without adding anything you actually
use.

> The package that don't matter are just taking a little bit of space,
> and it is hardly worth building a system by hand to save a tiny amount
> (percentage-wise) of space.

But storage footprint is re-emerging as an issue with the rise of
"cloud" systems like AWS that bill for storage in fine granularity.
Sure, it is difficult to find a new physical machine with less than
256GB of SSD or 1TB of spinning rust these days, but if you're running
virtual machines on someone else's hardware with that sort of root
storage you are almost surely Doing It Wrong and bleeding money
pointlessly.


--
Bill Cole
[hidden email] or [hidden email]
(AKA @grumpybozo and many *@billmail.scconsult.com addresses)
Currently Seeking Steady Work: https://linkedin.com/in/billcole
Reply | Threaded
Open this post in threaded view
|

Re: Suggestion: make compilation fail if m4 is not installed

@lbutlr

On 2018-02-25 (20:17 MST), Bill Cole <[hidden email]> wrote:
>
> On 25 Feb 2018, at 2:54 (-0500), @lbutlr wrote:
>
>> I used to do the same back int eh 90s where trying to  slim down the OS was really worth the effort. Now though, a full base install is well under a handful of GB, or less than 1% of a small hard drive.
>
> There is also the independent issue of security. For many systems where you have 'minimal' and 'full' options and maybe various in-betweens, lots of that extra stuff you'll never use isn't just eating space, it has a vanilla default configuration and enabled running services. This means you have an attack surface whose scale and diversity adds to how much you need to do to keep it safe without adding anything you actually use.

Really? What runs services automatically? The last time I setup freeeBSD 11.1 (last month) it wasn't even running sshd until I specifically enabled it.

>> The package that don't matter are just taking a little bit of space, and it is hardly worth building a system by hand to save a tiny amount (percentage-wise) of space.
>
> But storage footprint is re-emerging as an issue with the rise of "cloud" systems like AWS that bill for storage in fine granularity. Sure, it is difficult to find a new physical machine with less than 256GB of SSD or 1TB of spinning rust these days, but if you're running virtual machines on someone else's hardware with that sort of root storage you are almost surely Doing It Wrong and bleeding money pointlessly.

By far the largest part of my base install is the ports tree, which is larger than the rest of the system combined.

Excluding /usr/ /var/ and /mnt

2.0M    /bin
 91M    /boot
3.5K    /dev
4.0K    /entropy
2.9M    /etc
8.6M    /lib
156K    /libexec
4.0K    /media
4.0K    /net
4.0K    /proc
7.8M    /rescue
6.3M    /root
5.9M    /sbin
  0B    /sys
116K    /tmp
  0B    /www
125M    total

(Ports is 4.5GB)

The size of a sever is nearly all "user" data. Mail, web, databases, files for users. The underlying base system needed to just provide these services/data is tiny.

--
Lady Astor: "If you were my husband I'd give you poison." Churchill: "If
you were my wife, I'd drink it."

Reply | Threaded
Open this post in threaded view
|

Re: Suggestion: make compilation fail if m4 is not installed

Stephen Satchell
In reply to this post by Bill Cole-3
On 02/25/2018 07:17 PM, Bill Cole wrote:

>> The package that don't matter are just taking a little bit of space,
>> and it is hardly worth building a system by hand to save a tiny amount
>> (percentage-wise) of space.
>
> But storage footprint is re-emerging as an issue with the rise of
> "cloud" systems like AWS that bill for storage in fine granularity.
> Sure, it is difficult to find a new physical machine with less than
> 256GB of SSD or 1TB of spinning rust these days, but if you're running
> virtual machines on someone else's hardware with that sort of root
> storage you are almost surely Doing It Wrong and bleeding money
> pointlessly.

Storage footprint is also back in the sysadmin's eye when deploying
equipment on very small boxes or blades with SSDs implemented in M.2
form factor.

Further, deleting unnecessary packages reduces the effort in performing
security audits, particularly for "sensitive" computers.  The fewer
things you have to examine, the faster the audit goes.

Any process requiring M4, or whatever, should check for it FIRST, to
minimize the need to pick out a failure in the middle of a log.  To do
otherwise could be easily classed as "astonishing" behavior.
Reply | Threaded
Open this post in threaded view
|

Re: Suggestion: make compilation fail if m4 is not installed

Bill Cole-3
In reply to this post by @lbutlr
On 26 Feb 2018, at 0:52, @lbutlr wrote:

> Really? What runs services automatically?

The whole RedHat family, even Fedora Core through last month. SLES &
openSUSE at least through 11.1. Whatever the current stable Ubuntu was
about a year ago.

> The last time I setup freeeBSD 11.1 (last month) it wasn't even
> running sshd until I specifically enabled it.

Yes, FreeBSD has always held the line on what's in 'base.' Linux distros
not so much.
Reply | Threaded
Open this post in threaded view
|

Re: Suggestion: make compilation fail if m4 is not installed

Stephen Satchell
In reply to this post by @lbutlr
On 02/25/2018 09:52 PM, @lbutlr wrote:
> Really? What runs services automatically? The last time I setup freeeBSD 11.1 (last month) it wasn't even running sshd until I specifically enabled it.

There are other distributions of POSIX-compliant operating systems.
(Let's forego the religious war about *BSD, please.)  Depending on the
options selected at install time, those options may indeed launch an
outside-facing service, such as DNS, or use Remote Procedure Call
protocol (RPC).

That means that a sysadmin new to a particular distribution may not be
aware of the defaults built into the installer, or where there is a
variety of options the defaults for each installer option.

In my $DAYJOB I used CentOS, and sometimes am astonished what Red Hat
thinks is "important".  Mint has a different set of "why?" services it
installs by default.

By starting with the absolute minimum system install, and adding only
what is absolutely necessary, the sysadmin keeps the astonishment factor
to a minimum.

(There I go, using the word "astonish" again.)

I would suspect that a large number of PostFix admins are running
systems on hardware where disk space is not an issue, and the firewall
blocks the worst of the resulting attack service.  Defense in depth says
that you block the socially inconsiderate in multiple places, and not
put your faith solely in firewalls and such.

"Place Not Your Faith in an Ace Kicker." -- _Number of the Beast_,
Robert Heinlein, 1980
Reply | Threaded
Open this post in threaded view
|

Re: Suggestion: make compilation fail if m4 is not installed

Wietse Venema
Folks, if you build Postfix from source code, then it needs UNIX
tools including a compiler and various scripting languages.

Such a configuration is fundamentally in conflict with the idea of
running a minimal system.

I have added a check for a missing 'm4' command, but I do not expect
to enumerate all the utilities that Postfix requires.


        Wietse