mbox format?

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

mbox format?

Ronald F. Guilmette-2
Apparently, and much to my surprise, there is more than one mbox format.

I just now stumbled across this, because I am going to be (re-)writing
some small tools I have that do useful things with mail messages stored
in "mbox format":

    https://en.wikipedia.org/wiki/Mbox

In the above Wikipedia page, four different flavors of "mbox format"
are described: mboxo, mboxrd, mboxcl, and mboxcl2.

When Postfix hands a message to something... say a script invoked via
some ~/.forward file... which one of these four formats will the message
be in?
Reply | Threaded
Open this post in threaded view
|

Re: mbox format?

Viktor Dukhovni
On Wed, Jun 26, 2019 at 12:39:02PM -0700, Ronald F. Guilmette wrote:

> When Postfix hands a message to something... say a script invoked via
> some ~/.forward file... which one of these four formats will the message
> be in?

See: https://github.com/vdukhovni/postfix/blob/master/postfix/src/global/mail_copy.c#L235

So the answer is mboxo.

--
        Viktor.
Reply | Threaded
Open this post in threaded view
|

Re: mbox format?

Wietse Venema
Viktor Dukhovni:
> On Wed, Jun 26, 2019 at 12:39:02PM -0700, Ronald F. Guilmette wrote:
>
> > When Postfix hands a message to something... say a script invoked via
> > some ~/.forward file... which one of these four formats will the message
> > be in?
>
> See: https://github.com/vdukhovni/postfix/blob/master/postfix/src/global/mail_copy.c#L235
>
> So the answer is mboxo.

According to 'man 8 local', section 'EXTERNAL COMMAND DELIVERY':
       The local(8) daemon prepends a "From sender time_stamp" envelope header
       to each message, prepends an X-Original-To: header with  the  recipient
       address  as given to Postfix, prepends an optional Delivered-To: header
       with the final recipient  envelope  address,  prepends  a  Return-Path:
       header with the sender envelope address, and appends no empty line.

This seems to match the implementation.

In local/command.c:

    copy_flags = MAIL_COPY_FROM | MAIL_COPY_RETURN_PATH
        | MAIL_COPY_ORIG_RCPT;
    if (local_deliver_hdr_mask & DELIVER_HDR_CMD)
        copy_flags |= MAIL_COPY_DELIVERED;

In global/mail_copy.h:

#define MAIL_COPY_FROM (1<<2) /* prepend From_ */
#define MAIL_COPY_DELIVERED (1<<3) /* prepend Delivered-To: */
#define MAIL_COPY_RETURN_PATH (1<<4) /* prepend Return-Path: */
#define MAIL_COPY_ORIG_RCPT (1<<7) /* prepend X-Original-To: */

Note: this does not prepend '>' to lines starting with 'From '.

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: mbox format?

Viktor Dukhovni
On Wed, Jun 26, 2019 at 04:27:06PM -0400, Wietse Venema wrote:

> Viktor Dukhovni:
>
> > On Wed, Jun 26, 2019 at 12:39:02PM -0700, Ronald F. Guilmette wrote:
> >
> > > When Postfix hands a message to something... say a script invoked via
> > > some ~/.forward file... which one of these four formats will the message
> > > be in?
> >
> > See: https://github.com/vdukhovni/postfix/blob/master/postfix/src/global/mail_copy.c#L235
> >
> > So the answer is mboxo.
>
> According to 'man 8 local', section 'EXTERNAL COMMAND DELIVERY':
>        The local(8) daemon prepends a "From sender time_stamp" envelope header
>        to each message, prepends an X-Original-To: header with  the  recipient
>        address  as given to Postfix, prepends an optional Delivered-To: header
>        with the final recipient  envelope  address,  prepends  a  Return-Path:
>        header with the sender envelope address, and appends no empty line.

Oops, sorry, my answer was about delivery by local(8) directly to a mailbox,
not a command invoked via a pipe.

--
        Viktor.
Reply | Threaded
Open this post in threaded view
|

Re: mbox format?

Ronald F. Guilmette-2
In reply to this post by Viktor Dukhovni
In message <[hidden email]>,
Viktor Dukhovni <[hidden email]> wrote:

>On Wed, Jun 26, 2019 at 12:39:02PM -0700, Ronald F. Guilmette wrote:
>
>> When Postfix hands a message to something... say a script invoked via
>> some ~/.forward file... which one of these four formats will the message
>> be in?
>
>See: https://github.com/vdukhovni/postfix/blob/master/postfix/src/global/mail_copy.c#L235
>
>So the answer is mboxo.

Thanks Viktor.

I realize now that I should have asked one additional question:  When Postfix
-stores- a message, for example, in some user's /var/mail/<<user>> file,
what flavor of mbox is used in that case?

Reply | Threaded
Open this post in threaded view
|

Re: mbox format?

Viktor Dukhovni
On Wed, Jun 26, 2019 at 02:00:53PM -0700, Ronald F. Guilmette wrote:

> In message <[hidden email]>,
> Viktor Dukhovni <[hidden email]> wrote:
>
> >On Wed, Jun 26, 2019 at 12:39:02PM -0700, Ronald F. Guilmette wrote:
> >
> >> When Postfix hands a message to something... say a script invoked via
> >> some ~/.forward file... which one of these four formats will the message
> >> be in?
> >
> >See: https://github.com/vdukhovni/postfix/blob/master/postfix/src/global/mail_copy.c#L235
> >
> >So the answer is mboxo.
>
> I realize now that I should have asked one additional question:  When Postfix
> -stores- a message, for example, in some user's /var/mail/<<user>> file,
> what flavor of mbox is used in that case?

Turns out I assumed that's what you were asking.  So you already
have that answer.  For pipes in the aliases or .forward files, you
get what Wietse said.

--
        Viktor.
Reply | Threaded
Open this post in threaded view
|

Re: mbox format?

Ronald F. Guilmette-2
In reply to this post by Wietse Venema
In message <[hidden email]>,
 Wietse Venema <[hidden email]> wrote:

>According to 'man 8 local', section 'EXTERNAL COMMAND DELIVERY':
>       The local(8) daemon prepends a "From sender time_stamp" envelope header
>       to each message, prepends an X-Original-To: header with  the  recipient
>       address  as given to Postfix, prepends an optional Delivered-To: header
>       with the final recipient  envelope  address,  prepends  a  Return-Path:
>       header with the sender envelope address, and appends no empty line.
>...
>Note: this does not prepend '>' to lines starting with 'From '.

ACK.

And I see now that the behavior of local(8) when delivering to files is
also nicely and clearly documented... as I should have assumed.  (My bad!
Next time I'll RTFM before asking.)


Regards,
rfg
Reply | Threaded
Open this post in threaded view
|

Re: mbox format?

Andrey Repin-2
In reply to this post by Ronald F. Guilmette-2
Greetings, Ronald F. Guilmette!

> Apparently, and much to my surprise, there is more than one mbox format.

> I just now stumbled across this, because I am going to be (re-)writing
> some small tools I have that do useful things with mail messages stored
> in "mbox format":

>     https://en.wikipedia.org/wiki/Mbox

> In the above Wikipedia page, four different flavors of "mbox format"
> are described: mboxo, mboxrd, mboxcl, and mboxcl2.

> When Postfix hands a message to something... say a script invoked via
> some ~/.forward file... which one of these four formats will the message
> be in?

Guess why I'm using Maildir?


--
With best regards,
Andrey Repin
Friday, June 28, 2019 14:28:30

Sorry for my terrible english...

Reply | Threaded
Open this post in threaded view
|

Re: mbox format?

@lbutlr
On Jun 28, 2019, at 5:28 AM, Andrey Repin <[hidden email]> wrote:
> Guess why I'm using Maildir?

Because mbox was designed for a tens of Kilobytes of email?



--
What's another word for Thesaurus?


Reply | Threaded
Open this post in threaded view
|

Re: mbox format?

Andrey Repin-2
Greetings, @lbutlr!

>> Guess why I'm using Maildir?

> Because mbox was designed for a tens of Kilobytes of email?

I have doubts it was at all designed. More like thrown together for the sake
of plugging the hole of mail storage.

On a sidenote, I've seen a variation of mbox format that uses an FF (0x0C)
character as message delimiter, and do not do any body encoding.


--
With best regards,
Andrey Repin
Friday, June 28, 2019 15:34:04

Sorry for my terrible english...

Reply | Threaded
Open this post in threaded view
|

Re: mbox format?

Wietse Venema
Andrey Repin:
> Greetings, @lbutlr!
>
> >> Guess why I'm using Maildir?
>
> > Because mbox was designed for a tens of Kilobytes of email?
>
> I have doubts it was at all designed. More like thrown together for the sake
> of plugging the hole of mail storage.

Allow me to set the record straight.

mbox goes back 40+ years, when computers were a lot smaller (1 MB
memory, 10 MB disk) and file systems could not handle thousands of
files per directory without becoming intolerably slow. UNIX wasn't
the only system with monolithic mailbox files; VMS had them for
similar reasons.

For an early UNIX implementation from 1979, see
https://minnie.tuhs.org/cgi-bin/utree.pl?file=V7/usr/src/cmd/mail.c 

As for the format, in the spirit of UNIX 40+ years ago, it was made
no more complex than was needed for the job.

For more about that spirit:
http://dreamsongs.com/RiseOfWorseIsBetter.html

        Wietse

> On a sidenote, I've seen a variation of mbox format that uses an FF (0x0C)
> character as message delimiter, and do not do any body encoding.
>
>
> --
> With best regards,
> Andrey Repin
> Friday, June 28, 2019 15:34:04
>
> Sorry for my terrible english...
>
>
Reply | Threaded
Open this post in threaded view
|

Re: mbox format?

@lbutlr
In reply to this post by Andrey Repin-2
On Jun 28, 2019, at 6:39 AM, Andrey Repin <[hidden email]> wrote:
> Greetings, @lbutlr!
>
>>> Guess why I'm using Maildir?
>
>> Because mbox was designed for a tens of Kilobytes of email?
>
> I have doubts it was at all designed. More like thrown together for the sake of plugging the hole of mail storage.

To be fair, email used to be a lot smaller, systems used to be a lot smaller. Even as late as 1992 when I got a backup of my entire home folder on a single 9 track tape my 5+ years of email and all my files added up to less that 100MB or so (the capacity of a 9-Track).

But then again, we didn’t have HTML or even MIME that I recall back then, and if we had MIME it was suspiciously new.

> On a sidenote, I've seen a variation of mbox format that uses an FF (0x0C)
> character as message delimiter, and do not do any body encoding.

That is MMDF. I think the first M stands for mutt? I’ve never seen one of these files in use.

Mbox was fine for what it was intended for, which was never single files in the 100s of MB much less in the GBs. There are many better mailbox formats now.

I use Maildir which I like because I can still dither with it easily on the command line, but it’s by no means the best one out there, and large servers do not use it.

(Well, really no one uses milder anymore, but rather maildir++ but that is a technically with nearly no meaning to anyone).

Dovecot implemented dbox in several flavors, but I really don’t know anything about it or how prevalent it is. I hear it is much better than maildir for large user bases, but it is only supported by dovecot. Maildir is widely supported (postfix, exam, courier, mail drop, even proemial!).

For a while I fought with Apple’s Mail.app which used MH, but now I do all my mail tweaking on the server and leave Mail.app alone. I am not sure if it is still using MH or not.

Exim uses (used to use?) mailstore by default. I remember running exim when I switched from UW-IMAP to Courier briefly long ago and quickly switching to postfix, but I don’t remember what it was about exim I disliked.


--
No sense being pessimistic. It wouldn't work anyway.