Seperate maps for virtual domains?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Seperate maps for virtual domains?

Julian Kippels
Hi,

would it be faster to have several smaller files for alias_maps and
transport_maps for each virtual domain, or have one giant file each with
all users domain from all virtual domains in one file? Around 90% of
traffic is for one domain and the rest is split among 32 other domain.

Thanks
Julian
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Seperate maps for virtual domains?

Patrick Ben Koetter-2
* Julian Kippels <[hidden email]>:
> would it be faster to have several smaller files for alias_maps and
> transport_maps for each virtual domain, or have one giant file each with
> all users domain from all virtual domains in one file? Around 90% of
> traffic is for one domain and the rest is split among 32 other domain.

Hard to tell. If they are static, binary maps Postfix will read them all into
memory and work with the in memory copies. So you don't gain any speed
improvements from a giant file.

It might be more comofortable to work with many files (one per domain) or
better to organize.

p@rick

--
[*] sys4 AG
 
https://sys4.de, +49 (89) 30 90 46 64
Schleißheimer Straße 26/MG,80333 München
 
Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263
Vorstand: Patrick Ben Koetter, Marc Schiffbauer, Wolfgang Stief
Aufsichtsratsvorsitzender: Florian Kirstein
 
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Seperate maps for virtual domains?

Viktor Dukhovni
On Mon, Jun 12, 2017 at 10:32:18AM +0200, Patrick Ben Koetter wrote:

> * Julian Kippels <[hidden email]>:
> > would it be faster to have several smaller files for alias_maps and
> > transport_maps for each virtual domain, or have one giant file each with
> > all users domain from all virtual domains in one file? Around 90% of
> > traffic is for one domain and the rest is split among 32 other domain.
>
> Hard to tell. If they are static, binary maps Postfix will read them all into
> memory and work with the in memory copies. So you don't gain any speed
> improvements from a giant file.

A single CDB, LMDB or Berkeley DB file is much more efficient than
multiple smaller files.

--
        Viktor.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Seperate maps for virtual domains?

Patrick Ben Koetter-2
* Viktor Dukhovni <[hidden email]>:

> On Mon, Jun 12, 2017 at 10:32:18AM +0200, Patrick Ben Koetter wrote:
>
> > * Julian Kippels <[hidden email]>:
> > > would it be faster to have several smaller files for alias_maps and
> > > transport_maps for each virtual domain, or have one giant file each with
> > > all users domain from all virtual domains in one file? Around 90% of
> > > traffic is for one domain and the rest is split among 32 other domain.
> >
> > Hard to tell. If they are static, binary maps Postfix will read them all into
> > memory and work with the in memory copies. So you don't gain any speed
> > improvements from a giant file.
>
> A single CDB, LMDB or Berkeley DB file is much more efficient than
> multiple smaller files.

At which message throughput rate will this make a difference?

p@rick

--
[*] sys4 AG
 
https://sys4.de, +49 (89) 30 90 46 64
Schleißheimer Straße 26/MG,80333 München
 
Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263
Vorstand: Patrick Ben Koetter, Marc Schiffbauer, Wolfgang Stief
Aufsichtsratsvorsitzender: Florian Kirstein
 
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Seperate maps for virtual domains?

Wietse Venema
Patrick Ben Koetter:

> * Viktor Dukhovni <[hidden email]>:
> > On Mon, Jun 12, 2017 at 10:32:18AM +0200, Patrick Ben Koetter wrote:
> >
> > > * Julian Kippels <[hidden email]>:
> > > > would it be faster to have several smaller files for alias_maps and
> > > > transport_maps for each virtual domain, or have one giant file each with
> > > > all users domain from all virtual domains in one file? Around 90% of
> > > > traffic is for one domain and the rest is split among 32 other domain.
> > >
> > > Hard to tell. If they are static, binary maps Postfix will read them all into
> > > memory and work with the in memory copies. So you don't gain any speed
> > > improvements from a giant file.
> >
> > A single CDB, LMDB or Berkeley DB file is much more efficient than
> > multiple smaller files.
>
> At which message throughput rate will this make a difference?

Always. Because you're replacing hashing with linear search.

        Wietse
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Seperate maps for virtual domains?

Patrick Ben Koetter-2
* Wietse Venema <[hidden email]>:

> Patrick Ben Koetter:
> > * Viktor Dukhovni <[hidden email]>:
> > > On Mon, Jun 12, 2017 at 10:32:18AM +0200, Patrick Ben Koetter wrote:
> > >
> > > > * Julian Kippels <[hidden email]>:
> > > > > would it be faster to have several smaller files for alias_maps and
> > > > > transport_maps for each virtual domain, or have one giant file each with
> > > > > all users domain from all virtual domains in one file? Around 90% of
> > > > > traffic is for one domain and the rest is split among 32 other domain.
> > > >
> > > > Hard to tell. If they are static, binary maps Postfix will read them all into
> > > > memory and work with the in memory copies. So you don't gain any speed
> > > > improvements from a giant file.
> > >
> > > A single CDB, LMDB or Berkeley DB file is much more efficient than
> > > multiple smaller files.
> >
> > At which message throughput rate will this make a difference?
>
> Always. Because you're replacing hashing with linear search.

If you compare hashing to linear search, yes. But I am not sure this is what
the OPs question was about?

He wrote "would it be faster to have several smaller files (...) or have one
giant file". The way I understood it, he would not compare hashing vs. linear
search, but many small(er) hashed maps vs. one large hashed map.

I understood the latter and that's why I came up with the question of "message
throughput rate". The goal I am heading for is: If someone runs a platform at
x msg/sec and x is below the threshold where message throughput rate sinks
because of "too many small maps" why bother. Stick with many small maps if you
gain any other advantage until then.

p@rick


--
[*] sys4 AG
 
https://sys4.de, +49 (89) 30 90 46 64
Schleißheimer Straße 26/MG,80333 München
 
Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263
Vorstand: Patrick Ben Koetter, Marc Schiffbauer, Wolfgang Stief
Aufsichtsratsvorsitzender: Florian Kirstein
 
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Seperate maps for virtual domains?

Wietse Venema
Patrick Ben Koetter:

> > > At which message throughput rate will this make a difference?
> >
> > Always. Because you're replacing hashing with linear search.
>
> If you compare hashing to linear search, yes. But I am not sure this is what
> the OPs question was about?
>
> He wrote "would it be faster to have several smaller files (...) or have one
> giant file". The way I understood it, he would not compare hashing vs. linear
> search, but many small(er) hashed maps vs. one large hashed map.

You are doing N/2 table lookups to find the table that contains the
data. That is, you're doing linear search on top of hashing.

        Wietse
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Seperate maps for virtual domains?

Julian Kippels
Am Tue, 13 Jun 2017 10:17:53 -0400 (EDT)
schrieb [hidden email] (Wietse Venema):

> Patrick Ben Koetter:
> > > > At which message throughput rate will this make a difference?  
> > >
> > > Always. Because you're replacing hashing with linear search.  
> >
> > If you compare hashing to linear search, yes. But I am not sure
> > this is what the OPs question was about?
> >
> > He wrote "would it be faster to have several smaller files (...) or
> > have one giant file". The way I understood it, he would not compare
> > hashing vs. linear search, but many small(er) hashed maps vs. one
> > large hashed map.  
>
@Patrick: You understood me correctly there

> You are doing N/2 table lookups to find the table that contains the
> data. That is, you're doing linear search on top of hashing.
>
> Wietse
@Wietse: N/2 is a little pessimistic, but thats only because I know the
makeup of my mailboxes… 90% of it is in one virtual domain and that
would of course be the first file I take a look at. The other files
would be sorted descending by relevance.
Anyway, I am going for a single big file now. I assume its not a problem
that this map is 40MB big?

Julian


--
---------------------------------------------------------
| | Julian Kippels
| | M.Sc. Informatik
| |
| | Zentrum für Informations- und Medientechnologie
| | Heinrich-Heine-Universität Düsseldorf
| | Universitätsstr. 1
| | Raum 25.41.O1.36
| | 40225 Düsseldorf / Germany
| |
| | Tel: +49-211-811-4920
| | mail: [hidden email]
| | jabber: [hidden email]
---------------------------------------------------------

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Seperate maps for virtual domains?

Wietse Venema
Julian Kippels:
> > You are doing N/2 table lookups to find the table that contains the
> > data. That is, you're doing linear search on top of hashing.
> >
> @Wietse: N/2 is a little pessimistic, but thats only because I know the
> makeup of my mailboxes? 90% of it is in one virtual domain and that

The point is that you're slower than one table, whether it's N/2
or N/somethingelse.

> Anyway, I am going for a single big file now. I assume its not a problem
> that this map is 40MB big?

With a competent key-value store implementation, reading from a
40MB table should be no problem.

        Wietse
Loading...