New to PF, IO bound query

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

New to PF, IO bound query

Eddie b
Hello, Well firstly, I hope my postfix experience is going to better than the pathetic ancient mailing list server that took 3 attempts with gmail in subscriber confirmations...

Anyways... I am looking at replacing our outdated Qmail/ Vpopmail system, I've read the docs and it seems simple enough using MySQL, and it appears using dovecots LDA is preferred method,  great, we use noth right now, the question I have is, we have some virtual domains with up to 90K users, now vpopmail cleverly splits them into no more than 100 users or domains per directory, creating  countless/sub/dirs/of/domansuser/accounts etc etc (I hope you get my drift) it is blindingly fast, as there is next to no IO impact, however nothing I've read about postfix leeds me to believe, it can split these up, like  /var/spool/vmail/domain/part-name/another-part-name/etc/etc/user   to keep the IO efficient, as you can imagine the IO issues we will have if you throw 90K into one single directory, we are intending to do this via mysql input, and then moving the mail over (no problems with that,we've done it before with vpopmail to vpopmail)...

So hoping someone else here has dealt with this many users under a domain and can shed some light..short of writting more perl scripts to count dirs and then increase and create another sub of a sub and so on, we are tryin g to keep this as mess-free and painless as possible :)

Cheers
Ed


Reply | Threaded
Open this post in threaded view
|

Re: New to PF, IO bound query

Sahil Tandon
Eddie b <[hidden email]> wrote:

> Hello, Well firstly, I hope my postfix experience is going to better
> than the pathetic ancient mailing list server that took 3 attempts
> with gmail in subscriber confirmations...

I hope you're not saying the postfix-users mailing list is pathetic; that
would be a silly (not to mention inaccurate) way to start an email asking
for help.
 
If you are in love with vpopmail, then just use it with Postfix.  There
are a few examples on the web on how to do this.

--
Sahil Tandon <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: New to PF, IO bound query

mouss-2
In reply to this post by Eddie b
Eddie b wrote:

> Hello, Well firstly, I hope my postfix experience is going to better than
> the pathetic ancient mailing list server that took 3 attempts with gmail in
> subscriber confirmations...
>
> Anyways... I am looking at replacing our outdated Qmail/ Vpopmail system,
> I've read the docs and it seems simple enough using MySQL, and it appears
> using dovecots LDA is preferred method,  great, we use noth right now, the
> question I have is, we have some virtual domains with up to 90K users, now
> vpopmail cleverly splits them into no more than 100 users or domains per
> directory, creating  countless/sub/dirs/of/domansuser/accounts etc etc (I
> hope you get my drift) it is blindingly fast, as there is next to no IO
> impact, however nothing I've read about postfix leeds me to believe, it can
> split these up, like
> /var/spool/vmail/domain/part-name/another-part-name/etc/etc/user   to keep
> the IO efficient, as you can imagine the IO issues we will have if you throw
> 90K into one single directory, we are intending to do this via mysql input,
> and then moving the mail over (no problems with that,we've done it before
> with vpopmail to vpopmail)...
>
> So hoping someone else here has dealt with this many users under a domain
> and can shed some light..short of writting more perl scripts to count dirs
> and then increase and create another sub of a sub and so on, we are tryin g
> to keep this as mess-free and painless as possible :)
>

Start by playing with postfix on a test machine, until you get confortable.

then configure postfix for your virtual setup:
http://www.postfix.org/VIRTUAL_README.html#virtual_mailbox

once you have it working and delivering to where you want, you can
configure it to deliver with dovecot LDA. see
http://www.postfix.org/VIRTUAL_README.html#in_virtual_other

this requires a working dovecot of course. so check dovecot wiki and ask
on dovecot mailing list if you have dovecot issues.


Postfix is friendly, but you need to read the documentation
        http://www.postfix.org/documentation.html
In particular,
        http://www.postfix.org/BASIC_CONFIGURATION_README.html
        http://www.postfix.org/STANDARD_CONFIGURATION_README.html
        http://www.postfix.org/VIRTUAL_README.html

For troubleshooting, see
        http://www.postfix.org/DEBUG_README.html
In particular, if you want to ask here, check
        http://www.postfix.org/DEBUG_README.html#mail



Reply | Threaded
Open this post in threaded view
|

Re: New to PF, IO bound query

mouss-2
Eddie b wrote:
> [snip]
> I know I can configure it the way I want writing perl scripts to split off,
> using say the first and then second chars of users login name using
> postfix's internal virtual, but using dovecot gives me greater flexibility

dovecot LDA is useful if you want to use its sieve features.

> which I'd like to use ( most of the howto's also recommend using dovecots
> LDA,, but all searches on that point to only  /blah/domain/login , which
> brings me back to my 90K in one directory problem.


you talked about mysql, no?

select concat('/var/Mail/', %d, '/' , substring(%u, 1, 1), '/',
        substring(%u, '1', '2'), '/', %u, '/maildir/');

This will set the mailbox of [hidden email] to
        /var/Mail/example.com/f/fo/foobar/maildir/

you can use this query directly, or create a view to use it. You can
also trade storage for performance by storing the value in a table at
insert time. you can do this with a mysql trigger if you prefer mysql
code, or you can do it with an external script.



> This is why I postred here, hoping a mail admin with this many users in one
> domain and using dovecot has found away to break this up, or perhaps none of
> then have and have had to rely on postfix's virtual LDA instead, or would
> comment on what they found works best.
>
> Ed
>

Reply | Threaded
Open this post in threaded view
|

Re: New to PF, IO bound query

Eddie b


On Sun, Aug 31, 2008 at 2:48 AM, mouss <[hidden email]> wrote:

dovecot LDA is useful if you want to use its sieve features.

Yes we are hoping to , so we can move spam to a junk folder for them and so on.

 

select concat('/var/Mail/', %d, '/' , substring(%u, 1, 1), '/',        
       substring(%u, '1', '2'), '/', %u, '/maildir/');


It doesnt help with dovecot though, as DC's LDA processes what and where and only understands maildir:/some/path/domain/user (AFAIK) so your above example would work with postfix's internal virtual but not (AFAIK) with DC.

 Ed

Reply | Threaded
Open this post in threaded view
|

Re: New to PF, IO bound query

mouss-2
Eddie b wrote:

> On Sun, Aug 31, 2008 at 2:48 AM, mouss <[hidden email]> wrote:
>
>> dovecot LDA is useful if you want to use its sieve features.
>>
>
> Yes we are hoping to , so we can move spam to a junk folder for them and so
> on.
>
>
>>
>> select concat('/var/Mail/', %d, '/' , substring(%u, 1, 1), '/',
>>        substring(%u, '1', '2'), '/', %u, '/maildir/');
>>
>>
> It doesnt help with dovecot though, as DC's LDA processes what and where and
> only understands maildir:/some/path/domain/user (AFAIK) so your above
> example would work with postfix's internal virtual but not (AFAIK) with DC.
>


come on. that was an example. dovecot wants a string and mysql can
generate a string. concat('maildir:...', ...).
Reply | Threaded
Open this post in threaded view
|

Re: New to PF, IO bound query

Eddie b


On Sun, Aug 31, 2008 at 3:33 PM, mouss <[hidden email]> wrote:
Eddie b wrote:

It doesnt help with dovecot though, as DC's LDA processes what and where and
only understands maildir:/some/path/domain/user (AFAIK) so your above
example would work with postfix's internal virtual but not (AFAIK) with DC.



come on. that was an example. dovecot wants a string and mysql can generate a string. concat('maildir:...', ...).

I do stand corrected, I played on our dev box once I returned to work and have it working perfectly as I want it, but before i allow it to go live i need to ensure a few things, you see I'm a 20 year sendmail veteran, and a 10 year qmail veteran, but only 10 day postfix newbie :P

After researching I think the following would suite our main.cf, would you suggest any alterations (like whats in recpt should be in sender or un-needed etc etc etc) to these

The only ones I'm unsure on are...

smtpd_sender_restrictions = reject_unknown_sender_domain,permit_mynetworks,permi
t_sasl_authenticated,reject_non_fqdn_sender,permit

smtpd_recipient_restrictions = reject_unknown_recipient_domain,permit_mynetworks
,permit_sasl_authenticated,reject_unauth_destination,reject_unauth_pipelining,re
ject_invalid_hostname,reject_non_fqdn_hostname,reject_non_fqdn_sender,reject_non
_fqdn_recipient,reject_unknown_sender_domain,reject_rbl_client bl.spamcop.net,re
ject_rbl_client dnsbl.sorbs.net,reject_rbl_client zen.spamhaus.org,permit

smtpd_delay_reject = yes
smtpd_helo_required = yes
smtpd_helo_restrictions = permit_mynetworks,permit_sasl_authenticated,reject_inv
alid_helo_hostname,reject_non_fqdn_helo_hostname,reject_unknown_helo_hostname

I guess this is bloated but I'm trying to mirror, sendmails bad_helo, bad_mx, PTR and FWD enforcment and so on.

Ed


Reply | Threaded
Open this post in threaded view
|

Re: New to PF, IO bound query

mouss-2
Eddie b wrote:
> I do stand corrected, I played on our dev box once I returned to work and
> have it working perfectly as I want it, but before i allow it to go live i
> need to ensure a few things, you see I'm a 20 year sendmail veteran, and a
> 10 year qmail veteran, but only 10 day postfix newbie :P
>


I saw that Timo gave you a better way for dovecot.


> After researching I think the following would suite our main.cf, would you
> suggest any alterations (like whats in recpt should be in sender or
> un-needed etc etc etc) to these
>
> The only ones I'm unsure on are...
>
> smtpd_sender_restrictions =
> reject_unknown_sender_domain,permit_mynetworks,permi
> t_sasl_authenticated,reject_non_fqdn_sender,permit
>
> smtpd_recipient_restrictions =
> reject_unknown_recipient_domain,permit_mynetworks
> ,permit_sasl_authenticated,reject_unauth_destination,reject_unauth_pipelining,re
> ject_invalid_hostname,reject_non_fqdn_hostname,reject_non_fqdn_sender,reject_non
> _fqdn_recipient,reject_unknown_sender_domain,reject_rbl_client
> bl.spamcop.net,re
> ject_rbl_client dnsbl.sorbs.net,reject_rbl_client zen.spamhaus.org,permit
>
> smtpd_delay_reject = yes
> smtpd_helo_required = yes
> smtpd_helo_restrictions =
> permit_mynetworks,permit_sasl_authenticated,reject_inv
> alid_helo_hostname,reject_non_fqdn_helo_hostname,reject_unknown_helo_hostname
>
> I guess this is bloated but I'm trying to mirror, sendmails bad_helo,
> bad_mx, PTR and FWD enforcment and so on.
>

- it is simpler to put all your restrictions under
smtpd_recipient_restrictions (remove smtpd_sender_restrictions and
smtpd_helo_restrictions). the result is the same, but you have a
sequential list (easier to read and maintain) and you don't need to
repeat the "permit_*" whitelisting checks.

- by default (smtpd_delay_reject=yes), reject_unauth_pipelining is
useless before data stage (checks are evaluated at RCPT TO stage, when
pipelining is ok).


your restrictions could be written as:

smtpd_recipient_restrictins =
        reject_unknown_sender_domain
        reject_unknown_recipient_domain
        permit_mynetworks
        permit_sasl_authenticated
        reject_non_fqdn_sender
        reject_non_fqdn_recipient
        reject_unauth_destination
        reject_invalid_helo_hostname
        reject_non_fqdn_hostname
        # no point to query dns if address does not exist
        reject_unlisted_recipient
        reject_unlisted_sender
        # the following check will catch legitmate mail
        reject_unknown_helo_hostname
        reject_rbl_client zen.spamhaus.org
        reject_rbl_client bl.spamcop.net
        reject_rbl_client dnsbl.sorbs.net
       
smtpd_data_restrictions =
        reject_unauth_pipelining


why do you use the reject_unknown_mumble_domain "soon"? do you want to
reject your own users mail if the sender or rcpt domain doesn't resolve?
This may be undesirable: if you have a dns failure, you force the user
to retry later, which is not very friendly.

I suggest moving reject_unknown_sender_domain down (after
reject_non_fqdn_hostname or even after rbl checks), and removing
reject_unknown_recipient_domain (because after
reject_unauth_destination, the recipient domain is yours, so there is no
point checking it in DNS).

I also suggesting removing reject_unknown_helo_hostname because it does
catch legitimate mail.

if you want to enforce FcrDNS (ip->ptr->ip match), then you want
reject_unknown_client. however, this will catch legitimate clients (even
well configured ones if there is a DNS problem, such as if they are
under a DoS attack).

Reply | Threaded
Open this post in threaded view
|

Re: New to PF, IO bound query

Eddie b


On Thu, Sep 4, 2008 at 10:05 AM, mouss <[hidden email]> wrote:

The only ones I'm unsure on are...

smtpd_sender_restrictions =
reject_unknown_sender_domain,permit_mynetworks,permi
t_sasl_authenticated,reject_non_fqdn_sender,permit

smtpd_recipient_restrictions =
reject_unknown_recipient_domain,permit_mynetworks
,permit_sasl_authenticated,reject_unauth_destination,reject_unauth_pipelining,re
ject_invalid_hostname,reject_non_fqdn_hostname,reject_non_fqdn_sender,reject_non
_fqdn_recipient,reject_unknown_sender_domain,reject_rbl_client
bl.spamcop.net,re
ject_rbl_client dnsbl.sorbs.net,reject_rbl_client zen.spamhaus.org,permit

smtpd_delay_reject = yes
smtpd_helo_required = yes
smtpd_helo_restrictions =
permit_mynetworks,permit_sasl_authenticated,reject_inv
alid_helo_hostname,reject_non_fqdn_helo_hostname,reject_unknown_helo_hostname

I guess this is bloated but I'm trying to mirror, sendmails bad_helo,
bad_mx, PTR and FWD enforcment and so on.


- it is simpler to put all your restrictions under smtpd_recipient_restrictions (remove smtpd_sender_restrictions and smtpd_helo_restrictions). the result is the same, but you have a sequential list (easier to read and maintain) and you don't need to repeat the "permit_*" whitelisting checks.

OK, thats much neater, but if I choose to use sendmail access style file , include that there? or then I need to create sender restrictions? 



smtpd_recipient_restrictins =
       reject_unknown_sender_domain

       reject_unknown_recipient_domain
       permit_mynetworks
       permit_sasl_authenticated
       reject_non_fqdn_sender
       reject_non_fqdn_recipient
       reject_unauth_destination
       reject_invalid_helo_hostname
       reject_non_fqdn_hostname
       # no point to query dns if address does not exist
       reject_unlisted_recipient
       reject_unlisted_sender
       # the following check will catch legitmate mail
       reject_unknown_helo_hostname

       reject_rbl_client zen.spamhaus.org
       reject_rbl_client bl.spamcop.net
       reject_rbl_client dnsbl.sorbs.net
       
smtpd_data_restrictions =
       reject_unauth_pipelining


why do you use the reject_unknown_mumble_domain "soon"? do you want to reject your own users mail if the sender or rcpt domain doesn't resolve? This may be undesirable: if you have a dns failure, you force the user to retry later, which is not very friendly.

No point in our mail queues filling up if someone gets a virus that uses our server to relay, a bunch of goblygook that I cant return to them, best they have the problem then our servers, likewise for recipient domains that dont exist, I'm dissappointed I dont seem to be able to 55x those like I can with SM, otherwise everything is running pretty well on dev.
 

I also suggesting removing reject_unknown_helo_hostname because it does catch legitimate mail.

Aware of those risks, have been ujsing it for years without problems.
 

if you want to enforce FcrDNS (ip->ptr->ip match), then you want reject_unknown_client. however, this will catch legitimate clients (even well configured ones if there is a DNS pro

We dont need exact matching, but  ip must resolve and the hostname it resolves to must have an ip, that also resolves if it is different, no exact triple matching, like most ircd's perform.
 
Thanks for your help, much appreciated.

Reply | Threaded
Open this post in threaded view
|

Re: New to PF, IO bound query

Eddie b


On Thu, Sep 4, 2008 at 1:42 PM, Eddie b <[hidden email]> wrote:



OK, thats much neater, but if I choose to use sendmail access style file , include that there? or then I need to create sender restrictions? 




Scrap that! I think I'm getting the hang of it, it'll be in recipients :)

Time for a coffee I think :)  Thanks again for all your help