Advice: NFS, hardware, SATA vs SAS etc

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Advice: NFS, hardware, SATA vs SAS etc

venbian
Hello gurus,

As 2020 nears I wanted to ask opinions about the current state of hardware requirements for a small business email platform. $dayjob asked me to enhance our existing platform to improve performance and add redundancy.

Main questions (TLDR):

Can NFS handle heavy IMAP, LDA, HTTP workload?

Is direct attached SATA III 6Gb/s SSD in RAID 1 sufficient or is SAS needed?

Is gigabit ethernet the bottleneck in any case? I can upgrade to a 10 gigabit local network if advisable.

Can NFS peacefully co-exist with other mail system workloads without resource contention such as SQL DB or spamassassin or redis, etc?

More detail:

I had bad experiences with NFS approx 15yrs ago where IMAP load saturated controller link (yes, noatime was used on the mount) and was unusable. But in 2020 is it time to give NFS another look?

Current mail storage setup uses local attached large SATA SSD and does well, but it directly hosts HTTP, LDA, IMAP and Submission which could all be faster and it only does nightly backups. Adding SAN is probably out of $dayjob pricerange and SAS is borderline. Power consumption is also a factor so instead of a dedicated file server I thought it would make more sense to build a big server with direct attached fast SATA SSD in mirrored RAID that also has strong CPU and maximum memory so it can also run some of the backend process such as spamassassin, redis or SQL database etc. (we want to start using SQL DB for more which means it will be under heavy use)

What workloads can best co-exist with NFS where each does not contend for the other's resources?

I'd put a couple smaller machines in HA in front of that to proxy webmail, HTTP website, IMAP and Submission. Edge MTA is on a separate server and would probably stay that way, maybe adding a failover. It keeps a fraction of its mail in the local system but will make more heavy use of the SQL DB which I thought to also put on the file server(?)

Could NFS keep up with load for proxy of HTTP, IMAP, LDA, etc?

Is local attached SATA SSD in RAID 1 ok? Will 6Gb/s SATA III be a bottleneck in any possible scenario? I was looking at motherboards with multiple PCIe or M.2 slots thinking NVMe bandwidth (3GB/s) would be great but I'm unsure if NFS, gigabit ethernet or other components could even make use of it.

Or is that too amateur and local attached (hope not remote attached) SAS a minimum requirement? SAS SSDs are a newer thing I never used and expensive for the $boss. I read some people express doubts that SSD is suited for SAS at all which is one reason I thought just use SATA for more efficient power and cost.

Our workload:

We process a lot of mail but as you can guess, don't have tremendous storage needs.  We have several tens of thousands of users but a smaller fraction of that are actively using mail every day. Maildir storage is several TB. Exact daily mail volume is unknown but should be on the order of a few million, many which users have forwarded to to other accounts so a small fraction is stored locally.

We also have few TB of web data that is hosted from a server in the same location that I thought to unify into the NFS setup.

Thank you for reading and your insight.

Reply | Threaded
Open this post in threaded view
|

Re: Advice: NFS, hardware, SATA vs SAS etc

lists@lazygranch.com
You ruled out cloud solutions? 
(Original text deleted for brevity.) 

Reply | Threaded
Open this post in threaded view
|

Re: Advice: NFS, hardware, SATA vs SAS etc

venbian

You ruled out cloud solutions? 

Yes. Do any Postfix administrators with busy systems rely on NFS?
Reply | Threaded
Open this post in threaded view
|

Re: Advice: NFS, hardware, SATA vs SAS etc

Phil Stracchino
On 2019-12-05 14:52, venbian wrote:
>
>> You ruled out cloud solutions? 
>
> Yes. Do any Postfix administrators with busy systems rely on NFS?

That seems like a really bad idea, honestly.


--
  Phil Stracchino
  Babylon Communications
  [hidden email]
  [hidden email]
  Landline: +1.603.293.8485
  Mobile:   +1.603.998.6958
Reply | Threaded
Open this post in threaded view
|

Re: Advice: NFS, hardware, SATA vs SAS etc

Coy Hile
On 2019-12-05 14:58, Phil Stracchino wrote:
> On 2019-12-05 14:52, venbian wrote:
>>
>>> You ruled out cloud solutions? 
>>
>> Yes. Do any Postfix administrators with busy systems rely on NFS?
>
> That seems like a really bad idea, honestly.

I haven't used mail systems that had, e.g. /var/mail on NFS in more than
20 years.

--
Coy Hile
[hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Advice: NFS, hardware, SATA vs SAS etc

Wietse Venema
In reply to this post by venbian
venbian:
> > You ruled out cloud solutions?
>
> Yes. Do any Postfix administrators with busy systems rely on NFS?

It can be done, provided that one does not do stupid things like
logging into the NFS server and messing with files that an NFS
client is actively using. That will result in stale NFS handles and
other pain. Postfix has quite a few workarounds for NFS, as described
in http://www.postfix.org/NFS_README.html

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: Advice: NFS, hardware, SATA vs SAS etc

P.V.Anthony
On 6/12/19 8:12 pm, Wietse Venema wrote:

> venbian:
>>> You ruled out cloud solutions?
>>
>> Yes. Do any Postfix administrators with busy systems rely on NFS?
>
> It can be done, provided that one does not do stupid things like
> logging into the NFS server and messing with files that an NFS
> client is actively using. That will result in stale NFS handles and
> other pain. Postfix has quite a few workarounds for NFS, as described
> in http://www.postfix.org/NFS_README.html
>
> Wietse
>

Would it be a better idea to leave postfix as is and move the mail
storage to NFS. Like Dovecot with maildir on NFS. Best of both worlds.

Is this ok?

P.V.Anthony
Reply | Threaded
Open this post in threaded view
|

Re: Advice: NFS, hardware, SATA vs SAS etc

Wietse Venema
venbian:
> You ruled out cloud solutions?

Someone:
> Yes. Do any Postfix administrators with busy systems rely on NFS?

On 6/12/19 8:12 pm, Wietse Venema wrote:
> It can be done, provided that one does not do stupid things like
> logging into the NFS server and messing with files that an NFS
> client is actively using. That will result in stale NFS handles and
> other pain. Postfix has quite a few workarounds for NFS, as described
> in http://www.postfix.org/NFS_README.html

P.V.Anthony:
> Would it be a better idea to leave postfix as is and move the mail
> storage to NFS. Like Dovecot with maildir on NFS. Best of both worlds.

Postfix versus Dovecot is orthogonal to whether one uses NFS. It's
not like one is somehow immune to problems that the other is not.

See http://www.postfix.org/NFS_README.html for considerations
what NFS means for:
- The Postfix mail queue.
- Postfix delivery to mailbox/maildir files.

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: Advice: NFS, hardware, SATA vs SAS etc

venbian
In reply to this post by Phil Stracchino
> > Yes. Do any Postfix administrators with busy systems rely on NFS?
>
> That seems like a really bad idea, honestly.

Coy Hile also said:

> I haven't used mail systems that had, e.g. /var/mail
> on NFS in more than 20 years.

So NFS is a poor, outdated choice for mail storage in 2020 for a small/medium enterprise?

What is the more common solution for networked storage? Does everyone simply bite the bullet and invest in a SAN? What other options do we have?

True, mail delivery won't be postfix responsibility on larger platforms but the this mailing list is full of relevant experience.
Reply | Threaded
Open this post in threaded view
|

RE: Advice: NFS, hardware, SATA vs SAS etc

Patton, Matthew [Contractor]
> > > Yes. Do any Postfix administrators with busy systems rely on NFS?
> > That seems like a really bad idea, honestly.
>
> So NFS is a poor, outdated choice for mail storage in 2020 for a small/medium
> enterprise?

The problem is one of data consistency and locking. Running a farm of IMAP servers off of a common NFS store is acceptable if you make sure Attribute caching is off and you *carefully* review your software manual for gotchas on NFS. For example DoveCot has settings that make it safer for using on NFS.

When running a 'shared' sort of setup 'MailDir' style delivery (each message is a unique file) as opposed to 'mbox' style is required.

> True, mail delivery won't be postfix responsibility on larger platforms but the this
> mailing list is full of relevant experience.

What is the size of your org? It's customary to have a front-end Postfix host that ingests mail, and then sends it to the final delivery host which would also be your IMAP server. On any large number of users some kind of hash is used to distribute email storage across multiple nodes. You could run NFS as the backing store on a single-node (or consistent hash) setup bearing the above in mind. Or you could use local storage and NBD to replicate to another machine for Active/Passive failover.

In this day and age running your own email service is pretty unusual. There are lots of hosted alternatives, just provide credit card.

Reply | Threaded
Open this post in threaded view
|

RE: Advice: NFS, hardware, SATA vs SAS etc

venbian

> > > > Yes. Do any Postfix administrators with busy systems rely on NFS?
> > > > That seems like a really bad idea, honestly.
> >
> > So NFS is a poor, outdated choice for mail storage in 2020 for a small/medium
> > enterprise?
>
> On any large number of users some kind of hash is used to distribute email
> storage across multiple nodes.

Oh that's the obvious answer I didn't think of (but everyone else was thinking, right?). Let IMAP proxy LDA and IMAP traffic to a few file servers. Then those servers won't need expensive SAN as long as they have respectable SATA/RAID.

Thanks!
Reply | Threaded
Open this post in threaded view
|

Re: Advice: NFS, hardware, SATA vs SAS etc

Matus UHLAR - fantomas
>> > > > Yes. Do any Postfix administrators with busy systems rely on NFS?
>> > > > That seems like a really bad idea, honestly.

>> > So NFS is a poor, outdated choice for mail storage in 2020 for a small/medium
>> > enterprise?

>> On any large number of users some kind of hash is used to distribute email
>> storage across multiple nodes.

On 17.12.19 22:32, venbian wrote:
>Oh that's the obvious answer I didn't think of (but everyone else was
> thinking, right?).  Let IMAP proxy LDA and IMAP traffic to a few file
> servers.  Then those servers won't need expensive SAN as long as they have
> respectable SATA/RAID.

Just FYI, proxy won't help you whe one of the storage hosts fails.
We have used NFS cluster because of that.

Mails were processed locally, only delivered to the NFS storage.

Having queue on NFS would not be very efficient but would be safe against
outage of host with postfix.

However, this is off-topic in this queue. hopefully proposed solutions will
be enough for you, good luck.
--
Matus UHLAR - fantomas, [hidden email] ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Fucking windows! Bring Bill Gates! (Southpark the movie)