Before any discussion takes place, we have to make sure we are clear about what Tier-Zero storage is. Modern Storage systems have been traditionally grouped into Tiers from one through three or four, depending upon the organization managing this storage. With a recent adoption of flash-based (Solid-state) storage another performance tier was needed, because solid-state drives are certainly far better performing than traditional magnetic disk storage (i.e. traditional hard drives). Tier-Zero today is your highest-performance tier which typically consists of entirely-flash-based components. The key between massive flash-based caches and all-sold-state storage is that caches are by nature temporary and volatile, and there is of course always a cold-cache penalty that simply does not exist with solid-state storage.
In this document we are going to discuss advantages and disadvantage of solid-state storage and hopefully give you enough understanding to make a decision for whether or not this is something that your organization will benefit from. Focus in particular will be on ZFS-based storage, but the document addresses general hardware properties inherent to traditional and solid-state storage.
If you are already familiar with ZFS and the concept of pooled storage, you are likely also familiar with caching features of ZFS. In a typical ZFS-based system the keys to performance are: large amount of RAM to facilitate caching of reads and asynchronous writes, as well as second level of caching which typically consists of one or more solid-state disks. ZFS by nature is designed to cache attributes in data in RAM to facilitate reduced latency in access to the data, most importantly data that is accessed frequently or has been accessed recently. Caching has its limits of course and to extend these limits we traditionally will
This configuration is commonly known as a Hybrid-pool. The implication here is that a pool consists of traditional disk drives and fast cache devices. These configurations are most common and most effective in environments where reads are at least 50% of all operations. Some environments are particularly well suited to using this model. Typically, the more read-biased the environment is, the more it will benefit from caching and in particular level 2 cache. However, the challenges with building an effective caching configuration are many and too often complexities with these configurations result in improperly sized solutions with under-performing caching. One of the challenges with Hybrid-pools is the nature of volatility of cache, which becomes critical after some downtime has been sustained by a system, whether unscheduled or scheduled, when the system is back online cache is empty. This in turn translates into hours or even days of lower performance while cache warms-up, which again depending upon the environment may take quite some time. Another common challenge is sizing of ARC (RAM-based cache) and L2ARC (ssd-based cache), which is often done incorrectly and results in a system with sub-par performing cache.
When a pool is being built with solid-state drives, the design of the pool is much simpler than a Hybrid pool. In a Hybrid-pool we are mixing a number of spinning disks in raid groups with volatile cache devices, whereas an all-ssd pool will only contain ssd drives, and no second level cache. This reduces the complexity of the system and eliminates certain failure scenarios. For example, if an environment relies heavily on caching, and a Hybrid-pool configuration suffers a failure where the caching SSDs end-up being faulty and go offline, performance may come to a halt. This again implies that a poor design of a Hybrid-pool could jeopardize a system, whereas with an all-ssd pool, this complexity is absent entirely. Architecting such pool requires less effort to maximize performance of the primary cache.
Unlike read-biased environments, write-biased environments do not benefit from second level cache nearly as much, especially if the working set size is larger than the amount of cache on the system. Unlike reads which can come from cache and never trigger a disk seek, writes have to eventually access blocks on disk and while writes are also cached with ZFS, they are cached and ordered to improve performance of committing writes to the disks. Of course this kind of caching while it does improve performance overall, will not significantly reduce IO requirements to the disks. Unlike with Hybrid-pools, all-ssd pools perform extremely well in environments that are heavily write-biased. Because there is no second level caching with ZFS, there is much less concern for cold cache performance and poor write performance as a result of a cold cache, which is quite possible during times of heavy read access.