2018 Brings Once-in-a-Generation Shift in Data Storage

I have to be honest. Much of what I predict to be significant for the data storage industry in 2018 may go unnoticed by most IT professionals, even though some of these concern major progress that hasn’t been experienced in nearly a full generation!

So much has been happening in the development of exciting new storage-class memory (SCM) that will help make arrays faster and more efficient in processing general purpose workloads, and this year I believe we’ll see a tangible payoff with the mainstreaming of this technology. This big news is all about Non-Volatile Memory Express (NVMe) and the emergence of this highly anticipated interface within commercial storage arrays. With Artificial Intelligence and Machine Learning seemingly all the rage in IT these days, I expect to see emerging and increasingly practical use cases coming to commercial storage offerings that attempt to further automate storage management, taking concepts such as intelligent data placement and performance tuning to the next level. And momentum is building for purpose-built data center hardware that provides optimized data paths designed to accelerate specialized workloads.

But enough with the introductions, let’s dive into the details!

Prediction #1: NVMe Will Go From Promise to Productivity in Commercial Storage

A combination of lower cost components being produced in larger volumes by multiple suppliers and a fully-developed specification will finally propel NVMe towards mainstream adoption in enterprise storage. With both storage and servers NVMe-enabled, enterprises will be offered compelling choices that take advantage of a more efficient hardware/software stack that delivers a performance benefit trifecta: low latency, reduced CPU utilization and faster application processing to accelerate performance over that of non-NVMe flash SSDs.  While NVMe-enabled storage arrays will get an initial boost in performance, the true potential for NVMe will be realized later in 2018 when next-generation SCM becomes available (more on that below).

Although NVMe-based flash drives and a handful of storage arrays have been offered for several years, they have typically been 30-50% more expensive than equivalent All-Flash arrays. At this kind of premium, the jury was out on whether these NVMe products were worth the price, since most enterprises wouldn’t have noticed a cost-effective difference in aggregate for general purpose workloads. However, that is changing. Technology maturity means volumes are rising, component costs are coming down and multiple NVMe SSD suppliers are finally ready for prime-time.

We’re only at the beginning of the NVMe innovation cycle. This overnight success has been 10 years in the making, with both Dell and EMC playing prominent roles along the way. In 2018, our industry vision, intellectual property investment and long-term strategic supplier partnerships will pay off. Although a few proprietary NVMe storage products were launched in 2017, broad mainstream solutions will require long-term commitment and dedicated investment to keep up with the latest advances in flash and SCM. We’re ready.

Another underappreciated aspect of NVMe is lower CPU utilization. The NVMe software stack executes fewer instructions. It’s highly optimized for the parallelism of contemporary multi-core processors. With lower CPU utilization for storage, you will have more of your server available to run your applications, which translates to better TCO, improved infrastructure efficiency and software license cost reduction. This kind of performance advantage will be highly sought after by organizations running OLTP and real-time analytics.

Prediction #2: NVMe Over Fabrics (NVMeOF) Will Continue to Emerge and Develop

Hardly anyone will be adopting NVMeOF for production until the industry gets a rich, interoperable set of ecosystem components.  However, we will see incremental progress in 2018, first with NVMeOF Fibre Channel for the incumbent SAN, and then with NVMeOF Ethernet solutions for next-gen data centers. It’s all in line with the development of new interfaces and new storage protocols, but none of it will happen overnight. We need the ecosystem to come along, with new switch ports, software stacks, new host bus adapters (HBAs), etc. In order for adoption to grow, all these factors will need to be developed into a full-fledged ecosystem. As a practical example, at Flash Memory Summit 2017, there must have been a dozen different NVMeOF solutions announced, and I’d guess that no two interoperated with one another. That just reflects the current state of development. I’m a believer and a champion, but it’s early days still. When NVMeOF does hit primetime, watch out. Vendors who do the homework to vertically integrate NVMe in their network, compute and storage products will be at an advantage to offer an incredible performance package for organizations looking to super-charge their SANs. Early adopters will likely be in the HPC, scientific and Wall Street high-frequency trading domains, though enterprises upgrading to modern data centers running cloud-native, IoT and AI/ML applications won’t be far behind.

Prediction #3: Storage Class Memory (SCM) for Storage Will Become a Reality in 2018

Our industry has largely been about spinning hard drives and dynamic random-access memory (DRAM) forever. In 2008, Flash came in and pushed out hard drives as the leading storage media type. In 2018, for the first time in a generation, there are several viable emerging memory candidates in this space. First is Intel with 3DXP. They dipped their toe in the water last year and 2018 is when it becomes a part of mainstream storage architectures. This new SCM should operate in the 10-20 microsecond realm instead of the 100-200 microsecond range for flash. This 10x performance improvement will manifest as both storage cache and tier to deliver better, faster storage.

Of course, low latency applications, such as high frequency trading, will benefit tremendously from SCM. However, SCM is not just for the top of the pyramid workloads and lunatic fringe; the average enterprise will benefit anywhere the equation “Time = Money” comes into play. SCM will be leveraged for real-time risk management – at any time, your most important data needs to be accessed at the lowest possible latency.  And it’s still just the beginning. We don’t get to see a completely new media technology every day. As Pat Gelsinger, CEO of VMware, once said, “There have only been four successful memory technologies in history and I’ve seen over 200 candidates to become the fifth.” The fifth is here, and there are more to come.

Prediction #4: Advancements Will Be Made Around Artificial Intelligence and Machine Learning (AI/ML) in Storage

As an industry, we have been using machine learning techniques to tier data and implement unique solutions in storage for years. Take for example the “Call Home” functionality in VMAX. Our products send regular/high frequency telemetry of all aspects of our storage platforms to Customer Service. This data is analyzed for patterns and anomalies to proactively identify situations before they become problems.  We’re flattered that this approach has been imitated by others, such that now it is a best practice for the industry. Another win for customers.

For 2018, we’ll be seeing AI and ML integration accelerate. Intelligent data tiering will go finer-grained; we’ll see management of more types of media, such as SCM for example – and we’ll be designing in the use of new forms of hardware acceleration to enable that. We will adapt and adopt the latest innovations from the semiconductor processor world, such as graphics processing units (GPUs), tensor processing units (TPUs) or field-programmable gate arrays (FPGAs) to enable autonomous, self-driving storage.

New array applications for AI/ML capabilities will come into play in different ways. Consider the array dynamics when a new workload comes along. AI/ML spring into action, drawing upon telemetry from not only this particular array, but from the cloud-based analysis of all similarly configured arrays to derive the optimal configuration to accommodate this new workload without impact to existing applications.  AI/ML turns the global experience pool of all workloads on all storage arrays into an automated tuning subject matter expert.  Today’s capabilities of CloudIQ and VMAX Call Home are just the beginning. Eventually the idea is that we’ll be able to use cloud-based AI and ML to fully automate the operation of the storage. This will mean storage systems that do more of the data management themselves, enabling organizations to shift dollars away from today’s IT maintenance budgets over to tomorrow’s Digital Transformation initiatives.

Prediction #5: Pendulum Will Swing Back to Heterogeneous Infrastructure to Accommodate Specialized Workloads

Five years ago, the typical view of the data center would be centered on row upon row of identical x86 servers. These servers, automated and orchestrated as a cloud, delivered a standardized set of IaaS and PaaS capabilities for the vast majority of IT workloads. Today,

we’re seeing rapid growth in a new class of algorithmic workloads that are often better suited for specialized processors rather than general purpose homogeneous hardware. Purpose-built hardware often runs these workloads significantly faster and consumes an order of magnitude less power than general purpose compute running software-only solutions. This means that optimized infrastructure architectures will need the ability to deploy business solutions that take advantage of rapid advances in algorithmic processor technology, while keeping all the software-defined flexibility and agility of hybrid and private clouds. Think of this as “software-defined control plane meets hardware-optimized data pipeline.” This architecture may exploit GPUs for machine learning, FPGAs for custom functions, and offload engines for algorithmic data services such as dedupe, compression and encryption.

These raw materials eventually become services, delivered across a low-latency datacenter fabric. This forms the architectural substrate for truly composable infrastructure. Our job will be to manage and orchestrate the dynamic provisioning of those IT services. We’ll start seeing these new capabilities delivered as POCs in 2018.

2018 will be an exciting time for IT infrastructure and our customers. The contemporary IT Architect will have an unprecedented set of capabilities at their disposal, from new processing models, a new class of storage media, and advances in system and data center interconnects. This is especially the case in storage.  With major technology advancements in both capability and affordability on the horizon, 2018 will be a year where we can truly expect to do more for less.