In our first post we’d like to make a controversial statement sure to anger many of our readers: the storage industry has failed to meet market needs over the last 10 years. Why do we say that? Because top companies are building their own storage solutions instead of using available technologies.

The most famous and interesting example is, of course, Google. Google has developed its own in-house data storage infrastructure, providing it with a global, scalable, easy–to-manage solution. As far as we can tell, no commercial product today offers comparable capabilities.

Another example is Amazon, which developed its own in-house storage system, later offering it as the S3 service. And there are more examples.

These companies did not develop new network hardware in-house; they used existing off-the-shelf technology from CISCO and other networking companies. They did not develop their own operating system; they used Linux, or some other operating system. They did not develop new motherboards or processors; they used the best available options on the market. They did not reinvent a new network layer; they used TCP/IP.

We can already hear your objections, all variations on the theme that “Google did not implement a generic storage solution — they implemented a unique solution that tightly integrates storage and application.” That is true, of course, since when one has to develop one’s own solution, it is tuned to one’s own requirements. If Google had been forced to develop its own operating system, we are sure that it would not have been a generic, all-purpose one, but designed for its own needs. But they found that the current offerings in the operating system world were “good enough” for them, and they used the generic technology. The same applied to network hardware, network protocols and other technologies; current market offerings were “good enough.”

Are generic storage solutions adequate to the needs of these giants? We think not. Take everything we know about these solutions, and ask storage administrators who manage one petabyte or more (as well as most of the storage administrators who manage 50TB or more), and we are sure they would be delighted to pay astronomical prices for an appropriate solution, but no salesperson has ever knocked on their door and offered one.

So networking technology has been catching up to customer needs, operating system technology provides adequate solutions, server virtualization technologies are providing amazing benefits, but storage lags behind.

Where did storage fail? Here are some guesses:

• Scalability and global name space: there is no practical solution for a global name space in the SAN world, spanning multiple systems. When a volume is migrated from one system to another, hosts must be configured, switch zones must be set, and application downtime is required. NAS is getting a bit better with time, but still nowhere as good as networking technology.
• We still have to assign logical entities to physical entities, that is, volumes to controllers/spindles. The server virtualization companies have shown us how space, power, manageability, and utilization all improve when one virtualizes the logical server over a bunch of physical resources.
• Management standards have not matured yet, and there is no standard interface that allows easy management of heterogeneous environments.
• Host software is still proprietary and lagging behind. Microsoft’s blessed VSS and MPIO frameworks are a step in the right direction, and we really miss a VSS-like solution for Unix environments.
• Disaster Recovery (DR) methodologies are still project/solution oriented and not a packaged product. Although everyone needs one of a limited number of DR solutions, everyone has to implement an in-house integration project.

So what’s next? We don’t know. All these issues require new technologies or new standards, or a new industry-wide standard. We don’t know who will implement them, but we’re sure the market really needs them.

Welcome to the launch of ThinkStorage, the XIV blog.

As you see in the About section, the focus of this blog is the storage community: administrators, IT professionals, and technology developers. We aim to keep the focus on storage technology. We warmly invite others to join the dialog.

The first entry is by Moshe Yanai, a familiar name in storage. It starts with a mea culpa for us all: why are storage customers building their own solutions? Please share with us what you think.