The data warehouse market has seen substantial growth over the last 2-3 years despite the difficult economic conditions in the world’s more developed economies. Tech giants such as Microsoft, EMC, Oracle, IBM and Teradata have either rolled out new data warehouse appliances or upgraded their existing product lines.
The expansion of the data warehouse space during such tough times is largely due to a growing realization of the business value and competitive advantage such technology delivers. This business value is in the form of efficient analytics, better business intelligence, ease of forecasting and compliance with regulatory reporting requirements. In fact, the data warehouse has today become mission-critical infrastructure in many businesses.
As data warehouse vendors employ aggressive marketing tactics to stay ahead of the competition, end-users stand to gain considerably. That being said, customers should be careful not to buy into technology simply because of a persuasive sales pitch. Rather, data warehouse selection must be based on credible and candid references as well as viable POCs (Proofs of Concept).
Many current data warehouses lack content analysis capability, a reference to the ability of software to analyze and interpret unstructured data. Some reports have placed the number of such data warehouses at three-quarters of all currently in use and have projected most of these will be retired or overhauled within the next 5 years. Over the same period, column-store in-memory data warhouses are expected to replace at least 25 percent of conventional data warehouses now in use.
The above projections are not surprising given the changing realities of the business environment and especially the formidable challenges presented by big data. Interestingly, no data warehouse vendor small or large has to date managed to roll out a platform that wholly satisfies these expectations. Indeed, many of today’s large businesses running fairly sophisticated data warehouses, still have to incorporate secondary technologies to fully satisfy their requirements.
To stay ahead of the curve, data warehouse vendors have to develop ways of addressing the wide range of data formats – or risk being relegated to the sidelines in future. In this, there may be unexpected beneficiaries – vendors of data integration (DI) and data loss prevention (DLP) tools.
As client expectations of a data warehouse’s capability move from routine access and storage onto comprehension and metadata, context analysis becomes fundamental. Well established data integration and DLP vendors have over the years already developed expertise in the inspection and analysis (both context and content) of unstructured data.
There is a likelihood that some data warehouse vendors will choose to partner with developers of DI or DLP tools to tap into this knowledge. It can also not be ruled out that certain data warehouse vendors will go as far as acquiring smaller tech companies that have demonstrable knowledge and experience in this area. Acquiring small vendors could substantially reduce the cost and time for developing such expertise in-house.
But there is also a real possibility of some DI and DLP developers opting to go it alone and venturing into the data warehouse market themselves – effectively opening a new competition frontier for established data warehouse vendors. Vendors would underestimate the threat posed by DI and DLP developers at their own risk. DI and DLP require a sophisticated understanding of data rationalization that most data warehouses do not possess.
Identifying a data warehouse that satisfies the disparate challenges will be difficult. However, it is clear that any data warehouse vendor that intends to roll out a product that will efficiently support big data must:
- Recognize and analyze shortcomings in existing data warehouses and overall technology infrastructure that make it difficult for data warehouses to satisfy customer expectations.
- Identify new technology infrastructure that would facilitate the achievement of their data warehouse objectives
- Understand the big data instances their product is supposed to address. ‘Big data’ is a fairly broad term and data warehouse vendors may have to zero in on a specific industry if they are to develop a product that can be differentiated from the rest. Whereas developing a data warehouse DBMS product that finds application in a wide range of industries may sound good, vendors may find greater success if they focus on particular industries then build a DBMS that stands out for its precision.
More on data warehouse.