No other industry in the world experiences as much change as technology does. True to form, data warehousing as we know it is no longer what it was 5 or 10 years ago. New demands from customers, businesses and regulators continue to push the envelope on data warehouse capability and flexibility. The emergence of what is often referred to as ‘big data’ (terabytes of business information belonging to or under the custody of a single organisation) has introduced substantial challenges for businesses keen on staying on top of spiralling data volume, speed and complexity.
But it is perhaps important to start by briefly looking at where the concept of data warehouses originally came from. Data warehousing first came to the fore due to the need to analyze and report on ever increasing quantities of data. In particular, data warehouses helped to separate data analysis and reporting processes from transactional systems, ensure consistency by holding all enterprise data in one repository, and supporting overall business strategy through custom data queries, various calculations and service level agreements.
There is still widespread misunderstanding on just what a data warehouse is. Some technology analysts have estimated that roughly 3 out of every 4 enterprise systems referred to as data warehouses are in fact ordinary databases or single-department systems. At least 3 out of every 5 SMBs (small and medium sized businesses) do not use a data warehouse of any form, instead relying on the data manipulation capabilities of other business systems.
These statistics and more point to a significant gap in the market. Whereas many organizations have so far managed to survive on workarounds and generic data analytics systems, the need to remain competitive as well as pressure from regulators will eventually drive many into acquiring true data warehouse infrastructure.
At the beginning of 2011, the data warehouse market stood at roughly US$23 billion with that figure expected to increase by more than 7 percent in 2012. The growth will be driven not just by raw demand but also by the availability of better tools for integrating legacy systems with modern data warehouses – a trend that was already taking root in 2011.
The new technology is spawning data warehouses that can manage not just enormous quantities of data but also data sourced from multiple disparate systems. The new type of data warehouses are sometimes called logical data warehouses due to their emphasis on data logic and information processing. This is in contrast to the traditional data warehouse that is designed around specific software and hardware infrastructure.
Logical data warehouses are therefore more adaptable even though such warehouses currently constitute less than 5 percent of the entire data warehouse market.
Other emerging trends
– Willingness to pay more for quality – Balancing cost and functionality is the eternal dilemma in the IT procurement process. In cases where there is more than one solution that can do the job (or at least the basics), purchase price is usually the first factor in the elimination process.
But given the critical role data warehouses are playing (e.g. ensuring compliance with Solvency II and Basel III reporting requirements for insurers and banks respectively), 2011 has seen many businesses pay substantially more for data warehouse infrastructure than they otherwise would. The rationale here is simple – why buy cheap technology when management and administrative costs later on will negate any savings realized from the initial purchase?
– Data warehouse appliances are increasing in popularityt. The less-than-two-year-old Microsoft PDW (Parallel Data Warehouse) has experienced a weak response with many of the software giant’s data warehouse customers instead opting for MS-SQL Server’s Fast Track Data.
– Formalized professional support – As the statistics above demonstrate, many organizations are yet to acquire a true enterprise data warehouse. And many that use data warehouses have only recently started to do so – three years or less. Unlike more common business systems, there are not many subject matter experts within non-IT organizations. As such, data warehouse vendors are keen on not just delivering infrastructure but also positioning themselves to provide professional support and consultancy services before and after the sale.
– Consistency with business requirements – For a business, every expense must be weighed for its impact on the bottom line versus the benefit it delivers. Clients are drawn to particular data warehouse vendors not just because their system helps them with regulatory compliance but also because such warehouses increase overall business efficiency.
– More choices makes decision making difficult – As more vendors enter the market place, clients will have to make a decision from a larger variety of vendors than ever before. While this competition may mean higher quality products and proposals, the diversity can inadvertently drive some businesses into settling for short-term stop-gap solutions that are ineffective in the long term.
More on data warehouse.