Debunking the Top 5 Data Warehouse Myths and Misconceptions

Data warehouse (DW) and analytics professionals are under renewed pressure to develop new data sets and insights in shorter time frames than ever. Businesses are placing greater demands on them as they seek to make more use of modern, digital and data-driven practices. As the pace of business accelerates, data solutions need to be delivered faster than ever before and there is a demand for a broader range of analytics.

But while businesses require their DW team to achieve improvements, they are also under pressure to contain costs.

To achieve acceleration and other goals, DW teams are adopting modern methodologies and best practices. In a recent report, a majority of respondents (70%) highlighted agility – speed, productivity, flexibility and innovation – as a major requirement. A number of other modern development methods are usually applied hand-in-hand with agile methods, namely scrum, lean and rapid prototyping. Agility and other new methods can help a DW team achieve realignment to support new business directions.

In common with many other IT disciplines, there are a number of myths and misconceptions surrounding the acceleration of DW development. Some of these myths were identified in the report and it is important they are addressed, otherwise, they may inhibit an organisation’s success or willingness to adopt new tools and practices.

Myths that Need Busting

Myth 1

Developers lose their jobs due to software automation – This is false and is probably the most dangerous myth because it can deter an organisation from adopting progressive and modern practices. Fortunately, a large majority of respondents to the survey (69%) agreed there was no substance to this statement.

In reality, software automation helps developers in many ways. It accelerates the development process to deliver data solutions sooner, boosts developer productivity for greater numbers of solutions and frees them from many repetitive, manual and administrative tasks. In addition, it can give them more time for higher-level design tasks and help them adopt new and more agile development methods.

Myth 2

Development is the only DW aspect that needs automation and acceleration – A massive 83% of respondents stated, correctly, that this assertion was wrong. A comprehensive toolset for DW professionals is not just confined to development but will automate a wide range of tasks, including deployment, testing, monitoring, documentation and administration. When automated by special software, many of these tasks become more accurate, repeatable, compliant with data standards and conducive to collaboration.

Myth 3

Automation inhibits creativity, flexibility and unique customisation – A majority of respondents (55%) argued this statement was false. Even if a tool has its approach to automating a task or process, there is still room for developers to add their value and adapt tool usage to the organisation’s unique requirements.

Myth 4

Automation only works with traditional data structures in moderate volumes – Close to two-thirds of respondents (63%) disagreed with this statement. Although it can depend on the tool, most are adapting to the wide range of data structures, sources, types, containers and latencies that are increasingly common nowadays, as well as the burgeoning volumes of big data.

Myth 5

Automation tools are expensive compared to hiring employees or using consultants – Less than a fifth of respondents (19%) stated automation tools were less affordable than adding more staff or bringing in outside help.

In the past, DW teams augmented staff with new hires and/or consultants, but attitudes are changing. Payroll costs are often higher than software licensing, so acquiring modern tools is becoming more affordable by comparison.

Best Practices for Accelerating DW Development

There are a number of ways for DW and analytics professionals to meet the challenge of accelerating data warehouse development. They can start by placing a greater emphasis on accelerated DW methods and automated DW tools. This will help them keep pace with business change, make the data warehouse relevant in a modern world and modernise the DW and analytics in general. Many have already accepted the significance of DW acceleration. A large majority of survey respondents (91%) agreed it was important.

Development methods should be modernised, not just data platforms and tools. The highest priority should be the adoption of agile methods that have been adapted to data-driven development.

DW acceleration and automation should be widened to cover many data disciplines, not just the central DW. Many supporting data disciplines around it need acceleration and automation, including data integration, quality, modeling, reporting and analytics.

The DW software portfolio should also include tools that support automation and agility. Acceleration, automation and agility require modern toolsets.

Following best practices and dispelling these myths is critical as data warehouses and other analytics have become increasingly important for organisations seeking to understand and predict change and use it for their advantage. To achieve that objective, DW professionals need to have the appropriate tools, resources and guidance to accelerate development and deliver effectively.

Simon Spring, Account Director EMEA, at WhereScape

Simon joined WhereScape nearly ten years ago and throughout this time has worked effectively with hundreds of organisations looking to utilise data analytics and data warehouse automation to transform their business.