Project of modernization of data warehouse and business intelligence system.
Over the past 20 years, we have been developing our Business intelligence platform, but the amount of data and the need for their comprehensive interpretation began to exceed the capabilities of the old architecture. Therefore, we began to wonder if it makes sense to write a new system that will be more flexible and much more efficient. Today we tell the story that we have gone through so far. From the preparation, through the definition of the assumptions and their validation, the proof of concept phase to the start of the proper implementation.
Project objective data warehouse architecture
There is a comprehensive modernization of PZU group’s unique business intelligence platform developed for more than 20 years. In recent years, there has been a dynamic increase in the analytical needs of the PZU group staff, combined with a growing awareness of the value of the information resources held by the PZU group. The growing demand for BI services at a rate of several percent per year has led to the expansion of the data warehouse to the limits of scalability. In order to meet the further increase in the demand for updated business knowledge gathered in the data on a daily basis, and thus enable the PZU group to further increase its competitive advantage in central and eastern Europe, it was necessary to completely change the approach to the architecture of the DWH/BI environment.
At the turn of 2015 and 2016, the group began preparations for the launch of the HD architecture project. The first step was to validate the analytical platforms available on the market (such as Appliance and solutions built from components on the customer’s side) in terms of their capabilities in terms of:
Analytical performance, including m.in.:
- Continuous bandwidth in many GB / s
- The ability to use advanced technologies for optimizing analytical processing, such as storage and column compression
Stability under heavy loads, including m.in.:
- Stability for processing thousands of analytical processes per day
- Stability with thousands of users
Safety, including m.in.:
- The highest level of IT security
- High Availability and Disaster Recovery
The effectiveness of cooperation with the existing analytical environment
BI Service Level Monitoring and management capabilities, including m.in.:
- Resource management
- Live performance monitoring
- I / O throttling
We invited the world’s leading providers of analytical databases and top integrators to cooperate on a proof-of-concept basis. In a short time, several analytical platforms appeared in our server room, which we tested. As a result, we had a unique opportunity to compare, according to common criteria, completely different in terms of the internal approach of the analytical platforms. We were able to compare the performance, stability, scalability of “real-life” solutions of the shared-nothing class with the shared-everything architecture and innovative concepts of in-memory, big data, etc. Thanks to this, we gained practically inaccessible knowledge on the Polish market about matching the above technologies, on the scale of implementation of hundreds of TB, with a platform used by employees using various applications through various access channels (including mobile).
Based on the experience gained during the POC phase, we decided to base the new HD architecture on the latest version of appliance from one of the leading suppliers of database systems. The design of the new environment to meet our goals was based on many processing centers and equipped with the latest analytical technologies.
The new environment, created from scratch, includes several hundred TB of data and dozens of computing machines that support the entire BI platform from receiving data, through their processing to delivering them to business customers. The very design of such a large and complex environment, designed to meet the characteristics of Ha and Dr, required many months of work by the best experts from Poland and abroad.
It is also a real challenge to transfer existing analytical solutions developed over 20 years to a new platform while taking advantage of the added value of the new HD architecture (m.in. multiple acceleration of current processes). However, the challenge for the best is to complete this task in less than two years. To this end, specially selected experts from the world work together to produce:
- New logical models for metadata management and data security
- Robots automating process migration tasks
- Robots automating the testing and implementation of migrated analytical solutions (BIOps)
- New Change Management and Release Management processes
- New reference and dictionary systems,
- New BI platform performance management and billing processes
- New Service Level Agreements
The change requires a thorough revision of all architectural assumptions, tools and technical processes, as well as many business processes.
We are always open to new ideas that will help us better face the challenge ahead. We are looking for experts who want to become part of a talented and unique design team consisting of selected employees of the PZU group, as well as the best specialists (including the manufacturer’s staff and leading integrators). If you want to enrich your career with this unique opportunity and are not afraid of the biggest challenges-this place is for you.