Managing application compatibility during ETL tools and environment upgrades

Abstract

Extract, transform and load (ETL) tools are software applications responsible for data extraction from multiple sources, cleansing, customisation and insertion into a data warehouse. ETL tools integrate data, which enables the building of the enterprise data warehouse. To meet different business requirements and in order to bring a system up to date or to improve its data integration capabilities, it is necessary to upgrade ETL tools periodically. This paper reviews issues related to software and tools upgrades in data warehousing environments. It reads as a research report for a practitioner audience. We propose the managing of ETL application compatibility during software upgrades through automated tools and processes. We will discuss the challenges involved in ETL application compatibility evaluation during ETL tool upgrade and propose step-by-step methods to approach this. We also share suggestions and recommendations for improvement realised through recent upgrade activities conducted in an enterprise data warehouse production environment.