The cost of implementing traditional ETL software is so high, however, that only 25% of data warehouse projects have opted to use them. As is typical in a growing market, proprietary solutions dominate the market at first, generally at a high price tag, until an industry standard emerges. When the industry standard is adequate to achieve what the majority of the customers need, it becomes widely adopted and the proprietary solutions disappear or must refocus on more specific projects. This is a scenario that will likely play out in 2004, with SQL becoming the industry standard for ETL.

Will the proportion of ETL projects hand coded using SQL increase or decrease in 2004?

The responses obtained when a fairly large number of people involved in ETL projects were asked why they chose hand coding were not surprising: they have opted for this solution because of the size of the projects. Typically the initial scope of a project was for only 20 or fewer tables, and the cost of traditional ETL tools was too high in terms of dollars and learning curve.

On the other hand, the reasons they decided to stop hand coding and purchase an ETL tool were mainly the following:

Productivity or speed of development. Even though the SQL language provided with major RDBMSs today is extremely powerful – up to 20 times richer and more powerful than 10 years ago, with dedicated ETL features – SQL is still time consuming to write and hand-coded projects are delivered late more often than not. Maintaining hand-coded ETL processes can be just as costly, especially when the code is passed to other developers (see Point #3 below). Most ETL tools provide a dedicated development environment for data integration, including a powerful graphical interface that allows one to develop five times faster than hand-coding: moreover, objects can be re-used, code is often platform independent, and templates can be designed for easier coding.