In a previous article, we reached the conclusion that ESB and API management were two sides of the same coin. Does the same apply to ESB and ETL?<\/h2>\n\n\n\n
No, but their respective scopes do partly overlap: they both relate to data transport and transformation within an information system<\/strong>. Given their nature, it could be decided there is a choice to be made, depending on the data flow type. This was not the case between ESB and APIM<\/a>, which each complement the other.<\/p>\n\n\n\n The historical difference is primarily based on the architectural dimension.<\/strong> The specific features of ETL and ESB are not easy to see for those who are not experts in data management. In addition, the landscape has changed over recent years.<\/p>\n\n\n\n Historically, and perhaps somewhat simplistically, ETL is effective in handling huge volumes where performance is important, but the number of interchanges is not that great<\/strong>. It processes data as a set, wholesale<\/strong>. For example, if you want total turnover broken down by customer, ETL will aggregate all the rows and apply a process to all of them. This set-based approach is adopted in particular by Business Intelligence and Data Warehousing. In contrast, ESB is effective when processing large numbers of high-frequency interchanges, each with a limited volume of data, combined with an algorithmic aspect to the processing<\/strong>. It certainly plays a role in de-siloing data and supporting a Service-Oriented Architecture<\/a> by acting as a secure exchange bus across the entire IS.<\/p>\n\n\n\n To summarise using extreme examples, ETL is used to build data warehouses from ERP and CRM systems. ESB is for a business that wants to use a semi-interface to expose estimates, orders and customers from its CRM, for instance, and allow other applications to fetch this data by connecting to the application bus<\/u>.<\/a><\/p>\nEdouard CANTE, Deputy Managing Director of Product at Blueway<\/cite><\/blockquote>\n\n\n\n However, I firmly believe these historical differences now verge on the simplistic. <\/strong>Can anyone really say these days that they are buying an ETL system solely for BI?<\/strong><\/p>\n\n\n\n How process (BPM) and data (ESB) integration can create value for IT and business departments ?<\/p>\n\n\n\n Another difference often put forward is that ETL is a \u201cpull\u201d technology, that works on demand, whereas ESB is a \u201cpush\u201d technology, that produces messages. But from the customer\u2019s standpoint, is it feasible to only pull or only push?<\/strong> When a business has implemented an ETL system, and then needs to \u201cpush\u201d, it can\u2019t be built into each separate application individually. That requires specific developments and highlights the problem at the heart of this distinction.<\/p>\n\n\n\n While compartmentalising ESB and ETL was reasonable ten years ago, I firmly believe it no longer holds true from a business point of view. Different concepts do not need to entail different solutions. <\/strong>Asking the IT department to choose between two different tools depending on whether it wants to expose or interact is approaching the problem from the wrong angle. The business requirement cannot be constrained by technical factors.<\/strong><\/p>\n\n\n\n For the majority of businesses, there is no ROI from installing a pure ETL on the one hand and a pure ESB on the other.<\/p>\nEdouard CANTE, Deputy Managing Director of Product at Blueway<\/cite><\/blockquote>\n\n\n\n Originally, there were some genuine technological differences. The markets split and have developed separately.<\/strong> Some ETL publishers have encroached on ESB\u2019s patch to win market share, and vice versa. At the same time, marketing pitches have compartmentalised requirements based on technology, to confirm their relative positioning. ESB software publishers have also sought to define themselves as purists.<\/p>\n\n\n\n The battle has mainly been waged on a technological front, and not in relation to the actual requirements.<\/strong> This is a mistake!<\/p>\nEdouard CANTE, Deputy Managing Director of Product at Blueway<\/cite><\/blockquote>\n\n\n\n The boundaries consequently became blurred from the customer\u2019s point of view, and simplistic stances have been the result. There is widespread misunderstanding about what an ESB is, and its potential. An ESB was sometimes summarised as a data transporter, completely neglecting its role in organising data traffic in its entirety. The result is that many projects have failed because of these dogmatic approaches.<\/strong><\/p>\n\n\n\n I saw one example with a retailer, where the project started with a theoretical mapping of data traffic on a magnificent diagram, to argue the case for 100% ESB. Ultimately, when the project kicked off, this extreme position came up against the cold reality that some applications could not use it in real time. The project was a complete failure.<\/strong><\/p>\n\n\n\n The real issue is to see things the other way round.<\/strong> It is because of marketing pushing \u201ceither ESB or ETL\u201d that we have these failures! This dichotomy no longer makes any sense in most cases.<\/p>\n\n\n\n The IT department finds itself forced to choose between two tools when its actual requirement is for both<\/strong>: in most cases, it wants to de-silo data and circulate it around the various applications in the IS, produce some BI and centralise data within an MDM system. Matching each concept to a different solution forces it to deviate from the business requirement<\/strong> with no benefit to itself.<\/p>\n\n\n\n They should instead be joined. The solution should meet the business requirement, and not some technological dogma.<\/strong><\/strong><\/strong><\/strong><\/p>\nEdouard CANTE, Deputy Managing Director of Product at Blueway<\/cite><\/blockquote>\n\n\n\n This challenge arises even in user communities that have some proficiency in the tools rather than the concepts. Users then have a reduced perspective on data flow management<\/a> as such. It is by understanding the concepts that users can become more proficient, and more readily adopt different tools.<\/p>\n\n\n\n\n
![\"esb-bpm](\"https:\/\/www.blueway.fr\/wp-content\/uploads\/mockup\/en\/mockup-lb-esb-bpm-EN.png\")
<\/figure>Why do you believe that the distinction between ESB and ETL no longer holds true?<\/h2>\n\n\n\n
\n
How has the market developed since the appearance of the ETL and ESB concepts?<\/h2>\n\n\n\n
\n
So if making ETL and ESB mutually exclusive is a mistake, what is the real issue?<\/h2>\n\n\n\n
\n
![\"expert](\"https:\/\/www.blueway.fr\/wp-content\/uploads\/Mockup_ITW_Distinction_ESB_vs_ETL-EN.png\")
<\/figure>
![\"Schedule](\"https:\/\/www.blueway.fr\/wp-content\/uploads\/Equipes\/Collaborateurs\/visuel-rdv-blueway-1.png\")
<\/figure>