Looping Construct for ETL – Simplified by Talend DI

I am not a regular ETL developer but I do “dabble” in developing code from time to time. One requirement that I come across is looping (while, for, infinite) in a job and performing a task based on condition and then going to sleep for a period of time. This certainly can be accomplished through a scheduler as well but is often time easier to accomplish in the ETL job design.

I have had the opportunity to implement this construct using SSIS and Datastage. In my opinion, Datastage was the most complicated – took me a good 5 hours to get the code to do exactly what I wanted. SSIS was more cooperating – I was able to do the same in about 3 hours.

As I started working with Talend Open Studio earlier this week, I was amazed to see how simple it was to work with Talend.

As you open the Studio, you can find various orchestration steps very quickly – something the other two tools do not do. i.e. the IDE user interface is more intuitive for Talend


From here, it was drag and drop easy…

Step 1: Drag the while loop on the canvas – define the parameters.

Step 2: Create a job run step to be executed in each of the iterations

Step 3: Drag the “sleep” step and assign values.

I was done start to finish in about 20 minutes and the code generated in Java was pretty easy to read and understand. Very cool!

Complete the form to hear from us.

Leave a Comment

More from our blog

See all posts