I am not a regular ETL developer but I do “dabble” in developing code from time to time. One requirement that I come across is looping (while, for, infinite) in a job and performing a task based on condition and then going to sleep for a period of time. This certainly can be accomplished through a scheduler as well but is often time easier to accomplish in the ETL job design.
I have had the opportunity to implement this construct using SSIS and Datastage. In my opinion, Datastage was the most complicated – took me a good 5 hours to get the code to do exactly what I wanted. SSIS was more cooperating – I was able to do the same in about 3 hours.
As I started working with Talend Open Studio earlier this week, I was amazed to see how simple it was to work with Talend.
As you open the Studio, you can find various orchestration steps very quickly – something the other two tools do not do. i.e. the IDE user interface is more intuitive for Talend
From here, it was drag and drop easy…
Step 1: Drag the while loop on the canvas – define the parameters.
Step 2: Create a job run step to be executed in each of the iterations
Step 3: Drag the “sleep” step and assign values.
I was done start to finish in about 20 minutes and the code generated in Java was pretty easy to read and understand. Very cool!