This article is a detailed tutorial on leveraging Dell Boomi’s Integration Platform as a Service (iPaaS) to resequence out-of-order messages coming from Google Cloud Pub/Sub with the help of an In-Memory Data Grid. While there are many use cases of why you may want to integrate with the Google Cloud Platform, this article focuses on a particular use case that we run into often where you need guaranteed delivery of messages in the order in which they were generated. We leverage the Dell Boomi platform with Hazelcast to achieve the reordering and then publish the messages to a JMS queue.

Now that you know what to expect from this article, let me give you some background before diving into the detailed steps to accomplish this tutorial’s objectives.

Background: Dell Boomi Platform, Dell Boomi Integration Process & Google Cloud Pub/Sub

Dell Boomi Integration Platform

Boomi Integration Platform is a market leader in Gartner’s Enterprise iPaaS Magic Quadrant for seven years with end-to-end capabilities, including data discovery, application and data integration, API management, data quality governance, B2B integration, low code app development, and more. From an integration perspective, it provides connectors and tools to implement and access different phases of the integration lifecycle — building, deploying, testing, and managing processes with their dependent components.

Dell Boomi Integration Process

A Boomi Process represents a business process or transaction-level interface between two endpoints/systems that may contain a series of shapes connected to illustrate the steps to transport, route, or manipulate data from source to destination. The structure and format of Boomi data are represented as Profile. Boomi supports a broad range of connectors, but in this use case, connectors that have been used are Google Cloud Pub/Sub, JMS, and HTTP for integration.

For Google customers, Boomi delivers an easily scalable integration platform to manage, move, orchestrate, and govern data across applications, whether it’s on-premise or in the cloud.

Google Cloud Pub/Sub

Google Cloud Pub/Sub is an asynchronous messaging service that decouples services that produce events with the services that process events. Google Pub/Sub can be used as messaging-oriented middleware or event ingestion and delivery for streaming analytics pipelines. It also offers durable message storage and real-time message delivery with high availability and consistent performance at scale.

However, there’s a limitation in distributed messaging systems: it does not guarantee in-order or FIFO (first-in-first-out) delivery. Yet, in many real-time business use cases, our customers need the sequential order messages as it originated from the data source. With the help of Boomi and Data Grid, we can resequence message order and can publish to any target endpoints. I came up with the following solution to handle the above scenario by considering a real-time use case.

Resequencer rearranges out-of-sequence messages back into the order. In this use case, incoming message streams (random order) are received from Pub/SUB, and then Boomi sends them to the target endpoints in an orderly manner.

Introducing Our Use Case: Handling Purchase Order Events From A Point Of Sale System

A point of sale (POS) is where customers make their payments for products or services at the store. Every time a customer makes a purchase, they’re completing a POS transaction.

Purchase Order events are published from several POS Systems to the POS Server. These events need to be combined so that the target user/service will receive a complete Purchase Order. Also, we need to note that events received by POS Server may not be in the same sequence as they were created due to system processing time or network delay.

PO events are published to Google Cloud Pub/Sub to simulate the above scenario. For each PO, a source sends a series of events (XML messages with EventType as STARTPO,ENDPO,LINEITEM). Also, the PO ID of all event messages belonging to a Purchase Order are the same, and ENDPO event also contains the total count of Line Items for that Purchase Order.

Dell Boomi GCP Integration - Purchase Order Use Case
Purchase Order Use case
Dell Boomi GCP Integration - Message Resequencing Process Flow
Message Resequencing Boomi Process Flow

Boomi Platform provides Google Cloud Pub/Sub Connector to receive and send messages with PUBLISH/PULL/ACKNOWLEDGE operations. Boomi sends all messages to Data Grid via a REST API call and later publishes a complete Purchase Order into JMS with the same order sequence as they originated from the source. Here Resequencing of messages is based on PO ID and SEQUENCE ID of each line item of an event.

Without further ado, let us move on to the actual hands-on tutorial.

Tutorial: Using the Dell Boomi Integration Platform to resequence out-of-order messages coming from Google Cloud Pub/Sub

Prerequisites

Before you start the Boomi integration, keep the following requirements ready:

  1. Dell Boomi account
    Create your Boomi Platform account https://platform.boomi.com/#login;
  2. Google Cloud account
    Create your Google Cloud account https://cloud.google.com/gcp, create a project, configure IAM, create a topic and subscriber in Pub/Sub
  3. JMS
    Install and Configure JMS
  4. Data Grid
    Install and configure any Data Grid

Note: ActiveMQ (as JMS) and HazelCast (as Data Grid) have been used in this use case. Visit http://activemq.apache.org/ and https://hazelcast.com/ for installation.

Data Grid

Several Data Grid tools are available like HazelCast, Infinispan, Ehcache, etc. For this use case, I used HazelCast, an open-source in-memory data grid (IMDG) based on Java. Right now, there is no connector available in Boomi Platform to access IMDG, hence a REST Layer has been created to access Data Grid. (Hopefully, we’ll have a Data Grid Connector available soon in the future.)

The following is a snapshot of REST APIs created that Boomi used to access Data Grid.

Dell Boomi GCP Integration - Data Grid Access

Create the Boomi Process

Create a new process in Boomi Platform and add a Connector Shape, choose Google Pub/Sub as Connector Type, PULL as Action Type, then create Connection and Operation as follows:

Provide the Google Cloud project name that you created in your Google Cloud Account.

Dell Boomi GCP Integration - project name

Provide OAuth 2.0 information which includes Grant Type, Client ID, Client Secret, and Scope in OAuth2.0 tab.

Dell Boomi GCP Integration - connection information

Test your connection

Dell Boomi GCP Integration - test connection

Now, create an Operation in Boomi Connector Shape and Save it.

Dell Boomi GCP Integration - create operation

Save your connection details after adding Connection and Operation.

Dell Boomi GCP Integration - display name

Your Boomi Process should now look like this:

Dell Boomi GCP Integration - Boomi process

Update your Boomi process as follows.

Dell Boomi GCP Integration - update Boomi process

Add Process Route to identify the Type of inbound message and redirect it to corresponding Sub Processes. Creating a Sub Process is discussed later in this section.

Dell Boomi GCP Integration - Process Route Shape

Configure Process Route for STARTLINEITEM/LINEITEM/ENDLINEITEM

Dell Boomi GCP Integration - Configure Process Route

Add new Connector Shape and provide Type as Connection, Component Name, Folder and select Connector as JMS.

Dell Boomi GCP Integration - add new connector shape

Set JMS Connection with the following details:

Dell Boomi GCP Integration - set jms connection

Create JMS Operation and set destination with the Queue Name prefix with “dynamicQueues/”.

Dell Boomi GCP Integration - create jms operation

Note: Here Test is the queue name that will be created in ActiveMQ dynamically.

Create a sub-process to process STARTPO Events.

Dell Boomi GCP Integration - sub process
Start Line Item Sub Process

To access hazelCast REST API, add HTTP Client Shape.

Dell Boomi GCP Integration - access hazelCast

Note: HazelCast Connector is not available hence a RESTful web service has been developed with Spring boot framework, which is internally called HazelCast APIs.

Dell Boomi GCP Integration - data grid get api
Dell Boomi GCP Integration - find by PO ID

Similarly, create two more Http Client connectors for Cache PO and Send Complete PO as mentioned in the Start Line Item SubProcess.
Cache PO will persist Events into HazelCast whereas Send Complete PO will publish complete PO from DataGrid into JMS.

Create 2 more Boomi sub-processes:
LineItem SubProcess (process LINEITEM Events)
EndLineItem SubProcess (process ENDPO Events)
configure these processes with Process Route.

Write to us if you want in detail implementation of the above Sub-Processes.

Test Process

Before testing our Boomi Process first publish some sample data into Google Cloud Pub/Sub.

You can view your data in Pub/Sub Subscriptions console.

Dell Boomi GCP Integration - view data

Test Boomi process and verify documents are getting processed successfully.

Dell Boomi GCP Integration - test Boomi Process

Open Process Call and verify if the document is processed via sub-processes successfully.

Dell Boomi GCP Integration - open process call

Once Boomi Process sends messages into HazelCast, you can view PO IDs on HazelCast Dashboard.

Dell Boomi GCP Integration - HazelCast Dashboard

Once all messages are received from POS Server for a PO ID, Boomi will process and publish them into ActiveMQ in the same order as they originated from the source. Open ActiveMQ web console and explore Queue to find your complete PO.

Dell Boomi GCP Integration - ActiveMQ console
ActiveMQ Console

We are good with the testing so now let us deploy the process.

Deploy Process

Create a “Packaged Component” of Boomi Process, deploy it into the Atom Environment, and schedule it.

Dell Boomi GCP Integration - packaged component

Your Boomi process is now deployed and ready to receive messages from GCP!

References:

Google Cloud : https://cloud.google.com/
HazelCast : https://hazelcast.com/
ActiveMQ : http://activemq.apache.org/
Dell Boomi : https://boomi.com/
Google Cloud Pub/Sub : https://cloud.google.com/pubsub/docs/overview

More about JMS Configuration and GCP Certificate Creation and Import:

Configure JMS in Boomi:

JMS libraries must be placed in the Boomi platform Account Libraries, these libraries are shipped with the JMS provider. Required libraries (JAR files) are in the ActiveMQ installation folder (activemq-all-5.15.12.jar, geronimo-j2ee-management_1.1_spec-1.0.1.jar).

Add libraries into Account Libraries

Dell Boomi GCP Integration - JMS libraries

Account Libraries are used to upload and manage custom JAR files for integration processes.

Now, you can create a new Custom Library component. This component is a collection of Java Archive files (JARs) you can use to support integration processes. Custom Libraries are reusable, which is why we can use them in several Boomi Processes.

Create a new Component and choose Type as Custom Library, provide Component Name and choose a project folder.

Dell Boomi GCP Integration - create component

Add ActiveMQ jars in Custom Library and Save.

Dell Boomi GCP Integration - add ActiveMQ jars

Create Package Component and Deploy the Custom Library into Atom Environment.

Dell Boomi GCP Integration - create package component

Note: Listeners cannot be tested in the test environment.

Create Certificate/Private key in Google Cloud Platform:

Some configurations are required in the GCP for integrating with Boomi.

Certain connectors might require the use of a certificate to ensure security when transmitting data across the communication protocol. Connectors such as FTPS, SFTP, HTTPS, AS2, and many others require the use of certificates to encrypt data and verify the digital signature of the source sending the data. You can create two types of certificates in Boomi (i.e. X.509 and PGP). Here X.509 Boomi certificate (Private) has been used.

Login to your Google Cloud Platform and create a service account.

Provide Service account details

Dell Boomi GCP Integration - create service account

Provide appropriate permissions

Dell Boomi GCP Integration - provide appropriate permissions

Grant users (optional)

Dell Boomi GCP Integration - grant users

Once the service account has been created, go to Google’s Credentials page, service account should be listed.

Dell Boomi GCP Integration - credentials page

Now click on the service account highlighted above and create a new Key.

Dell Boomi GCP Integration - services account

Select P12 and save the key in your system.

Dell Boomi GCP Integration - p12

Certificate generation is done at GCP, this will be used in Boomi.

Import certificate in Boomi platform:

Login to your Boomi account and create a Certificate component and provide Component Name, Boomi project Folder, and Certificate Type as X.509.

Dell Boomi GCP Integration - import certificate

Import the certificate (key) you downloaded from the GCP service account.

Dell Boomi GCP Integration - import certificate

Save and Close

Dell Boomi GCP Integration - save and close

You imported the certificate successfully in the Boomi Platform!

Summary: Using the Dell Boomi Integration Platform to resequence out-of-order messages coming from Google Cloud Pub/Sub

There you have it, a detailed tutorial on using Boomi’s platform to resequence messages received out-of-order from Google Cloud Pub/Sub, and resubmit them to a message queue to be processed further with the guarantee that the messages will be in chronological order.

If you have questions on how you can best leverage Boomi or need help with your Boomi implementation, please reach out to us to get a conversation started.

Share This