Many of our clients utilize InRule for data preparation, data cleansing and ETL processing. Recently, we’ve seen a rise in demand in this area so we thought we’d detail how InRule can be used for this scenario.
Before we get going, I want to provide a quick overview of ETL: ETL stands for Extract, Transform and Load. ETL solutions pull data in from multiple sources, cleanse and shape that data before it is finally compiled into a singular source. Once the data is in this singular repository, it can be used for a variety of different applications and functions.
Rule applications are a great alternative for ETL processing because of performance, ability to handle algorithmic complexity and reuse of business logic. Recently, we were asked to demonstrate high-performing ETL operations for JSON data residing in MongoDB. We loved the request because it highlights a complete JavaScript lifecycle supported by our product and it shows all of the moving parts in the scenario.
Import the Schema
The first step in every rule application is to import the schema (you can also make it from scratch). irAuthor imports multiple formats including JSON. While using “Studio 3T” for MongoDB, we can browse our JSON documents.
When we right click on one of these documents, we can use a menu option to display the document view. For this rule application, we’ll select the “Mongo export Format,” select the JSON and place it into the clipboard.
Next, we’ll paste the JSON into irAuthor’s import schema dialog. When we’re done, we can see we were able to pull the schema into our rule application.
Author Rules
From here, we can write business rules against the imported entities. In our example, we have a rule checking the loan balance. The challenge was to reason against the customer’s state of residence and the balance of their loan. If the customer’s state of residence is in WA, CA, OR and the net balance of the loan is less than or equal to $1000, then set the status of the loan to near payoff and fire a notification with a friendly message.
Verify Rules
Next, we test this rule using irVerify to ensure that our notification fires and the status of the loan is set properly. We can see below that this is true. If this rule application matured further, we would typically swap out the notification for an action that would drive a specific customer interaction.
Package the Rule Application
We have a simple rule application working. When we click “JavaScript” tab on the ribbon of irAuthor and click the package option, we push the rule application to a packaging service and return a minified JS file.
Once we have selected the features we want in the package, we can click “Package.” The service returns the requested JS file that we deploy. It’s completely portable; we can deploy it to a file system, a mobile device or even wrap it into a service. Below is what it looks like when I open it up in an editor. Note that it offers instructions on how to call the decision.
Integrate
The next step is to integrate. For our purposes, we have written some JS code (Node.JS) that loops through all our JSON documents in MongoDB and then pushes the results to a new collection. We can look at what the code is doing below.
Note above that the entity state is retrieved from a MongoDB collection and each item is asserted to the engine without any modification or serialization steps. This reduces a lot of code and will perform better in the long run.
Execute
We can see what the execution of our rules looks like when we run it from the command prompt. The Node.JS process iterates throughout all the loans making decisions, firing notifications, and giving us feedback for errors.
If you are focused on JavaScript as your platform, InRule is a great choice since it handles for the full lifecycle—even if your goal is not specifically ETL and batch processing. Customers have deployed our JS packages in mobile devices, services and even inside of Java using Rhino. Let us know what you think in the comments below or send us a message at [email protected].
