The problem of introducing new “bugs” while making changes, corrections, or additions to a program is a universally familiar concern to programmers. But this concept might be new to business analysts just getting started with a rule engine. Because the term bug is synonymous with “logic error,” this concern applies as readily to a rule application as it does to a computer program.
Creating and testing your rules with irAuthor and irVerify is a relatively straightforward task, but one of the most common drivers for using a rule engine in the first place is the need to respond quickly to changing requirements. That means your pristine initial creation will be subject to maintenance – possibly by authors who have a different (or narrower) understanding of the business problem the rules address. Having a good set of regression tests can help avoid situations where what appears to be a simple change has unintended consequences in other areas of the rule application.
We are going to create a simple regression test suite using irVerify and execute it manually. I’ll also share some irSDK code that could be used to automate test execution.
Creating and Executing a Test Suite
Our tests will be based on the InvoiceSample rule application that ships with irCatalog.
Procedure:
1. Save a copy of the InvoiceSample rule application to the file system.
2. Open InvoiceSample in irAuthor, then open irVerify indicating you want to test the Invoice entity. (You will have to create an empty line item manually.) Test data
- CustomerID 7
- ProductID 2
- Quantity 3
3. Click on the Apply Rules button.
4. After the test executes, click on the Create Test button.
5. Create a Compare Test, which automatically creates a set of checks/assertions that every data element in the test must equal an expected value derived from the results of the test that just executed.
6. For the input state, specify that the input data should be the pre-execution values, the data that was provided in step 2 above.
7. For expected data state, use the values that resulted from running the test in irVerify.
8. Some housekeeping: Assign meaningful names to the test suite and individual test as shown below.
9. Next return to irAuthor and open an additional irVerify test session for the purchase of 3 computers (Product ID 6) and follow steps 3-8 above to create a second test called “High Value Transactions”. Test data:
- CustomerID 7
- ProductID 6
- Quantity 3
10. Execute these tests by clicking the Run All button. The tests should complete successfully (Status = passed). Save the test suite as InvoiceTests.testsuite.
Manual Regression Testing
Now comes the fun part; demonstrating a regression test failure.
The original InvoiceSample rule application contains a simple calculation field for the SubTotal (which currently sums the LineItemTotal field in the LineItems collection). Suppose you want to be able to offer a 10% discount on orders of $5000 or more. One way to accomplish this is to remove the existing calculation from the field and place the new calculation inside a rule set.
Take the following actions:
1. Right-click on the SubTotal calculation and select the option to convert it to a field of type Decimal.
2. Create the Auto Sequential rule set DiscountRules under the Invoice entity
3. Inside DiscountRules, create the following BL (business language) if/then rule.
4. Save the rule application as InvoiceSample_discount.ruleapp.
To execute regression testing on the revised rule application:
1. With the InvoiceSample_discount.ruleapp open, click Test to test in irVerify at the Invoice entity level.
2. From irVerify’s File menu, select Open Test Suite and select the InvoiceTests test suite by browsing to it on the file system.
3. Select Run All Tests. Both tests now fail. It’s not surprising that the High Value Transaction test has failed –we never adjusted the test’s ending data state to reflect a discount situation—however, the Low Value Transaction test results were not supposed to change, and that test has failed as well.
What happened? Well, our new Discount Rules rule set never sets a value for the Subtotal field if it’s less than $5000! We “got it working” but our adjustments had unintended consequences.
Automating Test Suite Execution
In real life, rule applications and logic are usually far more complex than our InvoiceSample. Additionally, there can be dozens or even hundreds of tests that must be run in order to pass extensive quality assurance requirements. In such an environment, manually running regression tests can be prohibitively time-consuming, and this process is better served by automated testing.
The good news is that you can use the InRule SDK to automate your regression testing. The bare essentials of executing a test suite from code are outlined below.
1. Running our regression test requires the following using directives and references:
2. A simplistic (and non-production) command-line program to exercise the InvoiceTests test suite we created is shown below, and follows a program logic very similar to the usual pattern for executing rules in an application.
The primary required steps are:
- Load the test suite definition
- Load the rule application definition to be tested
- Create a testing session manager and regression testing session
- Execute tests
- Iterate through results and process
The output of the program above, when executed against our revised rule application, is shown below. It provides basic success/failure information equivalent to running the test suite in irVerify.
Closing Thoughts
Regression testing can quickly get very involved and complex, and we have only touched the surface of the subject in this post. For a more complete view of what regression-testing options are available in InRule, see the section on Regression Testing in the InRule Help file, or on our support site.
