Riemann and Drools

Riemann (http://riemann.io/) is great tool for monitoring applications. When it comes to monitoring, there are good number for tools which help you monitor your infrastructure - things like cpu, disk, network, routers, databases etc. However, I've not seen many tools which can be used for custom application monitoring - exceptions, errors, statistics which are available inside a running program. Tools like Nagios, Zabbix and others do good job in monitoring the infrastructure, but what if you want to send your custom events, notifications or metrics ? The integration with these tools is not so easy ... usually a hack.

Custom application information is usually spit out to log files and there are separate scanners which run at periodic intervals and "grep" out this information. What we if spit out this information onto network and let something on other side collect this for us, apply filters/rules and notify us appropriately? That's exactly what Riemann does for you ...

Riemann aggregates events from your servers and applications with a powerful stream processing language. Send an email for every exception raised by your code. Track the latency distribution of your web app. See the top processes on any host, by memory and CPU. Combine statistics from every Riak node in your cluster and forward to Graphite. Send alerts when a key process fails to check in. Know how many users signed up right this second.

There is only one catch - you need to write your rules in Clojure! Clojure is lisp like language. Not many developers may know about it. What we need is a simple framework where we can set rules quickly. This is where Drools shines. Writing rules in DRL is pretty simple and intuitive. I built a simple wrapper which exports Riemann event to JSON, marshall it into Java object, insert it into Drools working memory and then apply rules. This has helped us to quickly setup custom rules.

Drools, ILOG ...

I recently had an opportunity to work on Drools. It basically involved migrating rules written in ILOG over to Drools. I had been studying rule engines lately, but never got chance to work on them. In this blog, I shall share my experiences working on Drools.

Why rule engines?

Rule engines are quite popular now-a-days, but what exactly is rule engine? Why are they better than plain programs? Rules are nothing but if ... then ... else statements. If you write these in any programming language, then at run time they would be executed one after the other in serial order. This means nth rule would evaluated only after n - 1 rules have be evaluated. Imagine you have 5000 rules in all, and the one you are interested in is at 4500th position - it would be awful lot of time before your rule is executed! 

Rules engines, on the other hand, are data driven. Data for rule engines in inserted into what is called "Working Memory". Rule engines pick those rules for whom matching data is available either partially or completely. When conditions of a particular rule(s) match completely, the rule fires; otherwise it remains in active state waiting for more data to arrive. Thus they are able to evaluate rules very fast. How do they do it? Well, they use special algorithm called RETE. This is fast pattern matching algorithm which allows rule engine to parse the rules and build an inverted tree. As the data is inserted, nodes in the tree are activated. When leaf (or root of inverted tree?) node become active - the rule fires i.e the consequence (action) of rule is executed.

There are two types of rule engines - forward chaining and backward chaining. ILOG and Drools are forward chaining mainly. Prolog on the other hand is backward chaining. What is forward chaining and backward chaining? Well, in Prolog you ask the engine if a particular consequence (or goal in Prolog) holds true? Prolog will analyse the facts and respond with correct answer. Drools/ILOG on the other hand execute the consequence as rule fires when conditions match as new data is inserted into working memory.

Preparing to migrate from ILOG to Drools

Getting back to the topic of migrating from ILOG to Drools, we need to collect the following assets. Please install ILOG rules studio and open your project.

Schemas

This is the first step in migration. Look for "schemas" folder and find all .xsd files. Use these to generate java classes to model your business domain. I use JiBX XML to Java mapping tool to automate code generation. Following sample command should help you generate java code ...

java -cp ../../lib/jibx-tools.jar org.jibx.schema.codegen.CodeGen
 otasubset/OTA_AirLowFareSearchRQ.xsd

Custom Functions

Look for "BOM" (Business Object Model) folder under your eclipse project. This contains objects modelling your business domain. ILOG rule authors add custom functions inside model classes. You will need to migrate these custom functions as well. Usually simply copy-and-paste these functions in above generated classes should work fine.

Flows

Look for .rfl files. Usually there should be only 1 flow file per project. It will graphically depict the order of execution of rules. You can choose to use Drools Flow or Drools Agenda to map them to Drools environment.

Rules

Finally the actual rules themselves. Look for "rules" folder. These will contain actual rules to migrate. In ILOG these could be files with .irl suffix. These files have to be manually rewritten to Drools rules. 

You will also need to migrate decision tables - files with .dta suffix. These can be easily converted to Drools decision tables.

Input/Output

After migrating above four components, you are ready to run the actual rules. In ILOG you'd be sending your input as a SOAP message. This message contains facts which ILOG engine marshall into domain objects (i.e POJO(s)). Facts are then inserted into working memory and rules are executed. The results are also stored in memory as objects (POJO); which can be serialized into XML. This XML can be returned back to caller as result.

Inp 

-> SOAP XML message 

 -> POJO(s) 

  -> Working Memory 

   -> Run Rules 

    -> POJO(s) 

     -> XML 

      -> Out

Marshalling and unmarshalling of java objects to XML can be easily done using JiBX library. It is pretty fast in reading XML and populating POJO(s). It also does good job in mapping POJO(s) back to XML.

Hope this helps! I will add more in near future ...