Event Processing and Streaming Query Engines
August 27, 2010 § 2 Comments
I just went over a nice article in the January issue of the Communications of the ACM, Data In Flight by Julian Hyde (he maintains a great blog by the way). I think that it is a good read for anyone questioning the added value of Event Processing software in general and CEP (Complex Event Processing) in particular.
The author argues successfully that Streaming Query Engines are at the heart of Event Processing applications and quickly recaps their advantages over a relational database, for example, when processing events:
- Event processing is rarely concerned with transactions, as such streaming query engines store queries only, not data, which makes them very efficient
- Streaming query engines are concerned with the latest data and keep very little history in memory: This varies from a few seconds to at most a business day worth of data, which relieves the engine from the need to maintain a persistent storage and support ad-hoc querying – of course this is just the typical usage, not a hard rule
- Streaming query engines process data asynchronously to achieve the highest level of performance
It is worth noting here that stream-oriented languages are just one flavor of event processing languages. IBM WebSphere, for example, uses a rule-oriented language (WebSphere Business Events). Understandably the JBoss offering in this space (DROOLS Fusion) uses a rule-oriented language too. Just to be clear, these two products are event processing software, not BRMS (Business Rule Management System) software. They may use a rule-oriented language to act on events, but they differ from a BRMS product where a) rules are invoked by requests made from within the application (as opposed to event processing where the processing is triggered by the occurrence of events or situations) and where b) rules operate on the various states (see for example http://www.jboss.org/drools/drools-flow.html).
I did not conduct an exhaustive survey but from my own experience I would venture to say that stream-oriented languages are the more popular ones out there to express event processing logic. That’s because they naturally lend themselves to express time constructs as a fundamental dimension of event processing. Stream-oriented languages do that by typically introducing time operators to SQL-like languages. Here is a list of a few vendors that went down this route:
The list above gives a flavor of the various Streaming Query Engines out there. There is enough value in the expressiveness of the languages alone to warrant their adoption as opposed to rolling your own half-baked event processing solution based on, say, pattern matching of messages on a bus. As soon as the rate of arrival/dissemination of the messages increases and the need to correlate between various out-of-order messages over variable windows becomes pressing I think that it makes sense to consider an event driven application.
