Tuesday, August 28, 2012

An old crack at a stream processing framework. WAS: Building procedures with composition

Often developers need to model a multi-step process.  Also common is the need to reuse pieces of the process:  maybe the steps are the same but the execution of the steps vary for different contexts, or maybe the step flow changes slightly in some cases but the logic of the steps is largely reused.

One common way developers address this situation is to implement a Template Method pattern.  For example, an insurance system might have several similar flows for processing feeds from different sources--partner EDI, consumer web site, provider applications--where many of the steps involved are largely reusable.

A Template Method basically uses inheritance to share common logic, while varying the implementation of steps by overriding methods representing particular steps.  However as the GOF posited and leading developers believe today, one should "favor composition over inheritance."  Inheritance abuse is a common and natural side effect of trying too hard to reuse logic the OO-way, but we can do better.

The essential problem with using inheritance with the Template Method is that the class tree can get pretty ugly if one needs to vary the implementation of steps AND the flow of steps.  Also, more often than not, the is-a relationship between common and specialized processors is restrictive and faulty.  For example if a processor implementation wanted to extend something else like ActiveRecord.

What if, rather than using a template method, one could compose flows of steps, where the steps are pluggable members?  Another way to build processes that can be reused and extended is with the Strategy pattern, where the algorithm (or steps thereof) are pluggable.  Using this approach, the developer creates the structure of the flow in one class but delegates the execution of steps to collaborators.

Now what if there was a fluent interface for building flows of plugable steps, as well as machinery and conventions for coordinating between steps?  Well, that is the idea of Flo:  https://github.com/jtroxel/flo
Some of the ideas from Spring Batch and Apache Camel have also inspired my thinking here, but Flo is much simpler and less specialized.  I was also motivated by the Unix programming model, little tools that do one thing well, strung together with pipes.

Note that Flo is a work in progress, awaiting her first real application.

No comments: