I happened to run across a new open-source project today I had not previously heard about: Hazelcast.

What really piqued my interest was the implementation of a distributed ExecutorService as I wrote something of similar functionality to façade Orbitz’s Jini infrastructure. The design and implementation solved a couple primary objectives:

• add timeouts to Jini which is unfortunately lacking such a feature
• bound the number of concurrent requests being processed
• bound the number of threads created for request processing

The design was elegant, imo, because the exact same code worked either client or service-side – it just mattered which way you twisted your head – and masked the complexities of both Jini and the ExecutorService. The timeouts were managed via the Future and the throttling of requests and threads by configuring the backing Queue and pool size. Spring wiring entirely hid the remote invocation machinery from the caller. In almost every case there were no code changes and the timeout and throttling features could be turned on and tuned entirely through a configuration change.

The primary flaws in the implementation were in the difficulty of passing around ThreadLocal required context but that’s a pain-in-the-ass regardless and should be avoided if possible. The other concerned a slightly awkward callback mechanism for managing the timeouts. Hazelcast returns the Future directly to the caller but we choose to abstract this away so the caller coded to an interface which offered nothing about the possibility of being invoked remotely. To compensate, a callback mechanism could at runtime change any pre-configured timeout values based on the interface and/or the parameters of the invocation.

I’ve spoken before about the trade-off of abstracting remote invocations. On one hand it ensures discovery, error handling and invocations are accomplished consistently but on the other it invites developers to ignore the realities of a distributed system which can lead to the if-it-looks-local-it-will-be-coded-as-though-it-is-local problem. Dan Creswell offers:

I believe the best chance we have for doing distributed right is not by providing some de-facto standard toolset, rather it’s through education and mentoring to encourage the correct mindset. Such a mindset allows a developer building a distributed system to choose the most appropriate tools and use them right.

I agree that’s definitely the best long-term solution but in the meantime the site needs to be up.