Home » Clustering

Choosing a communication library for a distributed application

3 January 2012 4 Comments

Together with some colleagues I started to put the base for a distributed application. Everything is looking good, but we have a big dilemma. We don’t know what communication API to choose. In first phase we didn’t have so many options to choose, but after some research, we found several good communications libraries, and now, is very hard for us to select the best solution.

A few words about this new project.

The main idea behind this distributed project is to have a frontend, a backend (processing application) and hundreds of agents which will collect data from some datacenters. The communication between agents and backend should be a simple as possible, we should have possibility to push configuration or to pull real time data from agents. A minimal security of the transport between agents and backend is required. Also we should have possibility to add/delete new nodes in backend cluster live. If the number of nodes of backend cluster is changing, the agents should redistribute their load (probably this should be made by backend through filtering). Usually the messages (between agents and backend) will not exceed 1500 bytes, but from time to time we should be able to send large packets (a good fragmentation management is a must).

What we need?

A communication library which should support reliable communication, groups/channels, Point-To-Point and Point-To-Multipoint messages, auto-discover services, security, supported by multiple languages and multiple operating systems.

What we found until now ?

  • jGroups
  • Spread
  • openpgm
  • 0mq
  • Hazelcast

1. jGroups

Every experienced programmer in Java knows jgroups. jgroups is a reliable multicast communication written in and for Java. From a lot of aspects jgroups is great,
but is missing a thing, doesn’t support other programming languages (just Java). Probably we can write a library based JNI to export the jgroups classes in C++,
but we don’t know if the time to do that is really worthing. Anyway the library can inter-operate with other programs through STOMP, but is not very reliable.

2. Spread

The idea behind Spread is to have a spread server on each machine that is part of the processor group. Each server runns it’s own instance of spread server and
the local applications are connecting to this instance and pass messages to the server which will distribute the message to all spread servers from the network.
Spread is supporting C++ and Java API, but it seems is not any more developed.

3. openPGM

openPGM is just a multicast reliable implementation. is supporting our principals demands, but is raw. We need to develop everything related to clustering logic.

4. zeromq (0mq)

0mq is using openPGM for it’s need’s but is working also on tcpip/ipc. The library doesn’t support auto-discover services and can’t return a list of all
conected peers which is an important feature for distributed applications. From other aspects of our problem, the library seems to be a very good choice.

5. Hazelcast

Hazel cast is not a “communication library”, is a in-memory data grid, but can solve a lot of problems in our projects. But, at this time, they support only Java.

Do you have a suggestion? Do you know other libraries? Please drop me a message with your opinion regarding this problem.


  • Andritoiu said:

    We have our own one which is used by O3S (http://www.opentekhnia.com/?page_id=124). If you are still looking for, just send an email. Is proprietary code based on Poco C++, but we can deliver sources also.

  • admin (author) said:

    I will take a look. Thanks.

  • Sean R Owens said:

    I know it’s been six years, I’m curious what you eventually settled on, and how well it worked.

  • admin (author) said:

    Dead project. 🙂 Now I’m almost in the same situation,. and I don’t know what to choose, Kafka or RabbitMQ.

Leave your response!

Add your comment below, or trackback from your own site. You can also subscribe to these comments via RSS.

Be nice. Keep it clean. Stay on topic. No spam.

You can use these tags:
<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

This is a Gravatar-enabled weblog. To get your own globally-recognized-avatar, please register at Gravatar.