Messaging Basics - Reliability

There does generally seem to be confusion amongst users about how reliable messaging works, so  I thought it would be worth going through some messaging basics again. Firstly, assume you use a hub and spoke architecture, with a broker(s) in the middle to provide location transparency and handle load for you. There are going to be a number of network hops between your Message Producer and Message Consumer:

Everything is asynchronous, we have location transparency, everything is very, very fast. But, it isn't reliable. If you care about message delivery you have to think about where things may go wrong, and even though the odds are in your favour that your messaging application will work 99.99% of the time, its always the time that it fails that you should design for.  The questions you need to think about are:

• what happens if the broker runs out of resource (memory/sockets/disk space)?
• what happens if my clients loose network connectivity?
• What happens if the hard disks crash on the broker's machine
• what happens in a power outage ?
• etc, etc.

Depending on the size of the message and the quality of the network, a  good producer can send tens of messages per millisecond - and it can take sometime (seconds or longer) before your producer realizes there's a problem - your producer client could have just lost thousands of messages and your application wouldn't even know about till its too late. So unless you have smarts built into the client to handle failure conditions (like ActiveMQ does), using asynchronous message publish to a broker isn't always the best thing to do. Its for this reason that the majority of message queue systems support a synchronous publish to the broker, so they can detect failure and resend the message. If you look at JMS, beanstalkd,Gearman, MQTT or STOMP synchronous publish to the broker is on by default. Its a sad fact that AMQP hasn't supported this basic messaging construct before 1.0 of the specification, where at least its optional. Its for this reason the ActiveMQ has not adopted AMQP till this point - you cannot have a JMS compliant product without synchronous message publish.  JMS is rightly strict about how the semantics work  - in that a receipt would not be returned to the publisher until the message has been received by the store and securely written to storage if its persistent - i.e. guaranteed delivery:

 For performance, you could cache messages, and write them to disk lazily if they haven't been consumed, however the very real downside is that the broker could disappear before the message has been stored. Your application publisher would believe the message was delivered, but this wouldn't actually be true if the broker disappeared.
BTW - some people confuse transactions with guaranteed delivery - for messaging, transactions should be used for logically grouping messages together. If you use JMS, never use a transaction to send a single message - its unnecessary.

I'm going to be talking about reliability and performance at an upcoming FuseSource Webinar next week - drop by if you have the time ;)

Extending Apache ActiveMQ with Apache Camel

Apache Camel - the excellent integration engine - can be embedded in an ActiveMQ broker to turn it into a full integration hub. Camel has probably more integration connectors with 3rd party apps than any other ESB - couple that with its ease of use and you get an understanding of why its so popular!

Camel actually started off as a sub-project of ActiveMQ - so the integration between the two is excellent (although it will be even better in ActiveMQ 6).

I blogged a while ago about the new Statistics Plugin - that you can use to request a message from the ActiveMQ broker about its running statistics (just in case you don't want to use JMX). Now it would kinda nice to automatically send these messages out from the ActiveMQ message broker periodically. Well its really easy to do that ...

Firstly enable the statics plugin in the ActiveMQ broker configuration and also import a camel config - e.g.



<beans>
   <broker brokerName="testBroker"  xmlns="http://activemq.apache.org/schema/core">
    <transportConnectors>
      <transportConnector uri="tcp://localhost:61616"/>
    </transportConnectors>
  </broker>
  <import resource="camel.xml"/>
</beans>


  
 Then in the camel XML configuration define an embedded ActiveMQ Connection -

<bean id="activemq" class="org.apache.activemq.camel.component.ActiveMQComponent" >
        <property name="connectionFactory">
          <bean class="org.apache.activemq.ActiveMQConnectionFactory">
            <property name="brokerURL" value="vm://testBroker?create=false&amp;waitForStart=1000" />
            <property name="userName" value="DEFAULT_VALUE"/>
            <property name="password" value="DEFAULT_VALUE"/>
          </bean>
        </property>
</bean>

Note: the vm:// transport (uses a broker's name to locate it in the same JVM)

Then just add a Camel route in the Camel configuration which uses a Timer  to request statistics from the broker and publish them on a Topic  every second - e.g:

<route>
   <from uri="timer://foo?fixedRate=true&amp;period=1000"/>
   <inOut uri="activemq:queue:ActiveMQ.Statistics.DestinationTest.Queue"/>
   <to uri="activemq:topic:Statistics.Topic"/>
</route>

Simple!

We are going to be on the road doing the popular  Fuse days again from September - but in the meantime - you can find out more about Apache Camel, ActiveMQ and ServiceMix from our Fuse webinars.