FuseSource in Numbers

• 1 million+ transactions every minute running on Fuse software
• 200,000 downloads a month
• 200+ enterprise customers and growing rapidly
• 24 hr Enterprise support - every day 
• 20 + Apache committers and contributors
• 15 + years of Enterprise Integration experience (ok slightly more - but we lie about our age)
• #1 provider of Open Source Integration and Messaging

FuseSource software is based on Apache ServiceMix, ActiveMQ, Camel and CXF.

We wrote the code!

Podcast on Apache ActiveMQ and FuseSource

I recently got the chance to ramble to James Denman from SearchSoa about Apache ActiveMQ, ActiveMQ in Action and a little bit about the FuseSource launch. You can listen to it on the SearchSoa site here.  There's also a FuseSource webinar on ActiveMQ in the Enterprise - if you want to get a flavour of some of the topics covered in the ActiveMQ in Action book.

ActiveMQ in Action in Production!

The ActiveMQ in Action book, that has been a background part of my working life for the past couple of years has finally gone into production. Its actually bang up to date - and covers features that are applicable to the latest Apache ActiveMQ 5.4 releases.

To celebrate - Manning are giving 40% of your purchase of ActiveMQ in Action - just use the coupon code *activemq40* at the time of checkout.

FuseSource has launched!

FuseSource is now operationally independent - and has been spun out as a separate legal entity from the Progress Software Corporation - to emerge as the world leader in open source integration and messaging solutions for the Enterprise. Although FuseSource will remain a 100% owned subsidiary of Progress Software, it does mean we are bucking the current trend of consolidation in the open source industry and been given the freedom to concentrate on our own destiny.

FuseSource provides enterprise support for its open source core products, Apache ServiceMix, ActiveMQ, Camel and CXF, as well as providing products to better enable production deployment and management around those projects.

We actually started off over 5 years ago, as LogicBlaze - a start-up intent on providing the best open source integration solutions for developers. We had already been working on ActiveMQ and had just started on ServiceMix. We moved both of these projects to Apache, because we wanted to provide truly open solutions for our users and customers. Open source development should never be about simply the src code - its the community that matters, and the Apache Software Foundation simply brings about the best environment for open source. The Apache licence ensures that anyone can use their projects without fear (in contrast to copyleft licences) and the diversity in the community of Apache developers ensures that an Apache project will never be held hostage to the fortune of any single company or individual.

As a VC funded start up LogicBlaze came to the point in 2007 when it needed a new round of funding to grow the business, and at that time, although we had lots of customers, to gain a foot hold into providing enterprise integration for large organisations we really needed to  partner with a larger organisation who had the enterprise pedigree to back us up.  We had a lot of enterprises looking at our solutions, but they all wanted the assurance that we were still going to be around in 5-10 years before committing to a subscription. This was at a point in time just before open source was seen as a viable option for anything other than operating systems or project based databases for large organisations. A lot has happened in the last few years to change that view, but this was back in the day.  This convinced me of two things:  The Apache Software foundation was the right place to home open source integration solutions; and we really needed to be part of a larger organisation to start making inroads into large enterprises. Ideally for us, we wanted to be a subsidiary, to be independent, to be able to decide our own sales and marketing strategy, but have the financial clout and backing of a plc.

Actually at that time there weren't many companies we would have been happy to join, and we pretty much convinced that we would have to continue as a venture backed start-up - but we just happened to meet the folks at IONA Technologies - and there was an instant cultural fit. IONA had its own open source story, which ended up being Apache CXF which they were using to supplement their own licence based integration solution. We realised there would be potentially a good technical fit as well as a cultural one - and we were acquired in April 2007.We were looking at the potential of what we could grow together - and with IONA's CXF added to the project portfolio, we created a new way to glue everything together called Apache Camel - which is probably now the most widely used integration glue anywhere - and we were renamed as the Fuse group.

What we didn't realise was the extent of the financial storm that was about to hit the economy, and it 2008 become a very hard time for a lot software corporations, including IONA, and they were acquired by Progress Software in September 2008. I think its fair to say that Progress weren't entirely sure what to do with FUSE and how to market it effectively along side their own products - and Progress owned a lot of products! Progress have been going through a lot of changes in the last 2 years, as they have re-invented themselves away from being the owner of lots of disparate application and data integration solutions, into a cohesive solution for operational responsiveness.  I think its fair to say that Progress wasn't sure where Fuse should fit in, and we weren't sure if we should try and fit into the Progress story.

However, some hard facts shone through about the Fuse business. Fuse has managed over 100% growth year on year, has more than 200 large enterprise customers world wide (3 in the top Global 10) and more than 1million downloads of our software a year. Progress realised that Fuse should be seen as a strategic asset, especially with the big inroads we have made into large enterprises in past couple of years - its just that our business and culture is somewhat orthogonal to Progress Software.

I give much kudos to the Progress executive team for deciding the best way forward is for Fuse to become a separate legal entity, but wholly owned subsidiary - known as FuseSource. We have a great management team, exceptionally talented technical team and superb and supportive customers - and we have the full backing and financial support of Progress to continue to grow our business. We have a really innovative roadmap, not just for the Apache projects we contribute to, but also for our Fuse based products - which we will start rolling out through 2011!

Enterprise ActiveMQ Webinar

I'll be talking about using ActiveMQ in the Enterprise on Thursday 19th August in a live webinar.

As well as covering failover, high availability and scaling - I will also cover features in the ActiveMQ 5.4 and give some insight into the new architecture and features for ActiveMQ Apollo - the messaging system currently being built to meet the high demands of messaging for the next decade.

Messaging Basics - Reliability

There does generally seem to be confusion amongst users about how reliable messaging works, so  I thought it would be worth going through some messaging basics again. Firstly, assume you use a hub and spoke architecture, with a broker(s) in the middle to provide location transparency and handle load for you. There are going to be a number of network hops between your Message Producer and Message Consumer:

Everything is asynchronous, we have location transparency, everything is very, very fast. But, it isn't reliable. If you care about message delivery you have to think about where things may go wrong, and even though the odds are in your favour that your messaging application will work 99.99% of the time, its always the time that it fails that you should design for.  The questions you need to think about are:

• what happens if the broker runs out of resource (memory/sockets/disk space)?
• what happens if my clients loose network connectivity?
• What happens if the hard disks crash on the broker's machine
• what happens in a power outage ?
• etc, etc.

Depending on the size of the message and the quality of the network, a  good producer can send tens of messages per millisecond - and it can take sometime (seconds or longer) before your producer realizes there's a problem - your producer client could have just lost thousands of messages and your application wouldn't even know about till its too late. So unless you have smarts built into the client to handle failure conditions (like ActiveMQ does), using asynchronous message publish to a broker isn't always the best thing to do. Its for this reason that the majority of message queue systems support a synchronous publish to the broker, so they can detect failure and resend the message. If you look at JMS, beanstalkd,Gearman, MQTT or STOMP synchronous publish to the broker is on by default. Its a sad fact that AMQP hasn't supported this basic messaging construct before 1.0 of the specification, where at least its optional. Its for this reason the ActiveMQ has not adopted AMQP till this point - you cannot have a JMS compliant product without synchronous message publish.  JMS is rightly strict about how the semantics work  - in that a receipt would not be returned to the publisher until the message has been received by the store and securely written to storage if its persistent - i.e. guaranteed delivery:

 For performance, you could cache messages, and write them to disk lazily if they haven't been consumed, however the very real downside is that the broker could disappear before the message has been stored. Your application publisher would believe the message was delivered, but this wouldn't actually be true if the broker disappeared.
BTW - some people confuse transactions with guaranteed delivery - for messaging, transactions should be used for logically grouping messages together. If you use JMS, never use a transaction to send a single message - its unnecessary.

I'm going to be talking about reliability and performance at an upcoming FuseSource Webinar next week - drop by if you have the time ;)

Extending Apache ActiveMQ with Apache Camel

Apache Camel - the excellent integration engine - can be embedded in an ActiveMQ broker to turn it into a full integration hub. Camel has probably more integration connectors with 3rd party apps than any other ESB - couple that with its ease of use and you get an understanding of why its so popular!

Camel actually started off as a sub-project of ActiveMQ - so the integration between the two is excellent (although it will be even better in ActiveMQ 6).

I blogged a while ago about the new Statistics Plugin - that you can use to request a message from the ActiveMQ broker about its running statistics (just in case you don't want to use JMX). Now it would kinda nice to automatically send these messages out from the ActiveMQ message broker periodically. Well its really easy to do that ...

Firstly enable the statics plugin in the ActiveMQ broker configuration and also import a camel config - e.g.



<beans>
   <broker brokerName="testBroker"  xmlns="http://activemq.apache.org/schema/core">
    <transportConnectors>
      <transportConnector uri="tcp://localhost:61616"/>
    </transportConnectors>
  </broker>
  <import resource="camel.xml"/>
</beans>


  
 Then in the camel XML configuration define an embedded ActiveMQ Connection -

<bean id="activemq" class="org.apache.activemq.camel.component.ActiveMQComponent" >
        <property name="connectionFactory">
          <bean class="org.apache.activemq.ActiveMQConnectionFactory">
            <property name="brokerURL" value="vm://testBroker?create=false&amp;waitForStart=1000" />
            <property name="userName" value="DEFAULT_VALUE"/>
            <property name="password" value="DEFAULT_VALUE"/>
          </bean>
        </property>
</bean>

Note: the vm:// transport (uses a broker's name to locate it in the same JVM)

Then just add a Camel route in the Camel configuration which uses a Timer  to request statistics from the broker and publish them on a Topic  every second - e.g:

<route>
   <from uri="timer://foo?fixedRate=true&amp;period=1000"/>
   <inOut uri="activemq:queue:ActiveMQ.Statistics.DestinationTest.Queue"/>
   <to uri="activemq:topic:Statistics.Topic"/>
</route>

Simple!

We are going to be on the road doing the popular  Fuse days again from September - but in the meantime - you can find out more about Apache Camel, ActiveMQ and ServiceMix from our Fuse webinars.

ActiveMQ 5.4: JMS Timer and CRON Scheduler

I've added a persistent scheduler engine to ActiveMQ, so that messages that are scheduled for delivery will always survive restarts. The scheduling of some message in the future can be useful - but deciding  a scheduling syntax that is flexible, but can also fit into Message header properties is a little tricky.  I've decided the best approach is to use a combination of different concepts - simple delay, repeats and CRON.

There are a few choices of around scheduling messages to deliver in the future(inspiration from java.util.Timer)  - a simple delay - this one will deliver the message in a minute:

long time = 60 * 1000;
message.setLongProperty(ScheduledMessage.AMQ_SCHEDULED_DELAY, time); 

producer.send(message); 

Delivery at a fixed rate - this example will delivery 10 messages, waiting 10 seconds between each message - note the repeat count is 9 (9 + first one = 10).
long delay = 30 * 1000;
long period = 10 * 1000;
int repeat = 10;
message.setLongProperty(ScheduledMessage.AMQ_SCHEDULED_DELAY, delay);
message.setLongProperty(ScheduledMessage.AMQ_SCHEDULED_PERIOD, period);
message.setIntProperty(ScheduledMessage.AMQ_SCHEDULED_REPEAT, COUNT repeat);
producer.send(message);


For more advanced messaging you can use CRON scheduling - this one will deliver a message at 2AM on the 12 day of every month:

message.setStringProperty(ScheduledMessage.AMQ_SCHEDULED_CRON, "0 2 12 * *");
producer.send(message);

You can combine delays and repeats with CRON. For example, you may wish to send 10 messages, every 30 minutes, with a delay of 10 seconds between each one.

message.setStringProperty(ScheduledMessage.AMQ_SCHEDULED_CRON, "30 * * * *");
message.setLongProperty(ScheduledMessage.AMQ_SCHEDULED_DELAY, 10*1000);
message.setLongProperty(ScheduledMessage.AMQ_SCHEDULED_PERIOD, 10*1000);
message.setIntProperty(ScheduledMessage.AMQ_SCHEDULED_REPEAT, 9);

As I quickly realised whilst testing -  having a persistent message scheduler that survives re-starts really needs management support - so that you can delete those pesky jobs that you accidentally scheduled ;). So there's JMX support - and the ActiveMQ Web Console has been updated too:

A funny thing happened to my ActiveMQ Performance

In ActiveMQ we have a message cursoring system to ensure that there is no memory restriction on the size of the queues we use. However, we do cache messages to improve performances - as depicted below:

The cursor cache is limited by memory constraints that are defined in the ActiveMQ configuration. Usually the cache is valid, but if memory constraints hit, - or consumers lag to far behind producers, the cache is cleared down. However for every message dispatched through the cursor after the cache is cleared,  the cursor will check if the Queue is empty in the store to determine if it can start using the cache again.

This all sounds eminently sensible. However, the code to check if the Queue was empty in the store called for a count on the outstanding messages. In order to determine the size of the Queue, the BTree index used for KahaDB would iterate through every BTree node and, er count them. ActiveMQ would do this for every message dispatched after the cache became invalid.

Just to labour the point, for every message dispatched, every BTree node for the BTree index would be paged into memory in the same transaction  and counted ....

Although this is an issue with some serious side-affects - like being a memory hog when using KahaDB and seriously affecting performance -  it does highlight that for distributed systems there are always going to be behaviour  that you can't always anticipate.

The reason I found this issue so interesting is because it wasn't obvious - its an edge case that happens under only under certain load conditions with certain configurations - but was very straightforward to fix - which always makes me smile :)

This issue only affects KahaDB and is fixed in trunk, the ActiveMQ 5.3.1 branch and Fuse Message Broker versions - and for those interested - the code change is here.

Apache ActiveMQ 2009 in Numbers

So Claus started this with his blog on Apache Camel - 2009 in numbers - so I'd thought it would be interesting to do the same with ActiveMQ (does not include ActiveMQ C#, C++ or Stomp sub-projects). The numbers are not as good as Camel - but still good:

Number of posts to the ActiveMQ user forum in 2009: 4440
Number of posts to the ActiveMQ dev forum in 2009:  4447
Number of commits in 2009:  2893
Number of downloads 2009: - hard to know (because of Apache mirrors) - but about 60k a month

Number of tickets created in 2009 - 505 (295 of these are resolved)
Number of resolved issues in 2009 - 467

We didn't quiet manage to resolve more issues than were raised - but this is because as a community, until very recently we have been very lax at accepting issues - so a lot of issues raised don't even have the version, let alone the environment or a test case! If a test case is attached - we can usually resolve an issue very quickly - its reproducing the issue that always takes the time.