Tuesday, December 17, 2013

One Technology Trend for 2014: "The Internet Of Things"

I was reading some online articles and came across Technology Trends for 2014 - the number one being the 'Internet of Things' or IoT for short. This isn't exactly a new concept, - the promise of smart homes, where everything from intelligent lights to A.I. for washing machines that can be monitored remotely have been around for a while. And how couldn't resist the concept of a smart fridge that can stock itself? The term Internet of Things has been around for over decade, being firstly proposed by Kevin Ashton whilst Auto-ID Center at MIT, primarily driven by an interest in RFID, but the ideas and uses cases for an Internet of Everything has taken a while to mature.

The drivers behind the IoT have been several:  The demand for renewable energy means that smart grids have to be able to monitor and respond to demand in electricity generation in a more agile manner - allowing for bi-directional energy supply from small energy producers (potentially you and me) - requires smart metering and monitoring. There's the exponential growth of smart phones  - more people are always connected, and that trend will continue.

However, when the IoT was first envisaged all those years ago, there were some technology inhibitors:

1. The limitation of IPv4 in terms of the number of physical addresses that were available
2. The capacity of the internet for a fully connected IoT
3. The ability for mediators to scale to millions of concurrent connections
4. The ability to  store and analyse the data in a scalable way
5. The ability to analyse all the data to make sensible decisions in a timely manner.

Fast forward to today and we have most of these things from a technology perspective either solved (e.g. IPV6) or the pieces are available, and Red Hat is ideally placed to provide the whole solution for a scalable backend for the IoT - and to do it all on open source software.

Firstly, we need the ability to provide a standards based, horizontally scalable solution for handling connectivity to hundreds of thousands of concurrent connections. JBoss A-MQ is combining the best of Apache licensed middleware solutions from Apache ActiveMQ, QPid and HornetQ to form a highly scalable messaging solution that supports MQTT, AMQP,  WebSockets and STOMP.

The IoT will generate a lot of unstructured data, which needs to be correlated and analysed, and one of the leading NoSQL solutions for doing this is Hadoop. If you want Hadoop to scale and perform, then bester infrastructure to run it on is a combination of GlusterFS and OpenStack.

Getting real time data into Hadoop's HDFS can be problematic, but  JBoss Fuse already has some of the best solutions for doing just that.

Finally, if you want to use complex event processing, to make decisions based on the flow of data from your connected devices based on causality and temporal logic, then JBoss BRMS is the best open source solution on the market.

Red Hat  is going to be right at the centre of  IoT solutions in 2014.

Friday, September 06, 2013

Apache Camel Broker Component for ActiveMQ 5.9

Embedding Apache Camel inside the ActiveMQ broker provides great flexibility for extending the message broker with the integration power of Camel. Apache Camel routes also benefit in that you can avoid  the serialization and network costs of connecting to ActiveMQ remotely - if you use the activemq component.

One of the really great things about Apache ActiveMQ is that it works so well with Apache Camel.

If however, you want to change the behaviour of messages flowing through the ActiveMQ message broker itself you will be limited to the shipped set of ActiveMQ Broker Interceptors - or develop your own Broker plugin - and then introduce that as a jar on to the class path for the ActiveMQ broker.

What would be really useful though, is to combine the Interceptors and Camel together - making it easier to configure Broker Interceptors using Camel routes - and that's exactly what we have done for upcoming ActiveMQ 5.9 release with the broker Camel Component. You can include a camel.xml file into your ActiveMQ broker config -  and then if you want to take all messages sent to a Queue and publish them to a Topic, changing their priority along the way - you can do something like this:

A few things worth noting:

  • A broker component only adds an intercept into the broker if its started - so the broker component will not add any overhead to the running broker until its used - and then the overhead will be trivial.
  • You intercept messages using the broker component when they have been received by the broker - but before they are processed (persisted or routed to a destination).
  • The in message on the CamelExchange is a Camel Message, but also a JMS Message (messages routed through ActiveMQ from Stomp/MQTT/AMQP etc. are always translated into JMS messages).
  • You can use wildcards on a destination to intercept messages from destinations matching the wildcard.
  • After the intercept, you have to explicitly send the message back to the broker component - this allows you to either drop select messages (by not sending) - or, like in the above case - re-route the message to a different destination.
  • There is one deliberate caveat though,  you can only send messages to a broker component that have been intercepted - i.e.  routing a Camel message from another Component (e.g. File) would result in an error.
There are some extra classes that have been added to the activemq-broker package - to enable views of the running broker without using JMX - and to support the use of the broker component:
org.apache.activemq.broker.view.MessageBrokerView - which provides methods to retrieve statistics on a the broker, and from the MessageBrokerView - you can retrieve a org.apache.activemq.broker.view.BrokerDestinationView for a particular destination. This means you can add flexible routing inside the broker by doing something  like the following - to route messages when a destination's queue depth reaches a certain limit:

This is using the Camel Message Router pattern - note the use of Spring expression language (spel) in the when clause.

Wednesday, August 21, 2013

Fuse days are back

One thing I get constantly asked about is the Fuse days that FuseSource used to do around Europe and the US, and now FuseSource is part of Red Hat would they still be happening? Well the answer is an emphatic YES! After taking some time to settle in and find out where the tea bags are hidden in the Red Hat middleware group its time to start things rolling again. We have been working out the messaging and integration strategy and will be have an engineering face to face meeting in Dublin, Ireland in week beginning the 23rd September 2013. Its short notice - but we could hold an impromptu Fuse day in Dublin that week.
You may even get to find out what we are doing in 2014 before the engineers!

Drop me a line if you want to attend - it'll be free - you just have to get yourself to Dublin. I'll be posting dates for upcoming Fuse days in Europe and the US over the next couple of weeks - now where's my cup of tea ...

Friday, May 31, 2013

Connecting Applications Everywhere with ActiveMQ

This year at CamelOne there's going to be some exceptional presentations,  but I'm also presenting "Connecting Applications Everywhere with ActiveMQ".

The focus of this presentation is to demonstrate the many protocol options and deployment scenarios that are available to Apache ActiveMQ. After an introduction to the Apache ActiveMQ project, and presenting why the "Internet of Things"is going to be driving the agenda for integration and messaging over the next 5 years, I'll be demonstrating an example application, going from an Arduino Microprocessor using MQTT to a MQTT/AMQP ActiveMQ gateway and to an ActiveMQ broker that will service HTML clients over WebSockets - something like this:

With so many linked components in a live demo - what could go wrong ? ;)

I'm hoping to catch up with lots of folks at CamelOne - if you haven't registered, its not too late!


Unfortunately at this year's CamelOne  there wasn't any video recordings of sessions. Sharing slides is easy, but you do a demo you need video. Luckily I managed to re-create the whole CamelOne presentation and demo in a DevZone webinar - the video of which is below:

Wednesday, May 22, 2013

What to look forward to at CamelOne, June 10-11th, 2013!

This year is going to be the third CamelOne, and its going to be quite different from previous
CamelOne conferences.

Firstly the we have a new host, Red Hat who have kindly agreed to host CamelOne at the Hynes Convention Center, at the same time as JudCon and Red Hat Developer Exchange. This means people attending will be able to move between these different events and pick n' mix what they go to.
It also means there will be a chance for attendees to mingle - and see what's happening on both sides of the open source fence. 

The second thing you'll notice, is that is going to be a very strong emphasis on open source projects. I've no doubt the occasional product may get a mention, but the aim of CamelOne this year is to educate and share experiences of using the best open source integration software out there.  If you look at the agenda, you will see there's a real mix of customer experience stories and best way to use the Apache projects to be successful.

Thirdly, the overall theme is going to show how you the direction integration projects from the ASF (Apache Camel, ActiveMQ, CXF, Karaf, ServiceMix) are taking to address the integration needs for the next five years: The internet of things, the proliferation of cloud API's, and mobile. Attendees will also see the direction that the Fuse Engineering team inside Red Hat are developing for future projects, in particular better management and  Cloud-based Integration.

This is going to be the best CamelOne yet - we are expecting record attendance - and I look forward to seeing you there!

Thursday, February 28, 2013

Get ready for CamelOne 2013!!

Red Hat have kindly agreed to host the CamelOne event this year in Boston, MA from June 10th to 11th, and are announcing a call for papers. CamelOne is the best conference to hear about open source integration and messaging projects from the Apache Software foundation. There will be talks about  Apache CamelActiveMQServiceMixKaraf and CXF. To get a flavor of the conference, you can view last years sessions here (registration required).

Monday, February 04, 2013

What's the Future for Fuse at Red Hat ?

Its been just over five months since Red Hat acquired FuseSource, and that isn't a long time for an acquisition to find its feet, but Red Hat is an open source company, and shares a lot of the same culture and values that FuseSource did, which has made thing considerably easier.

FuseSource was a separate legal entity and run successfully as an independent company, but its biggest share holder was its former host Progress Software. When Progress lowered its sights last year, and changed its strategy, it very publicly announced its intention to divest a large number of its products including SonicESB, SonicMQ, DataXtend, Savvion BPM, Orbix etc. FuseSource was also bundled into that mix. To be honest, being put up for sale by the good folks at Progress was a shock: FuseSource was growing fast, had a large number of top brand customers and was out performing its open source competitors by some margin. You would expected revenues to fall off a cliff - but for us they didn't - we kept growing and that trend is accelerating now we part of the JBoss family at Red Hat.

What has caused some confusion is that FuseSource had well established products for messaging and integration, but so does Red Hat JBoss. There was an assumption that these products will get folded together,  but they actually address different users. JBoss has a comprehensive SOA story, but SOA isn't integration:

Fuse ESB Enterprise (which is based on Apache Camel, ActiveMQ, CXF and Fuse Fabric) will be able to work in conjunction with the JBoss SOA platform, and they will both share some of the same open source technologies. Fuse ESB Enterprise is going to be rebranded JBoss Fuse, for the upcoming release in Q2 this year, and as previously announced, the developer tooling has now been open sourced:

We aren't going to stop at rebranding though, development of future enhancements to the JBoss Fuse product have already started, its a product that will continue to evolve and has a very bright future a head of it.

Fuse MQ is also getting re-branded to JBoss A-MQ. The release in Q2 of this year is going to be based on Apache ActiveMQ 5.8, the biggest change to this release is the addition of AMQP 1.0 support:

Red Hat JBoss has more than one messaging product and over time this is going to get more consolidated, as we look to provide a messaging architecture that will last to 2020 and beyond. There is an accelerating demand for high performance, lower latency and scaling out to millions of concurrent connections. To be able to do that in a way that leaves none of our existing customers and users behind is going to take a while to get right - but we will do it.

So what's the future for Fuse at Red Hat ? A great one!

Friday, February 01, 2013

History of Open Source

I gave a talk last year at Walmart HQ about Fuse open source integration and messaging, the benefits of using open source and the benefits of contributing to the Apache Software Foundation. You can find a sanitized version on slideshare.net.

I also covered the history of open source, and like all histories there is no one 'right' story, but I tried to distill the timeline as best I could. On the chance this may be useful to someone else I tell the story here. Are you sitting comfortably? Then I'll begin ...

The IBM 704 was the first mass produced computer with floating point arithmetic hardware and was introduced to the fledgling computer market back in 1954. IBM managed to sell a whopping 123 of these beasts between 1955 and 1960. IBM bundled these machines with free software and source code - and in 1955 the organization SHARE was formed to allow like-minded individuals to share and swap code for the 704 machine. SHARE is the first public open source organization, and is still around today.

Everything was really rosy in open source land (though not good for IBM competitors) - until in 1969 the Department of Justice filed a suit alleging that IBM was monopolizing the computer market. The suit dragged on for 13 years, but the effect of the filing was that in 1969 IBM unbundled its software from distribution with its computers. Until this point, the software was free and the source code was distributed too.

Now we diverge in to the 70's-80's, and the UNIX wars, which are only slightly less hard to follow than the War of the Roses - so I'll try and distill the salient points.

UNIX started its development in the 1960's as Multics, at Bell Labs, which was then part of AT&T. Originally written in assembly language, it was ported in 1972 to the C programming language - allowing it to become more portable. AT&T were banned from selling computers and computer software - as result of an antitrust case brought back in the 1950's. UNIX was freely licensed to Universities and educational establishments, and became the Berkley Software Distribution (BSD) - first released by University of Berkley in 1984.

On a tangent timeline, in 1983 Richard Stallman creates GNU (GNU is not UNIX) which will lead to the Free Software Foundation and the the GPL (and LGPL) licenses.

Jumping back to 1984, US regulators broke up AT&T into regional telcos (the "baby bells"), but also allowed AT&T to enter the computing and software market, and to start selling its own distributions of AT&T UNIX, which it did aggressively.  When AT&T sued Berkley in the early 90's for license violations, the UNIX market was left in disarray.

From every conflict, there is opportunity, and Linus Torvalds' development of LINUX in 1991 started an open source fire, which ultimately led to the most successful open source company, Red Hat, coming into existence.

The Apache HTTP server started in 1994 from the requirement to maintain the old NCSA HTTP daemon. This collaboration led to the Apache Software Foundation in 1999.

The biggest revolution in open source has come about through the creation of GitHub (2008) - which just narrowly gets my vote as being the most important company in open source development, primarily because it uses social networking techniques to encourage collaboration - and in just a few short years has over 1 million users and over 2 million repos.

So we come full circle, with Gartner predicting that open source software will be included in mission critical software portfolios of 99% of the Global 2000 enterprises.

Sunday, January 20, 2013

Timeline of Fuse projects

We often get asked to describe the history of FuseSource (now part of Red Hat), and the projects that we have created/supported. I often use the following picture in presentations, to describe the timeline:

So the next time I get asked - I can point them to this blog post - and I may even talk about the timeline at some upcoming events this year - including these in February:

One question we've been asked again and again since joining Red Hat is what's happening to the Fuse products - are they going to be consumed into JBoss existing product lines ? The answer is a categorical no - they aren't going anywhere - they are just going to get even more awesome! But more about that later ...