Using Groovy and Camel for your scripting purposes

This topic has been displayed at other blogs, but it's such a great concept and served me very well quite recently so I think it warrants a revisit.

As a JVM developer (isn't that what we ought to call us nowaday when Scala/Groovy/Clojure is all the hype?) you're often sidelined by your scripting wiz fellows when it comes to writing quick "one-off" scripts for testing purposes and what-not.

I've personally gone as far as learning a bit of Python for this purpose, but I tend to miss my favorite java libraries and the ecosystem, although I must give the deepest respect to Python which is a really nice and powerful language. I must also confess that I've yet to find the time needed to become really proficient with it.

On our current project we had a requirement for a one-way http multicast solution in one of our test environments, as the backend system we were mediating requests to had more test environments than the rest of the chain. In EIP patterns you'd call this a fanout, multicast proxy or something similar.

Well... I thought this would be a really good fit for Camel, so I started setting up a standard Maven project. After a while it became quite obvious that while the Camel route itself was extremely small and simple, the project consisted of more Maven ceremony than actual code.

As this was an application for a test environment I was really not that interested in having a full build process for it. I'd rather just have something up and running in as few lines as possible.

Well, groovy scripts + grape to the rescue...

As any of you who've dealt with Groovy programming know, it is a great language for creating standalone scripts which are compiled and executed at runtime. Although Scala is my personal #1 choice when it comes to the JVM languages and offers scripting support as well, Groovy has one up in the Grape ecosystem which I've not yet seen in the Scala world.

So after moving the essential part of my application (the route) to it's own groovy script and adding a few Grape annotations to download the external Camel dependencies the solution was basically finished and executable by running "groovy Multicaster.groovy". Mind you, this is an extremely simple route and the dependencies will be downloaded at runtime which would be totally forbidden in a production environment. But for this purpose it fit the bill perfectly.

 @Grab('org.apache.camel:camel-core:2.10.0')  
 @Grab('org.slf4j:slf4j-api:1.6.6')  
 @Grab('org.slf4j:slf4j-log4j12:1.6.6')  
 @Grab('log4j:log4j:1.2.16')  
 @Grab('org.apache.camel:camel-jetty:2.10.0')  
 import org.apache.camel.*  
 import org.apache.camel.impl.*  
 import org.apache.camel.builder.*  
   
 def camelContext = new DefaultCamelContext()  
   
 camelContext.addRoutes(new RouteBuilder() {  
   def void configure() {  
     from('jetty:http://0.0.0.0:6080/?chunked=false')  
       .convertBodyTo(byte[].class) //To read multiple times, default is InputStream which is only readable once  
       .removeHeader(Exchange.HTTP_PATH) //So as not to forward URI to next call...  
       .wireTap("seda:multicaster")  
       .setBody().constant(STATIC_RESPONSE);  
       
     //Using SEDA here as it implies async handoff  
     from("seda:multicaster")  
          .multicast().parallelProcessing().to(  
   
         //bridgeEndpoint is used to indicate that http headers (SOAPAction etc) should be copied  
         'http://testserver1:6060/myService?bridgeEndpoint=true&throwExceptionOnFailure=false', //consumer1  
         'http://testserver2:6060/myService?bridgeEndpoint=true&throwExceptionOnFailure=false', //consumer2  
         'http://testserver3:6060/myService?bridgeEndpoint=true&throwExceptionOnFailure=false', //etc...  
         'http://testserver4:6060/myService?bridgeEndpoint=true&throwExceptionOnFailure=false',  
         'http://testserver5:6060/myService?bridgeEndpoint=true&throwExceptionOnFailure=false'  
       );  
   }  
   
   final String STATIC_RESPONSE = '''\  
             <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:ipl="http://www.acme.se/ipl">  
               <soapenv:Body>  
                <ipl:ipl>OK</ipl:ipl>  
               </soapenv:Body>  
             </soapenv:Envelope>'''.stripIndent()  
 })  
   
 camelContext.start()  
   
 addShutdownHook{ camelContext.stop() }  
 synchronized(this){ this.wait() }  

The combination of Camel in a scripting environment is extremely powerful. Just imagine the possibilities for statistic gathering utilities etc.. Now off to some more Groovying..

Free integration breakfast seminar in Stockholm, Oslo and Copenhagen

I'd just like to inform you that we at Redpill Linpro will be hosting a series of breakfast seminars about the FuseSource line of products, meaning Camel, CXF, ActiveMQ and ServiceMix.

The seminars will give you a technical quickstart to using Camel and ServiceMix in your projects and is intended for developers and architects. So there will be no fluffy slides or sales pitches but rather a more "down to the core" type of seminar with lots of coding and examples of why Camel is such an awesome integration framework to work with.

If you are interested and either in the Stockholm, Oslo or Copenhagen areas, just register at the specific event. The seminar is of course free. Hope to see you there!

Stockholm (12/6)
Oslo (13/6)
Copenhagen (14/6)

Until next time!


UPDATE

I'd like to update you with the links to the presentations as well as the source code. Many thanks to all of you who attended. Especially to Claus Ibsen (the Camel project lead) who took the time to show up in the audience at the Copenhagen session.

Slides at slideshare:
http://www.slideshare.net/RedpillLinpro

Code at github:
https://github.com/billybong/stockGwDemo

Bridging between JMS and RabbitMQ (AMQP) using Spring Integration

An old customer recently asked me if I had a solution for how to integrate between their existing JMS infrastructure on Websphere MQ with RabbitMQ.

Although I know that RabbitMQ has the shovel plugin which can bridge between Rabbit instances I've yet not found a good plugin for JMS <-> AMQP forwarding.
The first thing that came to my mind was to utilize a Spring Integration mediation as SI has excellent support for both JMS and Rabbit.

Curious as I am I started a PoC and this is the result. It takes messages of a JMS queue and forwards to an AMQP exchange that is bound to a queue the consumer application is supposed to listen to. I used an external HornetQ instance in JBoss 6.1 as the JMS Provider, but I am 100% secure that the same setup would work for Websphere MQ as they both implement JMS.

Be aware that I've done no performance tweaking or QoS setup yet as this is just a proof-of-concept. For a real setup you'd probably have to think about delivery guarantees versus performance and etc...

The code will be available at a GitHub repository near you soon..

SpringContext in XML:

<?xml version="1.0" encoding="UTF-8"?>
<beans:beans xmlns="http://www.springframework.org/schema/integration"
    xmlns:beans="http://www.springframework.org/schema/beans"
    xmlns:aop="http://www.springframework.org/schema/aop"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xmlns:int-amqp="http://www.springframework.org/schema/integration/amqp"
    xmlns:rabbit="http://www.springframework.org/schema/rabbit"
    xmlns:int-jms="http://www.springframework.org/schema/integration/jms"
    xmlns:int="http://www.springframework.org/schema/integration"
    xsi:schemaLocation="http://www.springframework.org/schema/integration/amqp http://www.springframework.org/schema/integration/amqp/spring-integration-amqp-2.1.xsd
        http://www.springframework.org/schema/aop http://www.springframework.org/schema/aop/spring-aop-3.0.xsd
        http://www.springframework.org/schema/integration http://www.springframework.org/schema/integration/spring-integration-2.0.xsd
        http://www.springframework.org/schema/integration/jms http://www.springframework.org/schema/integration/jms/spring-integration-jms-2.1.xsd
        http://www.springframework.org/schema/rabbit http://www.springframework.org/schema/rabbit/spring-rabbit-1.0.xsd
        http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.0.xsd">
        
    <beans:bean id="jndiTemplate" class="org.springframework.jndi.JndiTemplate">
        <beans:property name="environment">
            <beans:props>
                <beans:prop key="java.naming.factory.initial">org.jnp.interfaces.NamingContextFactory</beans:prop>
                <beans:prop key="java.naming.provider.url">jnp://localhost:1099</beans:prop>
                <beans:prop key="java.naming.factory.url.pkgs">org.jnp.interfaces:org.jboss.naming</beans:prop>
            </beans:props>
        </beans:property>
    </beans:bean>

    <beans:bean id="jmsQueueConnectionFactory"
      class="org.springframework.jndi.JndiObjectFactoryBean">
        <beans:property name="jndiTemplate">
            <beans:ref bean="jndiTemplate"/>
        </beans:property>
        <beans:property name="jndiName">
            <beans:value>ConnectionFactory</beans:value>
        </beans:property>
    </beans:bean>

    <!-- Channels and adapters for SI -->
    <int-jms:message-driven-channel-adapter connection-factory="jmsQueueConnectionFactory" destination-name="myJmsQueue" channel="rabbitChannel"/>
    <channel id="rabbitChannel"/>
    <int-amqp:outbound-channel-adapter channel="rabbitChannel" exchange-name="fromJmsExchange" amqp-template="rabbitTemplate"/>
    
    <!-- Connectivity to Rabbit -->
    <rabbit:template id="rabbitTemplate" connection-factory="cf"/>
    <rabbit:connection-factory id="cf" host="localhost"/>
    
    <!-- Rabbit entities, to be created at context startup -->
    <rabbit:admin connection-factory="cf"/>
    <rabbit:queue name="fromJMS"/>
    <rabbit:direct-exchange name="fromJmsExchange">
        <rabbit:bindings>
            <rabbit:binding queue="fromJMS"/>
        </rabbit:bindings>
    </rabbit:direct-exchange>
</beans:beans>

Maven POM:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>org.rl</groupId>
    <artifactId>si.jmstorabbit</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <packaging>jar</packaging>

    <name>si.jmstorabbit</name>
    <url>http://maven.apache.org</url>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <hornet.version>2.2.5.Final</hornet.version>
        <spring.integration.version>2.1.0.RELEASE</spring.integration.version>
    </properties>

    <repositories>
        <repository>
            <id>springsource-release</id>
            <url>http://repository.springsource.com/maven/bundles/release</url>
            <snapshots>
                <enabled>false</enabled>
            </snapshots>
        </repository>
        <repository>
            <id>springsource-external</id>
            <url>http://repository.springsource.com/maven/bundles/external</url>
            <snapshots>
                <enabled>false</enabled>
            </snapshots>
        </repository>
    </repositories>

    <dependencies>
    <dependency>
       <groupId>org.springframework.integration</groupId>
       <artifactId>spring-integration-core</artifactId>
       <version>${spring.integration.version}</version>
    </dependency>
    <dependency>
       <groupId>org.springframework.integration</groupId>
       <artifactId>spring-integration-file</artifactId>
       <version>${spring.integration.version}</version>
    </dependency>
    <dependency>
       <groupId>org.springframework.integration</groupId>
       <artifactId>spring-integration-amqp</artifactId>
       <version>${spring.integration.version}</version>
    </dependency>
    <dependency>
       <groupId>org.springframework.integration</groupId>
       <artifactId>spring-integration-jms</artifactId>
       <version>${spring.integration.version}</version>
    </dependency>
    <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <version>3.8.1</version>
      <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>org.springframework</groupId>
        <artifactId>spring-context</artifactId>
        <version>3.0.7.RELEASE</version>
    </dependency>
    
    <dependency>
        <groupId>jboss</groupId>
        <artifactId>jnp-client</artifactId>
        <version>4.2.2.GA</version>
    </dependency>
            
    
    <dependency>
            <groupId>org.hornetq</groupId>
            <artifactId>hornetq-core-client</artifactId>
            <version>${hornet.version}</version>
        </dependency>
        <dependency>
            <groupId>org.hornetq</groupId>
            <artifactId>hornetq-jms-client</artifactId>
            <version>${hornet.version}</version>
        </dependency>
        <dependency>
            <groupId>org.hornetq</groupId>
            <artifactId>hornetq-jms</artifactId>
            <version>${hornet.version}</version>
        </dependency>
           <dependency>
            <groupId>jboss</groupId>
            <artifactId>jboss-common-client</artifactId>
            <version>3.2.3</version>
        </dependency>
        <dependency>
            <groupId>org.jboss.netty</groupId>
            <artifactId>netty</artifactId>
            <version>3.2.7.Final</version>
        </dependency>
        <dependency>
            <groupId>javax.jms</groupId>
            <artifactId>jms</artifactId>
            <version>1.1</version>
        </dependency>
  </dependencies>
  
</project>

Gotcha when using camel-servlet

In my current project I'm using Apache Camel for the integration, and I bumped into a significant "oups"-moment while using the camel-servlet component.

I found that while I had multiple applications (WAR's) deployed I often started the wrong routes, even though I was hitting the correct endpoint url.

Say i.e. that you have two application, A and B. Both of these has the same endpoint addresses, i.e. servlet:/myservice

The logical thing would be that you could reach application A through:
http://localhost/A/myservice

and B through:
http://localhost/B/myservice

Now imagine my surprise when requests for the A application would up in the route for B. It took me a while to figure this out, but here's the reason for this...

When you expose your routes through servlet's you are using the CamelHttpTransportServlet class. An example web.xml could be something like this:

<!-- Camel servlet -->
<servlet>
<servlet-name>CamelServlet</servlet-name>
<servlet-class>org.apache.camel.component.servlet.CamelHttpTransportServlet</servlet-class>
<load-on-startup>1</load-on-startup>
</servlet>

<!-- Camel servlet mapping -->
<servlet-mapping>
<servlet-name>CamelServlet</servlet-name>
<url-pattern>/*</url-pattern>
</servlet-mapping>


Now, if you name your servlet the same for both applications you're actually only publishing one common servlet class. The camel servlet component does not take your context-root into account, which means that you hit the same endpoint irregardless whether you go to /A/myservice or /B/myservice. The one that gets hit is the one that was last deployed (or started).

Ok, easily fixed I thought... Just rename the servlet-name to something more unique for each application...
After that my routes where no longer reachable and I was just left with 404's each time I tried to hit the endpoints.
After some trial and error as well as googling I found out that if you rename your servlet from the default "CamelServlet" you *have* to specify the servlet name in your endpoint uri as well.
That is, the endpoint in application A would be i.e.:

servlet:/myService?servletName=AServlet

and B:

servlet:/myservice?servletName=BServlet

So, the lesson is:

- Always rename the camel servlet name, you never know which other applications will be deployed together on the same server. Your endpoints might clash with these and you'll have a mess figuring out why.

- Always include the servletName in your endpoint uri.

These tips should definitely be added to the documentation page to help others avoid making the same mistake.

'til next time!

Implementing your own "SOA" registry, the maven way

Saw a very interesting article on InfoQ by Ben Wilcock where he described a dead simple SOA R&R using a very pragmatic approach with Maven combined with a simple HTML app.
http://www.infoq.com/articles/SimpleServiceRepository

As an integration consultant I've worked with registries ranging from overly complex offerings (IBM Websphere Registry and Repository) to time-consuming simple wikis, but I have to say that his solution is arguably the most elegant yet.

The best part is that you both get the visibility as your schemas and contracts are available to all your stakeholders, while keeping your co-developers happy by just pointing them to a Maven project to include in their dependencies.

I'm definately hijacking this concept to my current project, and I find it very interesting that this solution has been under my nose all this time without reflecting on it.

Camel and HornetQ as JMS provider

In my current project we're utilizing JBoss HornetQ as the messaging bus, and I wanted to try out how Camel could connect to it. My intention was to have the HornetQ instance running externally from the Camel application and connect to it. There are a bunch of examples on how to embedd HornetQ 2.2.5 (the release I'm dealing with) inside Spring, but I could'nt find a single one for how to connect to an external instance.

For my own reference I post the working solution here which uses HornetQ's JNDI to lookup the ConnectionFactory as well as a Spring JNDI template.

<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:camel="http://camel.apache.org/schema/spring"
       xsi:schemaLocation="
          http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
          http://camel.apache.org/schema/spring http://camel.apache.org/schema/spring/camel-spring.xsd">

    <bean id="jndiTemplate" class="org.springframework.jndi.JndiTemplate">
        <property name="environment">
            <props>
                <prop key="java.naming.factory.initial">org.jnp.interfaces.NamingContextFactory</prop>
                <prop key="java.naming.provider.url">jnp://localhost:1099</prop>
                <prop key="java.naming.factory.url.pkgs">org.jnp.interfaces:org.jboss.naming</prop>
            </props>
        </property>
    </bean>

    <bean id="jmsQueueConnectionFactory"
      class="org.springframework.jndi.JndiObjectFactoryBean">
        <property name="jndiTemplate">
            <ref bean="jndiTemplate"/>
        </property>
        <property name="jndiName">
            <value>ConnectionFactory</value>
        </property>
    </bean>
    
    <bean name="jms" class="org.apache.camel.component.jms.JmsComponent">
        <property name="connectionFactory" ref="jmsQueueConnectionFactory"/>
    </bean>

    <camel:camelContext id="context1">
        <camel:route id="FirstRoute">
            <camel:from uri="jms:queue:inputqueue"/>
            <camel:log logName="jmsLog" message="Got message from JMS queue:"/>
            <camel:to uri="jms:topic:helloworld"/>
        </camel:route>
    </camel:camelContext>
</beans>

And the Maven POM:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>camelHornet</groupId>
    <artifactId>camelHornet</artifactId>
    <version>1.0-SNAPSHOT</version>

    <properties>
        <camel.version>2.8.0</camel.version>
        <hornet.version>2.2.5.Final</hornet.version>
    </properties>
    <dependencies>
        <dependency>
            <groupId>org.apache.camel</groupId>
            <artifactId>camel-core</artifactId>
            <version>${camel.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.camel</groupId>
            <artifactId>camel-spring</artifactId>
            <version>${camel.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.camel</groupId>
            <artifactId>camel-jms</artifactId>
            <version>${camel.version}</version>
        </dependency>
          <dependency>
            <groupId>org.hornetq</groupId>
            <artifactId>hornetq-core-client</artifactId>
            <version>${hornet.version}</version>
        </dependency>
        <dependency>
            <groupId>org.hornetq</groupId>
            <artifactId>hornetq-jms-client</artifactId>
            <version>${hornet.version}</version>
        </dependency>
        <dependency>
            <groupId>org.hornetq</groupId>
            <artifactId>hornetq-jms</artifactId>
            <version>${hornet.version}</version>
        </dependency>

        <dependency>
                <groupId>javax.jms</groupId>
                <artifactId>jms</artifactId>
                <version>1.1</version>
                <type>jar</type>
                <scope>runtime</scope>
        </dependency>
        <dependency>
            <groupId>org.hornetq</groupId>
            <artifactId>hornetq-logging</artifactId>
            <version>${hornet.version}</version>
        </dependency>
        <dependency>
            <groupId>org.jboss.netty</groupId>
            <artifactId>netty</artifactId>
            <version>3.2.3.Final</version>
        </dependency>
        <dependency>
            <groupId>jboss</groupId>
            <artifactId>jnp-client</artifactId>
            <version>4.0.2</version>
        </dependency>
        <dependency>
            <groupId>org.jboss.logging</groupId>
            <artifactId>jboss-logging</artifactId>
            <version>3.0.0.CR1</version>
        </dependency>
    </dependencies>
    <build>
        <plugins>
            <plugin>
              <groupId>org.apache.camel</groupId>
              <artifactId>camel-maven-plugin</artifactId>
              <version>${camel.version}</version>
              <configuration>
                <applicationContextUri>META-INF/context.xml</applicationContextUri>
              </configuration>
            </plugin>
        </plugins>
    </build>
</project>

Three mindblowing days of filling my brain with new stuff

This week we at Redpill-Linpro had the great fortune to be instructed by Charles Moulliard (FuseSource) on advanced topics covering ServiceMix and Camel.

Talking from my own experience I already had quite a good grasp of Camel, but ServiceMix was probably the eye-opener for me. Until now I've previously rolled my eyes each time SM was mentioned, thinking "why do we need yet another application platform when we have web servers, Spring, JEE servers etc already?".
What I did not understand until now is the flexibility you gain when deploying your integrations to a pure OSGI stack such as Karaf (which is the layer on top of your OSGI runtime) instead of using classic hierarchical class-loading servers.

One thing that also was clear is that ServiceMix, although labeled as an ESB, is actually sort of an application server in the sense that it is not only used for integration applications. Sure, you have a lot of Camel/CXF/ActiveMQ bundles already prebuilt and ready to be utilized, but looking at the architecture it is plain to see that any type of OSGI applications could benefit by being deployed into Karaf.

I can safely say that I've bought the hype of OSGI now after finally understanding what it actually gives you. I just hope for some better tooling when it comes to declaring your package exports and imports as that seems to be an area where you'll need to perform a lot of trial and error to get correct.

We also had some good discussions concerning the fact that a lot of supplementary (Activiti, JAMES etc..) and even competing frameworks already bundle Camel to expose transport options that are not already implemented by the framework itself. JBoss ESB i.e. comes with a Camel Gateway which could be used to handle messages originating from all types of transports available in Camel that the native ESB does not have adapters for.
It will be very interesting to see how long it takes for the big players (IBM and Oracle, I'm looking at you) to bundle Camel as part of their connector strategy. I can easily fantasize about i.e. a future Camel node in the Websphere Message Broker product as a way to extend the possible transport that it could provide.

'til next time!