Tuesday, September 21, 2010

Why ESB server hang after runing for a while?

A frequently asked question on FuseSource[1] forum or Apache Servicemix[2] mailling list is they encounter an issue that ESB server just hang after running a while, can't handle incoming request anymore. This issue could be caused for two major reason

1. customer working flow isn't correct, not all MEPs are handled correctly, especially when use sendSync, but some MessageExchange never end up with DONE/ERROR, this will cause thread lock, and threads would be used up in this case so that can't hanle incoming request anymore.
2. Too many cocurrent client request message cause threadpool used up, customer usually face this problem when they do performance/load test.The solution is quite straightforward for different cases,
for case1. correct your working flow, ensure that all MEPs are handled correctly, use send but not sendSync as much as possible.
for case2. configure threadpool[3] for the performance test component, ensure the threadpool big enough for your performance/load test.

I saw several cases that customer use cxfbc:consumer endpoint as facade to handle performance/load test request, the typical test flow could be
cxfbc:consumer==>servicemix-camel
cxfbc:consumer==>cxfse
cxfbc:consumer==>servicemix-bean
cxfbc:consumer==>cxfbc:provider
 Previously By default cxfbc:consumer endpoint use sendSync(this is a block mehtod,waiting for response message) to send message to NMR, this can cause deadlock for cocurrent request with default threadpool configuration, so if you wanna support heavy cocurrent request better,  you need add synchronous="false" attribute to cxfbc:consumer endpoint, this will leverage CXF continuation API(I will explain how in another blog) and use send(this is a non-block method) method to send message to NMR. Since servicemix-cxf-bc-2010.02, synchronous="false" is the default value, which means the default behavior of cxfbc is asynchronously.

Btw, to configure threadpool for NMR and each component in SMX4 is different with it in SMX3[3]. In SMX4, we leverage OSGi Configuration admin service to configure properties, so customer need edit a file named org.apache.servicemix.nmr.cfg in $SMX_HOME/etc folder to configure threadpool for nmr, and edit a file named org.apache.servicemix.components.component-name(like cxfbc).cfg in $SMX_HOME/etc folder for each component.
[1]http://fusesource.com/
[2]http://servicemix.apache.org/home.html
[3]http://servicemix.apache.org/thread-pools.html

Monday, September 20, 2010

Wired Exception caused by Proxy.equals

Weeks ago I encountered a wired problem when I shutdown FuseSource[1] ESB4/Apache Servicemix[2]4, I get exception like

org.osgi.service.blueprint.container.ServiceUnavailableException: Service is unavailable
at org.apache.aries.blueprint.container.ReferenceListRecipe$ServiceDispatcher.call(ReferenceListRecipe.java:201)
at org.apache.aries.blueprint.container.AbstractServiceReferenceRecipe$JdkProxyFactory$1.invoke(AbstractServiceReferenceRecipe.java:632)
at $Proxy12.repositoryEvent(Unknown Source)
at org.apache.karaf.features.internal.FeaturesServiceImpl.callListeners(FeaturesServiceImpl.java:939)
at org.apache.karaf.features.internal.FeaturesServiceImpl.internalRemoveRepository(FeaturesServiceImpl.java:186)
at org.apache.karaf.features.internal.FeaturesServiceImpl.stop(FeaturesServiceImpl.java:777)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)[:1.6.0_13]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)[:1.6.0_13]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)[:1.6.0_13]
at java.lang.reflect.Method.invoke(Method.java:597)[:1.6.0_13]
at org.apache.aries.blueprint.utils.ReflectionUtils.invoke(ReflectionUtils.java:230)[7:org.apache.aries.blueprint:0.1.0.r964701]
at org.apache.aries.blueprint.container.BeanRecipe.invoke(BeanRecipe.java:854)[7:org.apache.aries.blueprint:0.1.0.r964701]
at org.apache.aries.blueprint.container.BeanRecipe.destroy(BeanRecipe.java:761)[7:org.apache.aries.blueprint:0.1.0.r964701]
at org.apache.aries.blueprint.container.BlueprintRepository.destroy(BlueprintRepository.java:295)[7:org.apache.aries.blueprint:0.1.0.r964701]

I realize this exception is from ESB kernel, karaf FeaturesServiceImpl class. After digging into the FeaturesServiceImpl code, I found that if the FeaturesListener gone, it should invoke unregisterListener() mothed, so we shouldn't have get this exception at all.
the unregisterListener() mothed is
public void unregisterListener(FeaturesListener listener) {
        listeners.remove(listener);
    }

It's quite simple, and listeners is just a List instance. To figure out what's going on underlying, I launched remote debug for ESB process and set breakpoint in unregisterListener method, but what I saw then really confused me. I do see listeners List has a FeaturesListener instance, let's say with ObjectId 155, but when unregisterListener() get invoked with argument FeaturesListener instance exactly the same object with the one listeners List containing, listeners.remove(listener) doesn't work, listeners can't remove listener it actually contain.

I know List use Object.equals() method to determine if two object really equal, I guess there must be something wrong in equal method, so I then step into the List.remove method and then Object.equals method, aha, listener just not eqaul itself. Well, finally figured out that this is caused by we use aries blueprint to inject FeatureListener as OSGi servie using dynamic proxy, it's not the real FeatureListener, it's just the proxy of FeatureListenr, this can explain the wired exception, equals() method from a proxy actually will compare a proxy and the real object, which obviously will return false, so we should override equals method to handle the proxy comparasion case.

I googled a little bit, there's another good article to describe this Proxy.equals()[3] issue.And the issue from Apache Karaf to track is KARAF-136[4]
[1]http://fusesource.com/
[2]http://servicemix.apache.org/home.html
[3]http://www.javaspecialists.eu/archive/Issue126.html
[4]https://issues.apache.org/jira/browse/KARAF-136

jpa-osgi example in FUSE ESB 4

Since FuseSource[1] ESB 4.2.0-01-00, we introduce a new example jpa-osgi to demostrate how jpa works in OSGi based ESB container. We choose to use hibernate as JPA implementation in this demo, as hibernate is popularly used in existing system. Package hibernate jars as bundle isn't a easy job, but thanks for Spring Source's great work, we can simply reuse those Spring Source hibernate bundles in our demo.

We create a jpa-hibernate feature which put all necessary bundles together,
    <feature name="jpa-hibernate" version="4.3.0-fuse-01-00">
<bundle>mvn:org.apache.servicemix.specs/org.apache.servicemix.specs.java-persistence-api-1.1.1/1.5.0</bundle>
<bundle>mvn:org.apache.servicemix.bundles/org.apache.servicemix.bundles.commons-dbcp/1.2.2_5</bundle>
<bundle>mvn:org.springframework/spring-jdbc/3.0.3.RELEASE</bundle>
<bundle>mvn:org.apache.servicemix.bundles/org.apache.servicemix.bundles.dom4j/1.6.1_2</bundle>
<bundle>mvn:org.antlr/com.springsource.antlr/2.7.7</bundle>
<bundle>mvn:org.objectweb.asm/com.springsource.org.objectweb.asm/1.5.3</bundle>
<bundle>mvn:net.sourceforge.cglib/com.springsource.net.sf.cglib/2.2.0</bundle>
<bundle>mvn:org.jboss.javassist/com.springsource.javassist/3.9.0.GA</bundle>
<bundle>mvn:org.hibernate/com.springsource.org.hibernate.annotations.common/3.3.0.ga</bundle>
<bundle>mvn:org.hibernate/com.springsource.org.hibernate.annotations/3.4.0.GA</bundle>
<bundle>mvn:org.hibernate/com.springsource.org.hibernate.ejb/3.4.0.GA</bundle>
<bundle>mvn:org.hibernate/com.springsource.org.hibernate/3.3.2.GA</bundle>
<bundle>mvn:org.springframework/spring-orm/3.0.3.RELEASE</bundle>
</feature>

This demo is based on the cxf-wsdl-first scenario, which you can find it from both Fuse ESB 3.x and 4.x. The workflow is
cxfbc consumer====>cxfse endpoint.
cxfbc consumer handle external request, and cxfse endpoint retrieve PersonEntity according to ID from underlying hsqldb. We can put all complicated configuration in xbean used for cxf se endpoint.

<cxfse:endpoint>
<cxfse:pojo>
<bean class="org.apache.servicemix.samples.wsdl_first.PersonImpl">
<constructor-arg>
<ref bean="entityManagerFactory"/>
</constructor-arg>
</bean>
</cxfse:pojo>
</cxfse:endpoint>
<bean class="org.apache.servicemix.common.osgi.EndpointExporter" />
also configure JPA entityManagerFacotry in this file
<bean id="transactionTemplate" class="org.springframework.transaction.support.TransactionTemplate">
<property name="transactionManager">
<bean class="org.springframework.orm.jpa.JpaTransactionManager">
<property name="entityManagerFactory" ref="entityManagerFactory"/>
</bean>
</property>
</bean>
<bean id="jpaTemplate" class="org.springframework.orm.jpa.JpaTemplate">
<property name="entityManagerFactory" ref="entityManagerFactory"/>
</bean>
<bean id="entityManagerFactory" class="org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean">
<property name="persistenceUnitName" value="smx4"/>
<property name="jpaVendorAdapter" ref="jpaAdapter"/>
<property name="dataSource" ref="dataSource" />
</bean>
<bean id="jpaAdapter"
class="org.springframework.orm.jpa.vendor.HibernateJpaVendorAdapter">
<property name="databasePlatform" value="org.hibernate.dialect.HSQLDialect" />
</bean>
<!-- DataSource Definition -->
<bean id="dataSource" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close">
<property name="driverClassName" value="org.hsqldb.jdbcDriver" />
<property name="url" value="jdbc:hsqldb:mem:smx4_jpa" />
<property name="username" value="sa" />
<property name="password" value="" />
</bean>

And there's a JPA required persistence.xml in src/main/resources/META-INF/ folder
  <persistence xmlns="http://java.sun.com/xml/ns/persistence"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" version="1.0">
<persistence-unit name="smx4" transaction-type="RESOURCE_LOCAL">
<class>org.apache.servicemix.samples.wsdl_first.PersonEntity</class>
<!-- Hibernate -->
<properties>
<property name="hibernate.dialect" value="org.hibernate.dialect.HSQLDialect"/>
<property name="hibernate.hbm2ddl.auto" value="create"/>
</properties>
</persistence-unit>
</persistence>

After installing jpa-hibernate feature, we can see related bundles set
[ 179] [Active     ] [            ] [       ] [   60] Apache ServiceMix Specs :: JAVA PERSISTENCE API 1.4 (1.5.0)
[ 180] [Active     ] [            ] [       ] [   60] Apache ServiceMix Bundles: commons-dbcp-1.2.2 (1.2.2.5)
[ 181] [Active     ] [            ] [       ] [   60] Spring JDBC (3.0.3.RELEASE)
[ 182] [Active     ] [            ] [       ] [   60] Apache ServiceMix Bundles: dom4j-1.6.1 (1.6.1.2)
[ 183] [Active     ] [            ] [       ] [   60] ANTLR (2.7.7)
[ 184] [Active     ] [            ] [       ] [   60] ObjectWeb ASM (1.5.3)
[ 185] [Active     ] [            ] [       ] [   60] CGLIB Code Generation Library (2.2.0)
[ 186] [Active     ] [            ] [       ] [   60] Javassist Java Programming Assistant (3.9.0.GA)
[ 187] [Active     ] [            ] [       ] [   60] JBoss Hibernate Common Annotations (3.3.0.ga)
[ 188] [Resolved   ] [            ] [       ] [   60] JBoss Hibernate Annotations (3.4.0.GA)
                                       Hosts: 190
[ 189] [Resolved   ] [            ] [       ] [   60] JBoss Hibernate Entity Manager (3.4.0.GA)
                                       Hosts: 190
[ 190] [Active     ] [            ] [       ] [   60] JBoss Hibernate Object-Relational Mapper (3.3.2.GA)
                                       Fragments: 188,189
[ 191] [Active     ] [            ] [       ] [   60] Spring ORM (3.0.3.RELEASE)
We can see bundle 188,189 are fragment bundle, at the moment we add this demo, Apache Felix not fully support fragment bundle yet, one outstanding issue is FELIX-1919[2], so this demo only works with equinox at that time. But now it works with both felix and equinox.

As hibernate is LGPL license, we can't add this demo into Apache Servicemix[3], but we do have an issue[4] on Apache Servicemix to track it, we can use OpenJPA to replace hibernate for this demo an Apache Side later on.

[1]http://fusesource.com
[2]https://issues.apache.org/jira/browse/FELIX-1919
[3]http://servicemix.apache.org/home.html
[4]https://issues.apache.org/activemq/browse/SMX4-472

Sunday, September 19, 2010

How to use HTTP Basic Auth with servicemix-cxf-bc

For servicemix-cxf-bc[1] component in Apache Servicemix[2] or FuseSource[3] ESB, there are several ways to use security feature for both consumer and provider endpoint. For message level security, we can secured SOAP message itself with WS-Security, or for transport protocol level security, we can use HTTPs which provide a secured HTTP connection.
Actually we have testcases domenstrating how each scenario works
cxf bc consumer with ws-security testcase[4]
cxf bc provider with ws-security testcase[5]
cxf bc consumer with https testcase[6]
cxf bc provider with https testcase[7]

However, instead of using WS-Security or HTTPS, one straightforward way is use HTTP Basic Authentication. For cxfbc provider endpoint which play a role as a client, to enaable HTTP Basic Auth, customer need specify a busCfg which configure http:conduit, something like

<http:conduit name="{your_name_space}your_endpoint_name.http-conduit">
<http:authorization>
<sec:username>Betty</sec:username>
<sec:password>password</sec:password>
</http:authorization>
 </http:conduit>


For cxfbc consumer which play a role as a server, to enable server side HTTP Basic Auth, we need an interceptor to do it,  the basic idea is extract AuthorizationPolicy from the incoming message and compare the username/password, but it's not so complicated, here is a good article[8] to show how to do it.

Another way(as Dan Kulp  pointed out) to configure server side basic auth is configure the jetty instance to handle the authentication,  as Jetty has a "SecurityHandler" that can be configured into the handlers via CXF config. The SecurityHandler takes an Authenticator(they have a BasicAuthenticator) and a UserRealm Object (they have one for basic HashMap lookup things as well as a JDBC version)


[1]http://servicemix.apache.org/servicemix-cxf-bc.html
[2]http://servicemix.apache.org/home.html
[3]http://fusesource.com/
[4]https://svn.apache.org/repos/asf/servicemix/components/bindings/servicemix-cxf-bc/trunk/src/test/java/org/apache/servicemix/cxfbc/ws/security/CxfBCSecurityTest.java
[5]https://svn.apache.org/repos/asf/servicemix/components/bindings/servicemix-cxf-bc/trunk/src/test/java/org/apache/servicemix/cxfbc/ws/security/CxfBcProviderSecurityTest.java
[6]https://svn.apache.org/repos/asf/servicemix/components/bindings/servicemix-cxf-bc/trunk/src/test/java/org/apache/servicemix/cxfbc/ws/security/CxfBcHttpsConsumerTest.java
[7]https://svn.apache.org/repos/asf/servicemix/components/bindings/servicemix-cxf-bc/trunk/src/test/java/org/apache/servicemix/cxfbc/ws/security/CxfBcProviderHttpsTest.java
[8]http://chrisdail.com/2008/03/31/apache-cxf-with-http-basic-authentication/

How to get/set properties from/to JBI NMR within cxf se endpoint

As the typical way to use cxf se component in Apache Servicemix[1] or FuseSource[2] ESB is like
JBI consumer endpoint<=====> cxf se endpoint.Sometimes customer want to put extra properties during runtime in JBI NormalizedMessage with JBI consumer endpoint and retrieve it from cxf se endpoint later on.
With SMXCOMP-593[3], we provide a webservice friendly and CXF-ish way to retrieve and set properties in cxf se endpoint.
Customer just need add code like

    @Resource
    private WebServiceContext wsContext;
to inject WebServiceContext to this CxfSeEndpoint

and use
    javax.xml.ws.handler.MessageContext ctx = wsContext.getMessageContext();
    org.apache.cxf.message.Message message = ((org.apache.cxf.jaxws.context.WrappedMessageContext) ctx).getWrappedMessage();
    String propertyValue = (String) message.get("PROPERTY_NAME");//retrieve property
    message.put("PROPERTY_NAME", "newPropertyValue");//set property

to get/set properties within cxf se endpoint.

[1]http://servicemix.apache.org/home.html
[2]http://fusesource.com/
[3]https://issues.apache.org/activemq/browse/SMXCOMP-593

How to configure cxf bus for smx-cxf-bc in SMX4?

In Apache Servicemix[1] 3.x or FuseSource[2] ESB 3.x, we use busCfg to specify cxf bus configuration file for servicemix-cxf-bc[3] component, and busCfg is popularly used in serivicemix-cxf-bc for advanced features, such as ws-*, https, jms transport, etc.

How ever, there's an issue for using busCfg in Apache Servicemix[1]4.x or FuseSource[2] ESB 4.x which is based on an OSGi core container. In OSGi world, it use a very different classloader machenism with the old hierarchical one, so that the the busCfg file in customer cxfbc endpoint bundle isn't available for pre-installed cxf bundle. The way to use busCfg in SMX4 is that create a fragment bundle including the busCfg file, and attach it to cxf bundle, here cxf bundle play the role as host bundle and all resource in fragment bundle is available for host bundle. Actually we can encounter similar issue often in OSGi world, if the api is passing a configuration file, we can see the same problem, the bundle host the api code can't load the resource from the customer bundle which use the api. Here is a similar discussion from FUSE side jira(ESB-1245)[4], a problem caused by need pass WSS4J signature properties file to wss4j bundle.

Besides using fragment bundle to hold configuration files, we introduce providedBus(SMXCOMP-688)[5] in smx4 which init cxf bus bean in same context with cxf bc endpoint configuration, this way avoid passing configuration file between bundles.

[1]http://servicemix.apache.org/home.html
[2]http://fusesource.com/
[3]http://servicemix.apache.org/servicemix-cxf-bc.html
[4]http://fusesource.com/issues/browse/ESB-1245
[5]https://issues.apache.org/activemq/browse/SMXCOMP-688