Saturday, March 21, 2015

How to run a WSO2 MB 3.0.0 Cluster Locally.

Introduction

WSO2 Message Broker 3.0.0 is a distributed message broker that supports AMQP and MQTT. This post explains the necessary steps to run several instances of WSO2 MB(Message Broker) on a single machine(locally) with the minimum number of configurations. WSO2 MB comes with an inbuilt H2 database which acts as the message store. When configuring for a cluster, the existing H2 database cannot be used. This is because the H2 database cannot work as a centralized database. We can use MSSQL, MySQL or Oracle as the message store. Since we only need to do a quick setup, we will use MySQL. Make sure you have already installed MySQL in your machine

Configuring MySQL for the Message broker

Creating the MySQL database

Create a database in MySQL with the name "wso2_mb". One way of creating the database is through the MySQL command line.

mysql -uroot -proot; #To login to MySQL console
CREATE DATABASE wso2_mb; #Creating the database

Configuring WSO2 Message Broker 3.0.0

Lets take a scenario where we need to setup a 3 node cluster. Following are the files that needs to be configured.

1. axis2.xml
2. broker.xml
3. carbon.xml
4. master-datasources.xml

axis2.xml

The axis2.xml file resides in "<MB_HOME>/repository/conf/axis2" folder. The axis2.xml file needs to be configured in all 3 MB instances.

1. First we have to enable clustering mode for the node.
<clustering class="org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent" enable="true">
2. Change the "membershipScheme" parameter to "wka".
<parameter name="membershipScheme">wka</parameter>
3. Add the IP address of the machine to "localMemberHost" parameter. All 3 nodes will have the same IP address as it is running locally.
<parameter name="localMemberHost">192.168.0.100</parameter>
4. Next step is assigning the value for "localMemberPort" parameter. As all the nodes are running locally, we need to provide them with different ports. Lets assume that following ports are assigned.
mb1 - 4000
mb2 - 4001
mb3 - 4002
So in mb1, set "localMemberPort" parameter value as 4000. This value is set by default. But for mb2 and mb3 the value should be 4001 and 4002 respectively. For example, mb2 will have the following value.
<parameter name="localMemberPort">4001</parameter>
5. The next step is to add the members. When adding the members, we have to add the IP addresses and assigned ports of the nodes except for the current node. For mb1, the following members should be added.
<members>
 <member>
  <hostName>192.168.0.100</hostName>
  <port>4001</port>
 </member>
 <member>
  <hostName>192.168.0.100</hostName>
  <port>4002</port>
 </member>
</members>
For mb2, it will be as follows.
<members>
 <member>
  <hostName>192.168.0.100</hostName>
  <port>4000</port>
 </member>
 <member>
  <hostName>192.168.0.100</hostName>
  <port>4002</port>
 </member>
</members>

broker.xml

The broker.xml file resides in "<MB_HOME>/repository/conf/" folder. The broker.xml file needs to configured in all 3 MB instances.

1. Modify "thriftServerHost" value to the IP address.
<thriftServerHost>192.168.0.100</thriftServerHost>
2. Change the "bindAddress" value to the IP address.
<bindAddress>192.168.0.100</bindAddress>

carbon.xml

The carbon.xml file resides in "<MB_HOME>/repository/conf/" folder. The carbon.xml file needs to be configured in all 3 MB instances. This is to avoid the nodes from using the same ports.

1. Set the "offset" value to a unique value in each node. Assume that following are the assigned offset values.
mb1 - 0
mb2 - 1
mb3 - 2
For example, mb2 will have the following value.
<Offset>1</Offset>

master-datasources.xml

The master-datasources.xml file resides in "<MB_HOME>/repository/conf/datasources" folder. The master-datasources.xml file needs to be configured in all 3 MB instances. In this file we will be configuring the WSO2 registry database configuration and the WSO2 MB message store database configuration.

1. Fix WSO2 MB registry's url path to H2 database. In the master-datasources.xml, we can see the following section.
<datasource>
    <name>WSO2_CARBON_DB</name>
    <description>The datasource used for registry and user manager</description>
    <jndiConfig>
        <name>jdbc/WSO2CarbonDB</name>
    </jndiConfig>
    <definition type="RDBMS">
        <configuration>
            <url>jdbc:h2:repository/database/WSO2CARBON_DB;DB_CLOSE_ON_EXIT=FALSE;LOCK_TIMEOUT=60000</url>
            <username>wso2carbon</username>
            <password>wso2carbon</password>
            <driverClassName>org.h2.Driver</driverClassName>
            <maxActive>50</maxActive>
            <maxWait>60000</maxWait>
            <testOnBorrow>true</testOnBorrow>
            <validationQuery>SELECT 1</validationQuery>
            <validationInterval>30000</validationInterval>
        </configuration>
    </definition>
</datasource>
For the "url" value we can see the default value as "repository/database/WSO2MB_DB". We have to give the absolute file path for the H2 database. Since each node has their own H2 database, this needs to be done is all MB nodes with their specific paths. Performing this change would imply that each node has their own registry and that it is not shared. Lets say that my MB nodes resides in the following paths.
mb1 - /home/hemikak/Documents/wso2/mb1/
mb2 - /home/hemikak/Documents/wso2/mb2/
mb3 - /home/hemikak/Documents/wso2/mb3/
So for mb1, it should be as follows...
<datasource>
    <name>WSO2_CARBON_DB</name>
    <description>The datasource used for registry and user manager</description>
    <jndiConfig>
        <name>jdbc/WSO2CarbonDB</name>
    </jndiConfig>
    <definition type="RDBMS">
        <configuration>
            <url>jdbc:h2:/home/hemikak/Documents/wso2/mb1/repository/database/WSO2CARBON_DB;DB_CLOSE_ON_EXIT=FALSE;LOCK_TIMEOUT=60000</url>
            <username>wso2carbon</username>
            <password>wso2carbon</password>
            <driverClassName>org.h2.Driver</driverClassName>
            <maxActive>50</maxActive>
            <maxWait>60000</maxWait>
            <testOnBorrow>true</testOnBorrow>
            <validationQuery>SELECT 1</validationQuery>
            <validationInterval>30000</validationInterval>
        </configuration>
    </definition>
</datasource>
2. Next step is to comment the current H2 database connection configuration for the messages store. Comment the "WSO2 MB embedded H2 Store" section.
<!-- WSO2 MB embedded H2 Store     -->
<datasource>
    <name>WSO2_MB_STORE_DB</name>
    <description>The datasource used for message broker database</description>
    <jndiConfig>
        <name>WSO2MBStoreDB</name>
    </jndiConfig>
    <definition type="RDBMS">
        <configuration>
            <url>jdbc:h2:repository/database/WSO2MB_DB;DB_CLOSE_ON_EXIT=FALSE;LOCK_TIMEOUT=60000</url>
            <driverClassName>org.h2.Driver</driverClassName>
            <maxActive>50</maxActive>
            <maxWait>60000</maxWait>
            <testOnBorrow>true</testOnBorrow>
            <validationQuery>SELECT 1</validationQuery>
            <validationInterval>30000</validationInterval>
        </configuration>
    </definition>
</datasource>
3. Next step is to point the WSO2 MB message store to the MySQL database we configured earlier. Uncomment the "MySQL data source" section.
<datasource>
     <name>WSO2_MB_STORE_DB</name>
     <jndiConfig>
         <name>WSO2MBStoreDB</name>
     </jndiConfig>
     <definition type="RDBMS">
         <configuration>
             <driverClassName>com.mysql.jdbc.Driver</driverClassName>
             <url>jdbc:mysql://localhost/wso2_mb</url>
             <username>root</username>
             <password>root</password>
             <maxActive>50</maxActive>
             <maxWait>60000</maxWait>
             <minIdle>5</minIdle>
             <testOnBorrow>true</testOnBorrow>
             <validationQuery>SELECT 1</validationQuery>
            <validationInterval>30000</validationInterval>
         </configuration>
     </definition>
 </datasource>
4. Now we have to copy the MySQL driver to "<MB_HOME>/repository/components/lib" folder. You can get the driver from here.

Starting up WSO2 Message Broker 3.0.0

1. Open up the terminal or command prompt in the "<MB_HOME>/bin" folder and run the following command based on the OS.
./wso2server.sh -Dsetup
or
./wso2server.bat -Dsetup

Troubleshooting

1. "Address is already in use". This is because there are nodes that use the same "Offset" value in carbon.xml.
Exception during startup: org.wso2.andes.transport.TransportException: Could not bind to /192.168.0.100:5672
org.wso2.andes.transport.TransportException: Could not bind to /192.168.0.100:5672
        at org.wso2.andes.transport.network.mina.MinaNetworkTransport.accept(MinaNetworkTransport.java:147)
        at org.wso2.andes.server.Broker.startupImpl(Broker.java:254)
        at org.wso2.andes.server.Broker.startup(Broker.java:108)
        ...
        ...
Caused by: java.net.BindException: Address already in use
        at sun.nio.ch.Net.bind0(Native Method)
 at sun.nio.ch.Net.bind(Net.java:444)
 at sun.nio.ch.Net.bind(Net.java:436)
 at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
2. "Cannot access the registry". This happens when the WSO2 MB registry database is not properly configured in master-datasources.xml file.
[2015-03-21 10:06:29,050] ERROR {org.wso2.carbon.event.core.internal.builder.EventBrokerHandler} -  Can not create the event broker
org.wso2.carbon.event.core.exception.EventBrokerConfigurationException: Cannot get the subscriptions 
        at org.wso2.carbon.event.core.internal.CarbonEventBroker.loadExistingSubscriptions(CarbonEventBroker.java:90)
        at org.wso2.carbon.event.core.internal.CarbonEventBroker.init(CarbonEventBroker.java:67)
        at org.wso2.carbon.event.core.internal.CarbonEventBrokerFactory.getEventBroker(CarbonEventBrokerFactory.java:90)
        ...
        ...
Caused by: org.wso2.carbon.event.core.exception.EventBrokerException: Cannot access the registry 
        at org.wso2.carbon.event.core.internal.subscription.registry.RegistrySubscriptionManager.getAllSubscriptions(RegistrySubscriptionManager.java:291)
        at org.wso2.carbon.event.core.internal.CarbonEventBroker.loadExistingSubscriptions(CarbonEventBroker.java:78)
        ... 103 more
Caused by: org.wso2.carbon.registry.core.exceptions.ResourceNotFoundException: Resource does not exist at path /_system/governance/event/topicIndex
        at org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.get(EmbeddedRegistry.java:532)
        at org.wso2.carbon.registry.core.caching.CacheBackedRegistry.get(CacheBackedRegistry.java:172)
        at org.wso2.carbon.registry.core.session.UserRegistry.getInternal(UserRegistry.java:613)
        at org.wso2.carbon.registry.core.session.UserRegistry.access$400(UserRegistry.java:60)
        at org.wso2.carbon.registry.core.session.UserRegistry$5.run(UserRegistry.java:596)
        at org.wso2.carbon.registry.core.session.UserRegistry$5.run(UserRegistry.java:593)
        at java.security.AccessController.doPrivileged(Native Method)
        at org.wso2.carbon.registry.core.session.UserRegistry.get(UserRegistry.java:593)
        at org.wso2.carbon.event.core.internal.subscription.registry.RegistrySubscriptionManager.getAllSubscriptions(RegistrySubscriptionManager.java:266)
        ... 104 more

Hope you got the cluster up and running.

No comments:

Post a Comment