Introduction
WSO2 Message Broker 3.0.0 is a distributed message broker that supports AMQP and MQTT. This post explains the necessary steps to run several instances of WSO2 MB(Message Broker) on a single machine(locally) with the minimum number of configurations. WSO2 MB comes with an inbuilt H2 database which acts as the message store. When configuring for a cluster, the existing H2 database cannot be used. This is because the H2 database cannot work as a centralized database. We can use MSSQL, MySQL or Oracle as the message store. Since we only need to do a quick setup, we will use MySQL. Make sure you have already installed MySQL in your machine
Configuring MySQL for the Message broker
Creating the MySQL database
Create a database in MySQL with the name "wso2_mb". One way of creating the database is through the MySQL command line.
mysql -uroot -proot; #To login to MySQL console CREATE DATABASE wso2_mb; #Creating the database
Configuring WSO2 Message Broker 3.0.0
Lets take a scenario where we need to setup a 3 node cluster. Following are the files that needs to be configured.
1. axis2.xml 2. broker.xml 3. carbon.xml 4. master-datasources.xmlaxis2.xml
The axis2.xml file resides in "<MB_HOME>/repository/conf/axis2" folder. The axis2.xml file needs to be configured in all 3 MB instances.
1. First we have to enable clustering mode for the node.<clustering class="org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent" enable="true">2. Change the "membershipScheme" parameter to "wka".
<parameter name="membershipScheme">wka</parameter>3. Add the IP address of the machine to "localMemberHost" parameter. All 3 nodes will have the same IP address as it is running locally.
<parameter name="localMemberHost">192.168.0.100</parameter>4. Next step is assigning the value for "localMemberPort" parameter. As all the nodes are running locally, we need to provide them with different ports. Lets assume that following ports are assigned.
mb1 - 4000 mb2 - 4001 mb3 - 4002So in mb1, set "localMemberPort" parameter value as 4000. This value is set by default. But for mb2 and mb3 the value should be 4001 and 4002 respectively. For example, mb2 will have the following value.
<parameter name="localMemberPort">4001</parameter>5. The next step is to add the members. When adding the members, we have to add the IP addresses and assigned ports of the nodes except for the current node. For mb1, the following members should be added.
<members> <member> <hostName>192.168.0.100</hostName> <port>4001</port> </member> <member> <hostName>192.168.0.100</hostName> <port>4002</port> </member> </members>For mb2, it will be as follows.
<members> <member> <hostName>192.168.0.100</hostName> <port>4000</port> </member> <member> <hostName>192.168.0.100</hostName> <port>4002</port> </member> </members>
broker.xml
The broker.xml file resides in "<MB_HOME>/repository/conf/" folder. The broker.xml file needs to configured in all 3 MB instances.
1. Modify "thriftServerHost" value to the IP address.<thriftServerHost>192.168.0.100</thriftServerHost>2. Change the "bindAddress" value to the IP address.
<bindAddress>192.168.0.100</bindAddress>
carbon.xml
The carbon.xml file resides in "<MB_HOME>/repository/conf/" folder. The carbon.xml file needs to be configured in all 3 MB instances. This is to avoid the nodes from using the same ports.
1. Set the "offset" value to a unique value in each node. Assume that following are the assigned offset values.mb1 - 0 mb2 - 1 mb3 - 2For example, mb2 will have the following value.
<Offset>1</Offset>
master-datasources.xml
The master-datasources.xml file resides in "<MB_HOME>/repository/conf/datasources" folder. The master-datasources.xml file needs to be configured in all 3 MB instances. In this file we will be configuring the WSO2 registry database configuration and the WSO2 MB message store database configuration.
1. Fix WSO2 MB registry's url path to H2 database. In the master-datasources.xml, we can see the following section.
<datasource>
<name>WSO2_CARBON_DB</name>
<description>The datasource used for registry and user manager</description>
<jndiConfig>
<name>jdbc/WSO2CarbonDB</name>
</jndiConfig>
<definition type="RDBMS">
<configuration>
<url>jdbc:h2:repository/database/WSO2CARBON_DB;DB_CLOSE_ON_EXIT=FALSE;LOCK_TIMEOUT=60000</url>
<username>wso2carbon</username>
<password>wso2carbon</password>
<driverClassName>org.h2.Driver</driverClassName>
<maxActive>50</maxActive>
<maxWait>60000</maxWait>
<testOnBorrow>true</testOnBorrow>
<validationQuery>SELECT 1</validationQuery>
<validationInterval>30000</validationInterval>
</configuration>
</definition>
</datasource>
For the "url" value we can see the default value as "repository/database/WSO2MB_DB". We have to give the absolute file path for the H2 database. Since each node has their own H2 database, this needs to be done is all MB nodes with their specific paths. Performing this change would imply that each node has their own registry and that it is not shared. Lets say that my MB nodes resides in the following paths.
mb1 - /home/hemikak/Documents/wso2/mb1/ mb2 - /home/hemikak/Documents/wso2/mb2/ mb3 - /home/hemikak/Documents/wso2/mb3/So for mb1, it should be as follows...
<datasource>
<name>WSO2_CARBON_DB</name>
<description>The datasource used for registry and user manager</description>
<jndiConfig>
<name>jdbc/WSO2CarbonDB</name>
</jndiConfig>
<definition type="RDBMS">
<configuration>
<url>jdbc:h2:/home/hemikak/Documents/wso2/mb1/repository/database/WSO2CARBON_DB;DB_CLOSE_ON_EXIT=FALSE;LOCK_TIMEOUT=60000</url>
<username>wso2carbon</username>
<password>wso2carbon</password>
<driverClassName>org.h2.Driver</driverClassName>
<maxActive>50</maxActive>
<maxWait>60000</maxWait>
<testOnBorrow>true</testOnBorrow>
<validationQuery>SELECT 1</validationQuery>
<validationInterval>30000</validationInterval>
</configuration>
</definition>
</datasource>
2. Next step is to comment the current H2 database connection configuration for the messages store. Comment the "WSO2 MB embedded H2 Store" section.
<!-- WSO2 MB embedded H2 Store -->
<datasource>
<name>WSO2_MB_STORE_DB</name>
<description>The datasource used for message broker database</description>
<jndiConfig>
<name>WSO2MBStoreDB</name>
</jndiConfig>
<definition type="RDBMS">
<configuration>
<url>jdbc:h2:repository/database/WSO2MB_DB;DB_CLOSE_ON_EXIT=FALSE;LOCK_TIMEOUT=60000</url>
<driverClassName>org.h2.Driver</driverClassName>
<maxActive>50</maxActive>
<maxWait>60000</maxWait>
<testOnBorrow>true</testOnBorrow>
<validationQuery>SELECT 1</validationQuery>
<validationInterval>30000</validationInterval>
</configuration>
</definition>
</datasource>
3. Next step is to point the WSO2 MB message store to the MySQL database we configured earlier. Uncomment the "MySQL data source" section.
<datasource>
<name>WSO2_MB_STORE_DB</name>
<jndiConfig>
<name>WSO2MBStoreDB</name>
</jndiConfig>
<definition type="RDBMS">
<configuration>
<driverClassName>com.mysql.jdbc.Driver</driverClassName>
<url>jdbc:mysql://localhost/wso2_mb</url>
<username>root</username>
<password>root</password>
<maxActive>50</maxActive>
<maxWait>60000</maxWait>
<minIdle>5</minIdle>
<testOnBorrow>true</testOnBorrow>
<validationQuery>SELECT 1</validationQuery>
<validationInterval>30000</validationInterval>
</configuration>
</definition>
</datasource>
4. Now we have to copy the MySQL driver to "<MB_HOME>/repository/components/lib" folder. You can get the driver from here.
Starting up WSO2 Message Broker 3.0.0
1. Open up the terminal or command prompt in the "<MB_HOME>/bin" folder and run the following command based on the OS../wso2server.sh -Dsetupor
./wso2server.bat -Dsetup
Troubleshooting
1. "Address is already in use". This is because there are nodes that use the same "Offset" value in carbon.xml.
Exception during startup: org.wso2.andes.transport.TransportException: Could not bind to /192.168.0.100:5672
org.wso2.andes.transport.TransportException: Could not bind to /192.168.0.100:5672
at org.wso2.andes.transport.network.mina.MinaNetworkTransport.accept(MinaNetworkTransport.java:147)
at org.wso2.andes.server.Broker.startupImpl(Broker.java:254)
at org.wso2.andes.server.Broker.startup(Broker.java:108)
...
...
Caused by: java.net.BindException: Address already in use
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:444)
at sun.nio.ch.Net.bind(Net.java:436)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
2. "Cannot access the registry". This happens when the WSO2 MB registry database is not properly configured in master-datasources.xml file.
[2015-03-21 10:06:29,050] ERROR {org.wso2.carbon.event.core.internal.builder.EventBrokerHandler} - Can not create the event broker
org.wso2.carbon.event.core.exception.EventBrokerConfigurationException: Cannot get the subscriptions
at org.wso2.carbon.event.core.internal.CarbonEventBroker.loadExistingSubscriptions(CarbonEventBroker.java:90)
at org.wso2.carbon.event.core.internal.CarbonEventBroker.init(CarbonEventBroker.java:67)
at org.wso2.carbon.event.core.internal.CarbonEventBrokerFactory.getEventBroker(CarbonEventBrokerFactory.java:90)
...
...
Caused by: org.wso2.carbon.event.core.exception.EventBrokerException: Cannot access the registry
at org.wso2.carbon.event.core.internal.subscription.registry.RegistrySubscriptionManager.getAllSubscriptions(RegistrySubscriptionManager.java:291)
at org.wso2.carbon.event.core.internal.CarbonEventBroker.loadExistingSubscriptions(CarbonEventBroker.java:78)
... 103 more
Caused by: org.wso2.carbon.registry.core.exceptions.ResourceNotFoundException: Resource does not exist at path /_system/governance/event/topicIndex
at org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.get(EmbeddedRegistry.java:532)
at org.wso2.carbon.registry.core.caching.CacheBackedRegistry.get(CacheBackedRegistry.java:172)
at org.wso2.carbon.registry.core.session.UserRegistry.getInternal(UserRegistry.java:613)
at org.wso2.carbon.registry.core.session.UserRegistry.access$400(UserRegistry.java:60)
at org.wso2.carbon.registry.core.session.UserRegistry$5.run(UserRegistry.java:596)
at org.wso2.carbon.registry.core.session.UserRegistry$5.run(UserRegistry.java:593)
at java.security.AccessController.doPrivileged(Native Method)
at org.wso2.carbon.registry.core.session.UserRegistry.get(UserRegistry.java:593)
at org.wso2.carbon.event.core.internal.subscription.registry.RegistrySubscriptionManager.getAllSubscriptions(RegistrySubscriptionManager.java:266)
... 104 more
Hope you got the cluster up and running.
