As a follow up to my last blog post on Integration Liferay with Solr Cloud, I felt that this post will help enhance Liferay’s integration with Solr cloud further by reducing the number of indexing requests that are fired off to Solr instances thus reducing the load on Solr servers.
Liferay provides a Solr-web plugin that can be configured to send search and indexing requests to multiple Solr hosts by adding the following configuration to solr-web/WEB-INF/classes/META-INF/solr-spring.xml:
<bean id="com.liferay.portal.search.solr.server.LoadBalancedSolrServerSelector" class="com.liferay.portal.search.solr.server.LoadBalancedSolrServerSelector" />
<bean id="com.liferay.portal.search.solr.server.SolrServerFactoryImpl" class="com.liferay.portal.search.solr.server.SolrServerFactoryImpl">
<constructor-arg>
<map>
<entry>
<key>
<value>solr-01</value>
</key>
<bean class="com.liferay.portal.search.solr.server.BasicAuthSolrServer">
<constructor-arg type="java.lang.String" value="http://localhost:8180/" />
</bean>
</entry>
<entry>
<key>
<value>solr-02</value>
</key>
<bean class="com.liferay.portal.search.solr.server.BasicAuthSolrServer">
<constructor-arg type="java.lang.String" value="http://localhost:8280/" />
</bean>
</entry>
</map>
</constructor-arg>
<property name="solrServerSelector" ref="com.liferay.portal.search.solr.server.LoadBalancedSolrServerSelector"/>
</bean>
Configuring solr-web plugin with multiple Solr hosts is a good way of firing off all indexing requests to all the Solr instances configured within Liferay. However, Since Solr instances are configured as a Solr cloud, every indexing request is replicated on the other Solr instances in the cloud; increasing the number of concurrent search warmers that are indexing and warming the caches at the same time for the exact same indexing request that was already received by all the Solr instances in the cluster from Liferay.
To make Liferay Solr-cloud aware, Solr-web plugin has to be modified to make Liferay instances talk to ZooKeeper quorum to find the state of each Solr instance and send search and indexing requests accordingly. This solution will take a considerable amount of development and while definitely an ideal solution, might not be a feasible one.
We solved the issue of multiple indexing requests in Solr cloud by configuring a load-balancer for Solr instances. The load balancer is configured to receive search and indexing requests from Liferay and forward the request to one of the Solr instances in the cluster using round-robin load-balancing algorithm. Solr-web plugin within Liferay has been configured to send all search and indexing requests to the loadbalancer host:port.
<bean id="com.liferay.portal.search.solr.server.BasicAuthSolrServer" class="com.liferay.portal.search.solr.server.BasicAuthSolrServer">
<constructor-arg type="java.lang.String" value="http://loabalancer-host:8080/solr" />
</bean>
This approach has rendered the following benefits:
- With this approach, only one Solr instance receives the indexing request.
- Since Solr instances have been configured as Solr cloud, the indexing requests is then replicated on other Solr instances in the cluster automatically ensuring that the indexes are in sync.
- This approach has reduced the number of concurrent warmer errors considerably and has improved the performance of Solr instances.
- Solr instances can be added and removed from the loadbalancer for maintenance purposes without impacting search functionality on Liferay.