Saturday, September 10, 2011

Multicore master-slave replication in Solr Cloud

  1. If you've already done some work with Solr Cloud then you may want to start fresh by cleaning up any previous ZooKeeper configuration data in order to run this example exercise smoothly.
    cd /trunk/solr/example/solr
    rm -rf zoo_data
  2. We will create the following setup:
    1. there will be 2 Solr instances, each with 3 cores
    2. 1 of the 3 cores will be a master and the other 2 will be slaves
    3. the slaves of one instance will be configured to use the master of the other one
    4. The infrastructure will look like:
      • Solr-Instance-A
        • master1 (indexes changes for shard1)
        • slave1-master2 (replicates changes from shard2)
        • slave2-master2 (replicates changes from shard2)
      • Solr-Instance-B
        • master2 (indexes changes for shard2)
        • slave1-master1 (replicates changes from shard1)
        • slave2-master1 (replicates changes from shard1)
  3. We can reuse the multicore directory of the out-of-the-box example. It already has the core0 and core1 directories, lets create an additional core:
    cd /trunk/solr/example/multicore
    cp -r core0 core2
  4. If we were NOT using Solr Cloud which has us upload an universal configuration at startup, then we would perform the following sub-steps:
    • Replace any mention of core0 with core2
      sed -ibak 's/core0/core2/g' core2/conf/solrconfig.xml
      sed -ibak 's/core0/core2/g' core2/conf/schema.xml
      sed -ibak 's/zero/two/g' core2/conf/schema.xml
    But right now these are pointless because each individual core's configuration will not be used ... instead the configuration in ZooKeeper will be used.
  5. Edit solr.xml as follows:
  6. Copy over the example's zoo.cfg from the single-solr setup over to the multicore setup:
    cd /trunk/solr/example
    cp ./solr/zoo.cfg ./multicore/
  7. Copy example to example2 in order to create another Solr instance
    cd /trunk/solr
    cp -r example example2
  8. Edit solr.xml for example2 as follows:
    cd trunk/solr/example2/multicore
    vi solr.xml
  9. So where is the configuration that we will we be uploading to ZooKeeper? And what should we edit? Well, the most well formed configuration is sitting in the out-of-the-box single core example so let us simply upload it from there to ZooKeeper! And have it configured such that it can be applied conditionally to all our cores based on the java params that we specify at startup!
    1. Let us begin by editing the solrconfig.xml file of the single solr core example as follows:
      cd /trunk/solr/example/solr/conf
      vi solrconfig.xml
    2. We cannot pass the true/false values via -Denable.master or -Denable.slave at startup because it will end up applying globally to all the cores (1 master & 2 slaves) and there isn't a way to start only one core at a time from the command line. So we must leverage each individual multicore's solr.xml to provide core specific values as follows:
      cd /trunk/solr/example/multicore
      vi solr.xml
      cd /trunk/solr/example2/multicore
      vi solr.xml
    3. Now let us start the 1st instance of the multicore Solr with the appropriate java params and let ZooKeeper know exactly where to get its universal-config (bootstrap_confdir) from:
      cd /trunk/solr/example
      #java -Dbootstrap_confdir=./solr/conf \
      #     -Dsolr.solr.home=multicore \
      #     -DmasterHost=localhost -DmasterPort=7574 -DmasterCoreName=master2 \
      #     -DzkRun \
      #     -jar start.jar
      java -Dbootstrap_confdir=./solr/conf -Dsolr.solr.home=multicore -DmasterHost=localhost -DmasterPort=7574 -DmasterCoreName=master2 -DzkRun -jar start.jar
    4. Start the 2nd instance of the multicore Solr with the appropriate java params:
      cd /trunk/solr/example2
      #java -Djetty.port=7574 \
      #     -DhostPort=7574 \
      #     -Dsolr.solr.home=multicore \
      #     -DmasterHost=localhost -DmasterPort=8983 -DmasterCoreName=master1 \
      #     -DzkHost=localhost:9983 \
      #     -jar start.jar
      java -Djetty.port=7574 -DhostPort=7574 -Dsolr.solr.home=multicore -DmasterHost=localhost -DmasterPort=8983 -DmasterCoreName=master1 -DzkHost=localhost:9983 -jar start.jar
  10. Now, you can check the ZooKeeper status here:

And that's all there is to it, feel free to leave any feedback as comments below.


  1. Great post! This is exactly what I am looking for.
    I got it set up and running. However the zookeeper admin page is saying that shard1 has master1, slave1-master1 and slave2-master2, and shard2 has master2, slave1-master2 and slave2-master2. Shouldn't the slaves be the other way round?
    Also, neither of my dashboard is working (http://localhost:7574/solr/ and http://localhost:8983/solr/ ). Content is not loading. No error in the logs though. They both worked when I tried the singlecore SolrCloud following this page:

  2. Allow me to explain:
    1) shard1 is half of the entire dataset and all the indexing is done for it on master1.
    1a) The replicated instances which are used for searching are called slave1-master1 and slave2-master1.
    1b) The postfix called "-master1" is present in each of their names to indicate that they are replicated slaves of master1.
    1c) If that is the case then doesn't it make sense that they actually manage shard1 part of the dataset? Any clearer?

    If not, let me go on to what I think may be confusing you. You may be confused by the fact that the slaves of master1 are actually sitting on the Solr instance that is managing master2 and visa-versa. The reason behind this is that someone very smart in the Solr mailing-list alerted me to the fact that if one of the two instances of Solr goes DOWN then at least I have 2 slaves of a shard to read from (they can't be updated though) and one master of the other half of the shard to read & write to :)

  3. Litte typo remark: instancedir needs to be "instanceDir". It's case-sensitive.

  4. Hi,
    i configure as explain in a single server but i get follow problems

    SEVERE: Error while trying to recover.
    org.apache.solr.client.solrj.SolrServerException: Server at http://fc:8983/solr was not found (404).
    at org.apache.solr.client.solrj.impl.HttpSolrServer.request(
    at org.apache.solr.client.solrj.impl.HttpSolrServer.request(
    Jun 15, 2012 5:33:21 PM doRecovery
    SEVERE: Recovery failed - trying again...

    Any idea?

  5. another little remark:
    i add -Dcollection.configName=scaleDeep in java start

    otherwise i get the follow SEVERE pb
    SEVERE: Specified config does not exist in ZooKeeper:scaleDeep
    Jun 15, 2012 5:43:07 PM org.apache.solr.common.SolrException log
    SEVERE: Specified config does not exist in ZooKeeper:scaleDeep

  6. This comment has been removed by a blog administrator.