mongodb

Promoting a MongoDB Slave

MongoDB is one of the commonly used NOSQL document store. For smaller use cases, we might not need a full scaled replica set, instead we can use MongoDB in a traditional way like a Master-Slave architecture. In this blog, i’m going to explain how to convert a Standalone MongoDB server to a Master-Slave Model, and Promoting a Slave instance into a Master node in case of master crash.

Standalone to Master-slave Model.

First, on the master node, we need to add master=true on to the mongodb config file and restart the mongo service. On the new mongo node, which is going to be the slave, add the below config options to the mongodb configuration file.

slave=true
source=xxx.xxx.xxx.xxx:yyyyy        # replace with Mongo Master IP:PORT
autoresync=true

Now restart the mongo service on the slave node and tail the mongo logs, we can see the replication info on it. Below is a sample replication details that can be seen on the mongo logs.

2015-04-18T05:09:41.800+0000 I STORAGE  [replslave] copying indexes for: { name: "xxxxxxxx" }
2015-04-18T05:09:41.801+0000 I STORAGE  [replslave] copying indexes for: { name: "xxxxxxxx" }
2015-04-18T05:09:41.801+0000 I STORAGE  [replslave] copying indexes for: { name: "xxxxxxx" }
2015-04-18T05:09:41.802+0000 I STORAGE  [replslave] copying indexes for: { name: "xxxxxxx" }
2015-04-18T05:09:41.802+0000 I REPL     [replslave] resync: done with initial clone for db: testdb
2015-04-18T05:09:51.135+0000 I REPL     [replslave] repl:   applied 1 operations
2015-04-18T05:09:51.135+0000 I REPL     [replslave] repl:  end sync_pullOpLog syncedTo: Apr 18 05:15:40 5531e87c:1
2015-04-18T05:09:51.135+0000 I REPL     [replslave] repl: sleep 1 sec before next pass
2015-04-18T05:09:52.135+0000 I REPL     [replslave] repl: syncing from host:xxx.xxx.xxx.xxx:yyyyy
2015-04-18T05:10:01.135+0000 I REPL     [replslave] repl:   applied 1 operations
2015-04-18T05:10:01.135+0000 I REPL     [replslave] repl:  end sync_pullOpLog syncedTo: Apr 18 05:15:50 5531e886:1
2015-04-18T05:10:01.135+0000 I REPL     [replslave] repl: syncing from host:xxx.xxx.xxx.xxx:yyyyy
2015-04-18T05:10:11.135+0000 I REPL     [replslave] repl:   applied 1 operations

We can also check the replication status from the Mongo master cli via rs.printReplicationInfo() or db.serverStatus( { repl: 1 } ). We can also check the same on the slave nodes, but by default, read queries are not allowed on the slave and it will throw an error. We can allow reads by running db.getMongo().setSlaveOk() on the slave mongo shell. This will override the restriction and we can use the rs.printReplicationInfo() or db.serverStatus( { repl: 1 } ) to see the replication status.

Promoting a Slave node to Master

This is one the requirement that we keep slave nodes. In case of Master crash, we can easily promote the Slave node and can minimize the interruption. Now promoting a Slave node to Master, follow the below steps.

1) Stop the mongo service on the slave

2) remove all the local files from the mongo data directory

$ cd <mongo_data_directory> && rm -rvf local*

3) Remove the slave configurations from mongo config file, and set `master=true` (This is required if we have more than 1 Slaves, so that the rest of the slaves can connect to new master).

4) Restart the mongo service, now this new master ready to accept writes.

If we have multiple slaves, we need to change the slave source IP, so that they can connect to the new master. But even if the connect to the new master, replication will fail. So we have two methods, either remove the data and perform a new data replication or use force a complete resync to all the slaves using the below command

#On the mongo master shell, run

$ use admin

$ db.runCommand( { resync: 1 }      # This will force a complete resync on all the slaves.

This procedure is useful, if you are using a Standalone/Master-Slave method. For a real HA/Fault tolerant design, replica set proves to be more efficient, where primary master selection takes place automatically if the actual primary node crashes, thus preventing the down time to minimum.

Standard