Sometimes it happens that the MongoDB replica set node goes out of sync from replica set members and due to high lag, it cannot synchronise itself from the replica set members.
There are traditional ways to rebuild the node, like removing the replica set node from the cluster and adding it back, but the issue arises when your database size is in TBs, and node sync takes hours or days. Also, in some cases, it degrades the performance. You use the following steps and rebuild your replica set member in a few minutes.
Step1
Remove the node from cluster
Prod:PRIMARY> rs.remove("<IP Address or hostname of the node to be removed>:27017")
Step2
login to ssh to Secondary Node and delete all the content of mongodb directory folder
Step3
Log in to the Primary node. If you don’t know who is primary run this command rs. slave(ok) from any node and then run rs.status(). rs. status will give you the [rimary node IP address or Hostname. Just ssh or login to primary and shut the MongoDB services
--Stop MongoDB
--linux
service mongod stop
or
sudo systemctl stop mongodb
or
mongod --dbpath /path/to/your/db --shutdown
-- Windows
net stop MongoDB
--MongoDB prompt
> use admin
> db.shutdownServer();
or
mongo --eval "db.getSiblingDB('admin').shutdownServer()"
-- Mac
ps -ef| grep -i mongo
kill -9 <pid></pid>
Step 4
Zip/tar the MongoDB data folder from the Primary node and Copy or SCP to the secondary node Data folder according to mongo.conf. And start the MongoDB services of primary node
Step 5
Start the MongoDB process and if it fails check the MongoDB log. If required inrse the MongoDB service timeout to 800 (TimeoutStartSec=800) and reload the process using systemctl daemon-reload. Once MongoDB services are running fine add the node from the primary. After node addition check synchronization delay using rs. printslavereplicationinfo() and once it’s zero you are GOOD!!!.
Prod:PRIMARY> rs.add("<IP Address or hostname of the node to be removed>:27017")
rs.printslavereplicationinfo()
