This content originally appeared on Level Up Coding - Medium and was authored by Randula Koralage
The following error is common in improperly synced MongoDB replica sets. The reason can be, that the instance couldn’t sync with primary or secondary for a long time due to some reason, and then it has gone into a state that is never being allowed to sync with another.
Long-time cluster failures and long-time connection problems can be a cause to occurring this error.
Let’s Identify the Victim
Before going into a solution, let’s gather some information.
1. Check Replicaset Set Status
Go inside the primary mongo and login as db admin
kubectl exec -it podname mongo
use admin
db.auth(“ username ”, “ password ”)
View the replica set status using rs.status()
You will see the HeartBeatMessage as “Our replica set configuration is invalid or does not include us” and also will observe that syncSourceId is negative and syncingTo property value is empty.
2. The plan
We need to take our replica set back to a healthy state and all mongo instances should be synced properly. First please check whether all the network connections are properly working. Make sure the machines with MongoDB replicas can communicate with each other.
Simply what we are going to do is,
removing the particular error-prone MongoDB instance from the replica set and re-adding it accordingly. Sametime you have to allow the member to re-sync properly by deleting the data folder of the particular node.
In my case, the database wasn’t too large, hence deleting the data folder worked well. But when your database is too large you can copy the data folder from another secondary machine that is properly synced.
3. Follow the Steps
Step 1
Identify the machine which has the broken replica set member.
Step 2
Stopping mongodb pod in particular node.
You can not delete the data folder of a node when the mongodb pod is running on a particular node. Because of that, you have to stop the mongodb pod in that particular node. It is a tricky situation as you still delete the pod because Kubernetes automatically restart the pod to maintain the number of replicas.
To avoid making the pod, you can label the pod in a way it is restricted for mongodb instances.
ex: my nodes are labeled with database=yes and only these nodes are taken to create mongodb pods. I changed the label of the damaged node to databse- so that this node will be ineligible for mongodb pods.
Assume that worker1 is the node with error-prone replica set member mongo-3.
kubectl label node worker1 database-
kubectl delete pod mongo-3
This will avoid Kubernetes from making a mongodb pod in a particular node. Please note the label name depends on your application setup.
Step 3
Delete the data folder. This will allow to re-sync your node from the begining.
Step 4
Delete the member from the replicaset
kubectl exec -it podname mongo
use admin
db.auth(“ username ”, “ password ”)
rs.remove(‘mongo-3:27017”)
Step 5
Restore the correct label and allow the pod to get start. Please note the label name depends on your scenario.
kubectl label node worker1 database=yes
Now mongo pod should be created in this node again
Step 6
Re-add the replicatet member. In this step please use correct properties according to your senario.
rs.add({_id:3 host: “mongo-3:27017”})
Now your damaged mongo node should be started to start again and attempting to sync. Within few minutes you’ll see that it is change as a Secondry and a syncSource is met.
When there are two damaged replicaset members in rs.config() follow first 2 steps as it is and then remove both members at once. Then follow rest of the process one after another. Start mongo of one node and add it to the rs.config() and wait some time till it’s state change as Secondary. Then follow the steps for second node.
This is the way that I deal with the Our replica set configuration is invalid or does not include us and after the full process my replicaset status became normal and healthy.
If you know any other way to fix this issue or any logical reason for this please let me know as a response.
Thank you
Our replica set configuration is invalid or does not include us was originally published in Level Up Coding on Medium, where people are continuing the conversation by highlighting and responding to this story.
This content originally appeared on Level Up Coding - Medium and was authored by Randula Koralage
Randula Koralage | Sciencx (2021-12-22T00:58:03+00:00) Our replica set configuration is invalid or does not include us. Retrieved from https://www.scien.cx/2021/12/22/our-replica-set-configuration-is-invalid-or-does-not-include-us/
Please log in to upload a file.
There are no updates yet.
Click the Upload button above to add an update.