Cluster offline due to secondary in "RECOVERING" state and primary completely lost!

This is an account of what I faced when called in from a COVID-19 "stay-at-home" restriction to fix a MongoDB cluster that was essentially unusable.  What happened, and this is rare, is that our cluster suffered a network partition for an extended period of time for one of our shards, which I shall refer to as "shard2".  This was then followed by a total loss of data on the primary due to a catastrophic storage failure, which caused us to rebuild the server from scratch.  We use a "PSA" architecture - that is Primary, Secondary, Arbiter.  We use an arbiter instead of second secondary because one of our data centers can't meet the demands of a data-bearing node to make a Primary, Secondary, Secondary configuration. Additionally, we were using the default oplog size of 50GB.  Being that I was out due to COVID-19, the secondary that was partitioned moved to the state of "RECOVERING", once the network partition was resolved and it determined that its current optime was less than the minimum optime availble in the primary's oplog.  This is an odd name for this state, because it means that the secondary's data is STALE and has no hope of catching up to the primary, because the oplog entries that it needs have been deleted from the primary's oplog collection. The only way to "recover" the secondary that is in "RECOVERING" state, is to shut it down, remove its data, start it back up and let it do an "initial sync".  For that, you need either another secondary that is not "RECOVERING" or a primary.  In our case, we never have another secondary because we use the "PSA" architecture, that I mentioned earlier.  So, that leaves just a primary as the only option for recovering the secondary.  However, as I mentioned before, the primary was lost due to storage failure.  So I was left with a secondary in "RECOVERING" state and an arbiter.  The first thing that I tried was to shutdown the secondary, copy it's files to the newly rebuilt primary, and start the primary up.  Now, instead of having just one node in "RECOVERING" state, I had two.  That led me to conclude that there must be something telling each node that it didn't have the latest optime and so it couldn't become primary. So, I started digging around in the "local" database and I found a collection that contained a document that contained a timestamp that I believed was preventing the servers from becoming primaries.  The collection is replset.minvalid.  It has one document and that document has "ts" attribute that in my case was later (or higher) than the current optime shown in "rs.status()" for each node.  When you are working with a replica set without a primary, you cannot modify the data on any node.  So, what I did was to restart each node without the configuration that specifies replica set name or shard server options.  This brings each node up in "stand-alone" mode and allows you to alter the data that each contains.  I then set the "ts" attribute in the aforementioned document to the value found in the "optime" shown from rs.status().  I was able then to restart with the replication and sharding configuration and voila, I now had a working primary and secondary.  There is a big caveat here, though.  When the primary was lost, at that moment, whatever transactions it had processed since the network partition began, were also lost.  However, I was able to bring the cluster back online with minimal loss of data and that was a success.

JUnit and java.io.File

I was writing a new feature that deals with lots of Files for my favorite open-source tool, a database front-end known as SQuirreL SQL Client. Now, I'm a believer that unit tests should be written for most code as it helps you develop loosely coupled, highly cohesive code. So I set about to test my new feature that creates and deletes files using the Java file API (java.io.File). Alas, java.io.File has no interface and forces you to work with actual files! Of course I don't want my JUnit tests creating and deleting files - that would be a nightmare and just create test portability issues. I said to myself, wouldn't it be nice if the java.io.File API had a file factory interface that created file interfaces instead of concrete Files. Then I could simply inject my factory implementations whipped up by my favorite mocking tool (EasyMock) which would create file implementations that I could set expectations on. Well, if you have to use concrete classes, you could certainly wrap them in an implementation that implements an interface and is created by a factory. Hmmm... Eureka! So I wrote FileWrapper to be the interface for FileWrapperImpl - which wraps java.io.Files - and gave it the same interface as java.io.File. It took me about 20 minutes to generate the code thanks to Eclipse. And now I have a drop-in replacement for java.io.File that allows abstract away my dependence on java.io.File and test my code to my heart's content without having to access a single file on my filesystem. The code is LGPL, located in CVS here and consists of the following classes (Click each to download):

FileWrapper.java - same interface as java.io.File
FileWrapperImpl.java - implementation of FileWrapper
FileWrapperFactory.java - interface for the factory that creates FileWrapper
FileWrapperFactoryImpl.java - implementation for FileWrapperFactory

For example, suppose you have some code that simply news up files like:

    public void testFile() {
        File myFile = new File("/path/to/file");
        if (myFile.exists()) {
            doSomething();
        }
    }

It's kind of hard to test that doSomething get's called if File myFile exists and doesn't get called when it doesn't exist without creating and removing the file located at "/path/to/file". So re-write the code like this:

    FileWrapperFactory factory = null;
    public void setFileWrapperFactory(FileWrapperFactory factory) {
        this.factory = factory;
    }

    public void testFile() {
        FileWrapper myFile = factory.create("/path/to/file");
        if (myFile.exists()) {
            doSomething();
        }
    }

This allows you to inject the FileWrapperFactoryImpl from other classes for production code and allows you to inject a Mock FileWrapperFactory when unit testing. Notice how there is very little change to the existing code. That is mostly due to the FileWrapper interface having the same declared methods that are found in the concrete class java.io.File.

Enjoy!