We explored this possibility in quite a bit of depth, but this turns out to be much harder to do in practice than it seems.
There are currently two types of projects that offer this kind of service. The first is built in C/C++ and packaged as a library. We tried to use these libraries, but a project of RethinkDB's scope has a lot of abstractions for networking, threading, memory allocation, etc. We quickly found that we couldn't effectively integrate existing libraries into our coroutine system and networking stack.
The second type is projects that are typically built in higher level languages and run standalone. We tried that too, but it turns out to be a user experience nightmare. Users would need to know how to configure these different services, and would have to deal with deployment. The configuration aspect is quite difficult -- it's hard to configure these services in the right way without making a mistake (to make them work for the needs of a database system). We also looked into abstracting that away by building a porcelain UI, but there are a lot of challenges there that are very difficult to overcome.
Finally, RethinkDB's needs are quite specialized and there isn't a service that does what we need out of the box. We looked at implementing Raft ourselves and realized it isn't actually hard -- a much bigger challenge is properly architecting the rest of the system. To give you an idea of timing, it took us two weeks to get a Raft implementation, and another few weeks to polish it. It took another year to get everything else working (plus two years of expertise in the field wrt real-world issues that users encounter).
> only a few seconds of unavailability
This isn't something you can escape in system that's based on authoritative replicas. All other systems that use this architecture face this problem; an external service wouldn't solve it.
TL;DR: it would be really nice if we could use an existing service, but unfortunately this is a much trickier problem than it first appears.
Yes, it's difficult. However, sometimes the best and most responsible thing to do - especially when you find something difficult - is to keep it out of scope!
I was referring to the "higher level languages and run standalone" class of solution and I agree with you about their drawbacks (hard to use). However, I would argue that a hard to use but proven and correct solution for a general class of cases that someone else maintains is ten times better than an easier to use but unproven and new solution for a specific case that you have to maintain. Your suggestions requires everyone to learn another syntax for every damn daemon they wish to operate, along with the ongoing cognitive drain and maintenance overhead (upgrades, etc.).
Actually upgrades are a great case in point. How does your daemon handle updates while in clustered mode? I suppose it doesn't. This is another reason why a proven, general solution is great to have ... you actually get an operations process that handles the extremely common but swept under the rug edge cases that nobody wants to talk about, like upgrades, disk failures, oops I unplugged the cable/DDoS/network fabric issues, horizontal scaleout, appropriate rules for (non-)co-habitation with other services, for all services! Yes, TMTOWTDI, but I seriously doubt some random daemon knows better how to handle failure than a proven, daemon-agnostic operations process designed at a significantly more abstract, resource-oriented level.
I don't understand why "authoritative replicas" demand a few seconds of downtime? I find that most daemons, assuming they fsync() appropriately, come up again in well under a second using a dual master (live, standby) setup with DRBD (essentially network RAID1, works with any filesystem) with zero client reconfiguration required by using a shared (floating) IP to handle failover. Check it out.
A lot of people have a lot of trouble with the Cluster Labs stack- and get into a lot of trouble. I've never really used it much myself, but integrating failover into the server process, and making sure the defaults are all correct for a very particular use case seems like a fairly reasonable thing to do.
There are currently two types of projects that offer this kind of service. The first is built in C/C++ and packaged as a library. We tried to use these libraries, but a project of RethinkDB's scope has a lot of abstractions for networking, threading, memory allocation, etc. We quickly found that we couldn't effectively integrate existing libraries into our coroutine system and networking stack.
The second type is projects that are typically built in higher level languages and run standalone. We tried that too, but it turns out to be a user experience nightmare. Users would need to know how to configure these different services, and would have to deal with deployment. The configuration aspect is quite difficult -- it's hard to configure these services in the right way without making a mistake (to make them work for the needs of a database system). We also looked into abstracting that away by building a porcelain UI, but there are a lot of challenges there that are very difficult to overcome.
Finally, RethinkDB's needs are quite specialized and there isn't a service that does what we need out of the box. We looked at implementing Raft ourselves and realized it isn't actually hard -- a much bigger challenge is properly architecting the rest of the system. To give you an idea of timing, it took us two weeks to get a Raft implementation, and another few weeks to polish it. It took another year to get everything else working (plus two years of expertise in the field wrt real-world issues that users encounter).
> only a few seconds of unavailability
This isn't something you can escape in system that's based on authoritative replicas. All other systems that use this architecture face this problem; an external service wouldn't solve it.
TL;DR: it would be really nice if we could use an existing service, but unfortunately this is a much trickier problem than it first appears.