Distributed locks and leader election

Key points

• Run a nightly job on ten instances and it runs ten times. One of them has to hold a lock, or be elected the leader.
• A Redis lock is SET key owner NX PX 30000: exactly one caller wins, and the TTL stops a dead holder blocking forever.
• That TTL is the danger. A long GC pause can expire your lease while you still believe you hold it - and now two nodes are writing.
• The fix is a fencing token: each lease hands out a strictly larger number and the storage rejects any write with an older one.
• For real leadership use a consensus store - ZooKeeper, etcd, or Raft inside the database - rather than one key in a cache.
• Cheapest answer of all: make the job idempotent and let it run twice. Then the lock is an optimisation, not a correctness rule.

Example

public class Main {

    // A store that accepts a write only if its token is the newest it has seen.
    static final class Store {
        private long highestToken = 0;
        private String value = "empty";

        boolean write(String who, long token, String v) {
            if (token < highestToken) {
                System.out.println("REJECT " + who + " token=" + token
                        + " (already saw " + highestToken + ")");
                return false;
            }
            highestToken = token;
            value = v;
            System.out.println("ACCEPT " + who + " token=" + token + " value=" + v);
            return true;
        }

        String value() { return value; }
    }

    // Every successful lease hands back a strictly larger token.
    static final class LockService {
        private long nextToken = 41;
        private String holder = null;

        long acquire(String who) {
            if (holder != null) return -1;              // still leased to someone
            holder = who;
            return nextToken++;
        }

        void expire() { holder = null; }                // the TTL ran out
    }

    public static void main(String[] args) {
        Store store = new Store();
        LockService locks = new LockService();

        long tokenA = locks.acquire("worker-A");
        System.out.println("worker-A leased, token " + tokenA);

        locks.expire();                                 // worker-A is in a long GC pause
        long tokenB = locks.acquire("worker-B");
        System.out.println("lease expired, worker-B leased, token " + tokenB);

        store.write("worker-B", tokenB, "written-by-B");
        store.write("worker-A", tokenA, "written-by-A");   // wakes up far too late

        System.out.println("final value : " + store.value());
    }
}

A TTL alone lets two nodes believe they hold the lock. A fencing token is what stops the second one.