Fault tolerance and reconciliation
The central safety property is: a Spreadr trade can never be left with an unhedged, unaccounted-for position by a process failure. This is achieved by ordering durable writes ahead of irreversible actions.
Ledger-before-hedge
Every fill is written to a durable, append-only ledger with a monotonic sequence number before its hedge order is sent. The order of operations is strict:
- Maker fill observed on venue A.
- Fill durably recorded (state → Pending-Fill-Confirmation).
- Hedge order sent to venue B.
- Hedge fill durably recorded.
Because step 2 precedes step 3, there is no window in which a fill exists but is unknown to the system. If the worker dies at any point, the ledger unambiguously records how far the sequence progressed.
Recovery and reconciliation
When a worker fails, a replacement worker recovers the trade from the durable ledger and reconciles it against the venues' reported state:
- Orphan orders — orders the ledger shows as placed but which the trade no longer needs — are cancelled.
- Unhedged fills — fills recorded before a hedge completed — are hedged to restore delta-neutrality.
- Duplicate protection — because each fill and hedge carries a sequence number, a recovering worker can tell whether a hedge was already sent and will never double-hedge the same fill.
Reconciliation is continuous, not only post-crash: the worker periodically compares its ledger-derived view of positions and open orders against what each venue reports, and corrects drift.
Audit trail
Every fill, hedge, rejection, cancellation, and amendment is recorded with a sequence number and timestamp. The result is a complete, ordered history of a trade that a user can retrieve for their own accounting, reconciliation, or review.