The Atomicity Gap: Durable Ruby Workflows with Ductwork

Photo by Farnoosh Abdollahi on Unsplash

There is a gap in the Ruby ecosystem. We have no strong answer for durable workflows. Every background job system in wide use — Sidekiq, GoodJob, Resque, Solid Queue — will faithfully run a single job for you, and will even retry it if it fails. But the moment your "work" is actually a sequence of steps, each with its own side effects, all of them go quiet on the question that matters most: what happens when the process dies in between?

Two Patterns, One Silent Failure

Look past the framework you're using and you'll find the same two patterns everywhere, both of them quietly wrong.

A job enqueues another job. Some perform method finishes its work, then calls NextJob.perform_later. That looks like one operation. It is actually two, running against two different systems — your database and your queue — and nothing ties them together. The process can die between them. The DB commit and the enqueue can straddle a crash. Depending on which side of the crash you land on, the chain silently stops, or you get a duplicate hop nobody asked for.

A single job has more than one side effect. charge_card; send_receipt_email, in that order, inside one perform method. If the process dies after the charge but before the email — or the whole job gets retried after a partial success — you either drop a side effect or double one up. Nothing in the job system distinguishes "this half already happened" from "this half hasn't happened yet."

Both of these are the same underlying problem wearing different clothes. Call it the atomicity gap: the space between "the thing I cared about happened" and "the system durably recorded that it happened and moved on to the next thing." Rails' after_commit { SomeJob.perform_later } narrows that gap — at least the enqueue only fires after the write is durable — but it doesn't close it, because the enqueue itself can still fail after the commit. Wrapping the enqueue inside the same database transaction as the write doesn't help either, for the opposite reason: the queue isn't part of your database, so there's no transaction that spans both.

Ductwork's answer is to treat these as two different problems, because they are. It closes the first gap structurally: advancing from one step to the next is exactly-once, full stop, nothing for you to build. It does not — cannot — close the second gap structurally, and says so plainly. Instead it hands you the tool to close it yourself: at-least-once execution, made safe with an idempotency key. Conflating the two is exactly why the usual ad hoc fixes — a transaction around the enqueue, a bare retry_on with no dedupe — never fully work. They're solving for one guarantee while quietly needing the other.

Why the Naive Fix Doesn't Work

Picture the standard shape of a Sidekiq or ActiveJob chain:

class ChargeCustomerJob
  include Sidekiq::Job

  def perform(order_id)
    order = Order.find(order_id)
    Stripe::Charge.create(amount: order.total, customer: order.stripe_id)
    order.update!(status: "charged")

    SendReceiptEmailJob.perform_async(order_id)
  end
end

There are two failure interleavings here, and both are bad:

  • The process dies after order.update! commits but before perform_async reaches Redis. The order is marked charged, but no email job ever gets enqueued. The chain just... stops. Nothing errors, nothing retries, nothing tells you.
  • Flip the order — enqueue first, then update the order — and you get the opposite failure: the email job is queued, then the process dies before the charged update commits. Now a retry of ChargeCustomerJob sees a customer who was never marked charged, tries to charge them again, and you've got a duplicate Stripe charge with a receipt email already on the way for the first one.

Now look at a single job doing two things:

def perform(order_id)
  order = Order.find(order_id)
  charge_card(order)
  send_receipt_email(order)
end

Sidekiq's retry logic assumes the whole method is safe to re-run from the top. charge_card and send_receipt_email are not idempotent by default. A crash mid-method, followed by a retry, either re-charges the card or re-sends the email — whichever one already succeeded gets to happen again.

The common thread: generic job queues give you at-least-once delivery of a single job, full stop. They say nothing about the transition from one job to the next, and nothing about a job with multiple effects packed inside it. These are two different problems, and people reach for the same tool — "just retry it" — to paper over both.

Enough Vocabulary to Follow Along

Ductwork models a workflow as a Pipeline (or Workflow — same thing, different name for readability). A Pipeline produces a Run. A Run is made of one or more Branches — a branch exists because pipelines fan out and merge, and each parallel thread of execution is its own branch. Each Branch moves through a sequence of Steps, and each Step is backed by a Job that actually gets executed — potentially across multiple Executions, if it crashes or errors and needs a retry.

The one architectural fact the rest of this post leans on: Ductwork tracks "did the step's work finish" — that's Step, Job, Execution — completely separately from "did the pipeline move to the next step" — that's Transition and Advancement. Two different record types, two different guarantees, tracked independently. That separation is what makes both guarantees possible at the same time, instead of forcing a single record type to promise things it can't keep.

Closing Gap #1: Step Advancement Is Exactly-Once

Here's chain_branch, the code that runs when one step's output flows into the next step in sequence — the exact scenario from the ChargeCustomerJob example above:

def chain_branch(edge, transition, advancement)
  input_arg = Ductwork::Job.find_by(step: latest_step).return_value
  node = edge[:to].sole
  klass = run.parsed_definition.dig(:edges, node, :klass)
  started_at = Time.current

  with_claim_fence do
    latest_step.update!(status: :completed, completed_at: Time.current)
    next_step = steps.create!(
      run: run,
      node: node,
      klass: klass,
      status: "in_progress",
      to_transition: "default",
      started_at: started_at
    )
    Ductwork::Job.enqueue(next_step, input_arg)

    now = Time.current
    advancement.update!(completed_at: now)
    transition.update!(completed_at: now)
    release!
  end
end

Marking the previous step complete, creating the next Step and its Job, enqueuing it, and closing out the Transition/Advancement bookkeeping all happen inside one database transaction, guarded by with_claim_fence — a compare-and-swap on a claim_token that only lets the block run if this advancer still, provably, owns the branch. There is no window where step N is done but step N+1 was never created. There is no window where step N+1 gets created twice. This is a two-phase-commit-style record wrapped around the graph mutation itself — not just around one job in isolation, but around the boundary between "job A finished" and "job B exists," which is precisely the boundary Sidekiq and ActiveJob have no way to reach into.

If the process dies mid-advancement, the transaction simply never commits, and the branch is left claimed by a process that no longer exists. Every advancer runs as a Ductwork::Process row that refreshes a last_heartbeat_at timestamp on an interval; when a supervisor notices a process's heartbeat has gone stale past the configured timeout, it reaps that process, and reaping walks its open Advancements and calls process_crashed! on each — a conditional update guarded by completed_at: nil so a reap racing a real completion can't clobber it — then releases the branch, clearing its claim_token and dropping its status back to in_progress so a fresh advancer can reclaim it. (A second, backstop sweep catches claims whose owning Process row was destroyed before the claim was even created, closing a narrower create-vs-reap race.) The advancer that picks the branch back up doesn't try to patch a half-applied transition — there isn't one, because the transaction that would have applied it never committed. It re-runs chain_branch from scratch: same previous step (already completed from the prior attempt, or not — the whole block is transactional, so either the full write landed or none of it did), same next step to create. "Retry" here means retry the atomic unit, never patch a half-applied state.

One-line version: this is what a database transaction does for two rows in one table, except Ductwork is doing it across the boundary between "a job finished" and "the next job was created" — a boundary that lives partly in your database and partly in your queue, which is exactly the boundary generic job systems can't wrap a transaction around.

Gap #2 Doesn't Have a Structural Fix — So Ductwork Tells You the Truth About It

Here's Ductwork's README, verbatim, on what it will and won't promise:

Ductwork guarantees at-least-once, never exactly-once, execution of each step.

That's not a limitation apologized for in a footnote — it's the honest answer, and it's worth sitting with why it's the only honest answer. Ductwork can close the advancement gap because advancement is entirely inside Ductwork's own database: marking a step complete and creating the next one are both rows in tables Ductwork controls, so they can share one transaction. A step's side effect is different in kind. It might be a Stripe charge, an SMTP send, a webhook POST to a system Ductwork has never heard of. There is no transaction that spans your database and Stripe's API. No workflow engine — not Ductwork, not Temporal, not anything else — can wrap an arbitrary external call in a commit. Any tool that tells you otherwise is lying to you.

You can see the honesty in the shape of Execution#call:

begin
  output_payload = instance.execute
rescue StandardError => e
  errored!(e, owner_process_id)
  return
end

# AT-LEAST-ONCE CONTRACT: `instance.execute` has already run and any
# side effects it performed are now durable. The commit below can still
# fail (CommitFailed) if the reaper clobbered this claim, in which case
# `crashed!` creates a fresh availability and the job runs AGAIN.
succeeded!(output_payload, owner_process_id)

instance.execute — your step's actual code, your actual side effect — runs before any of Ductwork's own bookkeeping commits. If the process crashes after execute returns successfully but before succeeded!'s transaction lands, Ductwork has no way to know the side effect already happened. It follows its stated contract and lets the job run again.

So what's the design lever, if there's no structural fix? Keep steps small. One side effect per step. This is what actually addresses the second gap from the intro — not by making retries safe on their own, but by shrinking what a retry has to be safe for. Don't put charge_card and send_receipt_email in one job; make them two steps, chained. Each step now has exactly one thing to make idempotent, and gap #1's guarantee — exactly-once advancement — is what reliably gets you from the charge step to the email step once the charge is done.

That reframes the two gaps from the intro as one discipline, not two separate fixes: decompose your work until each unit has a single side effect, then let Ductwork's exactly-once advancement chain those units together safely. The granularity is your job. The sequencing is Ductwork's.

Making At-Least-Once Safe: idempotency_key

Every Ductwork::Step exposes an idempotency_key:

alias_attribute :idempotency_key, :id

That one line is doing more than it looks like. The key is the step's own database ID — stable across every retry or crash-recovery Execution of that step instance, because all of those executions share one Step row. But a genuinely new run of the pipeline gets a genuinely new step, and therefore a new key. There's no timestamp or attempt number folded in, and that's deliberate: retries of the same logical attempt should collide on the same key, not fan out into distinct ones. If attempt number were part of the key, every retry would look like a brand-new operation to whatever you're deduping against — the exact opposite of what you want.

Put it to work against Stripe, which accepts idempotency keys natively:

class ChargeCustomer < Ductwork::Step
  def initialize(order_id)
    @order_id = order_id
  end

  def execute
    order = Order.find(@order_id)

    Stripe::Charge.create(
      { amount: order.total, customer: order.stripe_id },
      { idempotency_key: idempotency_key }
    )

    order.id
  end
end

If this step gets re-run after a crash, Stripe sees the same idempotency key and returns the original charge instead of creating a second one. No app-level dedupe table required — Stripe's own idempotency layer does the work, and Ductwork just needed to hand it a key that's stable for exactly the right lifetime.

For a side effect that lives entirely in your own database, skip the external API and let a unique index do the same job:

class RecordChargeAttempt < Ductwork::Step
  def initialize(order_id)
    @order_id = order_id
  end

  def execute
    ChargeAttempt.upsert(
      { order_id: @order_id, idempotency_key: idempotency_key, status: "charged" },
      unique_by: :idempotency_key
    )

    @order_id
  end
end

An upsert on idempotency_key turns a re-run into a no-op update instead of a duplicate row.

Not every idempotency problem is "pass a key to an external system," though. Sometimes what you need is "have I already recorded X for this run" — a fact that isn't tied to one step's retries but shared across the whole pipeline. That's what Ductwork::Context is for: a run-scoped key/value store, backed by a [run_id, key] uniqueness constraint, where set raises Ductwork::Context::OverwriteError if the key is already there instead of silently overwriting it. Rescue that error and you have a race-safe "have I done X for this run" guard, enforced by the database rather than by careful callers remembering to check first. It won't replace a Stripe idempotency key, but it's the right tool for a smaller, secondary class of dedupe checks that live at the run level rather than the step level.

Before and After

Before: one Sidekiq job, charge_card then send_receipt_email, wrapped in hope.

After, as a Ductwork pipeline:

class ChargeOrderPipeline < Ductwork::Pipeline
  define do |pipeline|
    pipeline.start(ChargeCustomer)
            .chain(to: SendReceiptEmail)
  end
end

Two steps, chained. Now walk the same failure scenarios from earlier and see what actually happens to each one:

Crash after the charge succeeds, before the pipeline advances. The reaper retries the advancement, not the charge. The charge step is already marked completed — that write landed in the same transaction as everything else in chain_branch — so the next step gets created exactly once, never zero times, never twice.

Crash mid-send on the email step, then retried. SendReceiptEmail#execute runs again. This time it's a single, small, easy-to-make-idempotent operation — key the outbound Message-ID off idempotency_key, or check a sent_at timestamp before sending — instead of one half of a job where the other half already has irreversible side effects riding along with it.

That's the whole payoff, in one sentence: exactly-once advancement handles the sequencing risk, and the idempotency key handles the re-execution risk, and neither mechanism has to pretend it can do the other one's job.

Wrapping Up

Two gaps, two different guarantees, deliberately not blurred into one. Ductwork closes the sequencing gap for you — exactly-once bookkeeping for moving through the pipeline graph is something the framework builds so your application code doesn't have to. It cannot close the re-execution gap for you, because no framework can reach into an arbitrary external side effect and make it transactional. What it can do, and does, is guarantee at-least-once execution honestly, and give you idempotency_key as the primitive for making that safe — because only your application code knows what "duplicate" actually means for a given side effect.

If you want the crash-recovery mechanics in more depth — reapers, claim tokens, the full Advancement lifecycle — the README's Delivery Guarantees section is the place to start, and the code behind it is worth reading directly.

(Ductwork Pro pushes the same idea further at fan-in time: instead of locking a parent branch for the duration of a collapse, every completing sibling does one conditional counter bump, and whichever sibling's update finds the counter already one short of the expected count is the one that fires the merge. Same discipline as idempotency_key, just applied to coordinating N branches instead of one step — maybe a post of its own.)