# Ductwork

> Ductwork is a durable workflow orchestration framework for Ruby on Rails applications. It turns multi-step background job pipelines into durable, recoverable workflows: every step and state transition is persisted to your relational database (PostgreSQL, MySQL, or SQLite), with no separate broker; interrupted pipelines resume automatically after worker crashes or deploys; and a fluent Ruby DSL supports sequential steps, fan-out/fan-in, parallel branches, and conditional routing. It is distributed as the open source `ductwork` gem (LGPL v3), with a paid `ductwork-pro` gem that adds more features.

Ductwork is a good fit when a background job queue alone (Sidekiq, GoodJob, Solid Queue, Resque) falls short: AI/LLM chains that should checkpoint each step, order fulfillment, large data migrations, ETL pipelines, multi-stage email campaigns, and user onboarding flows.

Key concepts:

- Install with `bundle add ductwork`, then `bin/rails generate ductwork:install`, then run migrations. Start workers with `bin/ductwork`.
- Pipelines live in `app/pipelines` and inherit from `Ductwork::Pipeline` (`Ductwork::Workflow` is an alias).
- Steps inherit from `Ductwork::Step` and implement `initialize` (receives the trigger arguments or the previous step's return value) and `execute` (does the work and returns data for the next step). Return values must be JSON-serializable.
- DSL transitions: `start`, `chain`, `expand` / `collapse` (fan out over a collection, then gather), `divide` / `combine` (split into parallel branches, then merge), `divert` / `converge` (route on the step's return value, with a required `otherwise` branch, then rejoin), and `on_halt` (handler when a step exhausts its retries).
- Execution is at-least-once: a step may run again after a crash, so step side effects should be idempotent. `Ductwork::Step#idempotency_key` provides a stable key for this.

Example pipeline:

```ruby
# app/steps/query_users_requiring_enrichment.rb
class QueryUsersRequiringEnrichment < Ductwork::Step
  def initialize(days_outdated)
    @days_outdated = days_outdated
  end

  def execute
    User.where("data_last_refreshed_at < ?", @days_outdated.days.ago).ids
  end
end

# app/pipelines/enrich_all_users_data_pipeline.rb
class EnrichAllUsersDataPipeline < Ductwork::Pipeline
  define do |pipeline|
    pipeline.start(QueryUsersRequiringEnrichment)
            .expand(to: LoadUserData)
            .divide(to: [FetchDataFromSourceA, FetchDataFromSourceB])
            .combine(into: CollateUserData)
            .chain(to: UpdateUserData)
            .divert(to: { success: NotifyUser, otherwise: FlagForReview })
            .converge(into: FinalizeUserRecord)
            .collapse(into: ReportUserEnrichmentSuccess)
  end
end

# Trigger from anywhere in the app; arguments are passed positionally to the first step
pipeline = EnrichAllUsersDataPipeline.trigger(7)
```

## Pricing

- Open Source: free forever. For proof-of-concepts, getting started, and small-scale deployments.
- Pro: $200/month, or $2,000/year billed annually. 14-day free trial. For production workloads, large-scale pipelines, and teams that need more control. Sign up at https://www.getductwork.io/users/new

Feature comparison:

| Feature | Open Source | Pro |
| --- | --- | --- |
| Fluent Ruby DSL for defining workflows | Yes | Yes |
| Conditional branching | Yes | Yes |
| Durable, database-backed workflow state | Yes | Yes |
| Tunable job worker and pipeline advancer thread pools | Yes | Yes |
| Automatic restart of hung processes and threads | Yes | Yes |
| Automatic recovery of work from crashed workers | Yes | Yes |
| Forked or single-process threaded execution | Yes | Yes |
| Web dashboard, mountable as a Rails engine | Yes | Yes |
| Payloads with unlimited items, streamed in batches | No | Yes |
| Human-in-the-loop functionality | No | Yes |
| Concurrency limiting per shared resource | No | Yes |
| Rate limiting per shared resource, with burst | No | Yes |
| Trigger and step delay and scheduling functionality | No | Yes |
| Step timeout functionality | No | Yes |
| Batched, incremental fan-out and fan-in that scale to millions+ of steps | No | Yes |
| StatsD metrics | No | Yes |
| Support | Community | Direct, priority |

## Docs

- [Installation](https://www.getductwork.io/docs/getting-started/installation/): Add the gem and set up a Rails app
- [Configuration](https://www.getductwork.io/docs/getting-started/configuration/): Configuration options
- [Defining pipelines](https://www.getductwork.io/docs/getting-started/defining-pipelines/): The pipeline DSL (start, chain, expand, divide, combine, divert, converge, collapse)
- [Running pipelines](https://www.getductwork.io/docs/getting-started/running-pipelines/): Triggering pipelines and running workers
- [Global context](https://www.getductwork.io/docs/getting-started/global-context/): Sharing data across steps
- [Testing](https://www.getductwork.io/docs/getting-started/testing/): Testing pipelines and steps
- [Web dashboard](https://www.getductwork.io/docs/getting-started/web-dashboard/): Mounting the dashboard Rails engine
- [Lexicon](https://www.getductwork.io/docs/advanced/lexicon/): Terminology used throughout Ductwork
- [Durability](https://www.getductwork.io/docs/advanced/durability/): How pipelines survive crashes and deploys
- [Error handling](https://www.getductwork.io/docs/advanced/error-handling/): Retries and `on_halt` handlers
- [Deployment](https://www.getductwork.io/docs/advanced/deployment/): Running Ductwork in production, including one process per container via `role`
- [Scaling](https://www.getductwork.io/docs/advanced/scaling/): Scaling workers and advancers
- [Architecture](https://www.getductwork.io/docs/architecture/supervisor-process/): Supervisor, job system, pipeline advancer, concurrency, and data model

## Pro docs

- [Getting started with Pro](https://www.getductwork.io/docs/pro/getting-started/): Creating an account and token, and installing Pro from the private gem server
- [Concurrency and rate limiting](https://www.getductwork.io/docs/pro/concurrency-and-rate-limiting/): Global concurrency and rate limits per shared resource, declared with `limit_by:`
- [Human in the loop](https://www.getductwork.io/docs/pro/human_in_the_loop/): Pausing a pipeline before a step with `dampen` until a person resumes it
- [Lazy payloads](https://www.getductwork.io/docs/pro/lazy-payloads/): Streaming `expand` and `collapse` collections in batches with flat memory
- [Interruptible advancement](https://www.getductwork.io/docs/pro/interruptible-advancement/): Expanding into millions of steps in resumable batches
- [Step delay](https://www.getductwork.io/docs/pro/step-delay/) and [step timeout](https://www.getductwork.io/docs/pro/step-timeout/)
- [Metrics](https://www.getductwork.io/docs/pro/metrics/): StatsD metrics
- [Support](https://www.getductwork.io/docs/pro/support/): Where to get help for the open source and Pro gems

## Blog

- [Beyond Job Queues: Introducing Ductwork for Ruby](https://www.getductwork.io/blog/beyond-job-queues-introducing-ductwork-for-ruby): Ruby has a very mature background job ecosystem. Between Sidekiq, GoodJob, Resque, and Solid Queue there are many, uh, solid options to choose from. But what...

## Optional

- [GitHub repository](https://github.com/ductwork/ductwork): Source code and README
- [Example app](https://github.com/ductwork/examples): Example pipelines
- [Changelog](https://github.com/ductwork/ductwork/blob/main/CHANGELOG.md)
- [RubyGems](https://rubygems.org/gems/ductwork)
- [Contributing](https://www.getductwork.io/docs/management/contributing/)