Unexpected Interruptions and Auto-Resume

"What to do if the run is not completed at one time" is a question that production-level workflow must answer. Braidrun provides two processing solutions:

• Default: safe break — Mark the execution as INTERRUPTED and wait until you see it clearly before manually "retrying" from a certain step in the middle.
• Optional: Automatically Continue Running — After you explicitly declare "this workflow can continue + which steps are idempotent", the platform will automatically continue from the interruption point.

Default Behavior: Safe Break

When a platform-side interrupt occurs, the platform:

1. Mark the execution status as INTERRUPTED;
2. Keep the results, products, and logs of all completed steps;
3. You receive an on-site notification (email/Slack will also be configured according to your notification);
4. When you go to the execution details page, you can "retry from the failed step" - the platform reuses the results of the previous step.

This default is available to all workflows and requires no configuration. Suitable for:

• Workflow contains external side effects (sending emails/deducting money/modifying database)
• When production is not yet sure "whether it is safe to restart and continue running"

Optional: Automatically Continue Running

If your workflow is a pure data pipeline (read-only API + report generation), you can turn on automatic continuation - after an interruption occurs, the platform will automatically continue running from the interruption point, without you having to manually click.

Two-Tier Opt-In Enabled

For automatic continuation to take effect, you need to declare both layers in YAML:

1. workflow level: Statement recovery.autoResumeOnRestart: true
2. step level: Step-by-step instructions for safe reruns idempotent: true

Complete Example

name: daily-asa-digest
version: "1.0.0"

recovery:
  autoResumeOnRestart: true
  policy: RESUME_FROM_LAST_INCOMPLETE
  maxAutoResumeAttempts: 3

steps:
  - id: fetch_asa_data
    type: sub_workflow
    module: dingyue-module-asa-fetch
    idempotent: true           # 只读拉取，可重跑

  - id: build_excel
    type: sub_workflow
    module: dingyue-module-excel-report
    idempotent: true           # 生成文件到产物目录，可重跑

  - id: notify_team
    type: sub_workflow
    module: dingyue-module-slack-deliver
    # 不写 idempotent —— 默认 false，发消息不可重跑

Three Recovery Policies

idempotent How To Mark

The gold standard for determining whether a step is idempotent:

"If the same input is run twice, will there be any side effects?"

If you can (send external messages, charge money, write orders), don't add idempotent.

Several Situations That Will Not Occur During Automatic Continuation

Even if you configure recovery, the following executions will still not continue running automatically:

• The trigger source is a Webhook - the Webhook payload may be at-most-once, and the platform does not know whether the upstream will retry.
• The trigger source is the Scheduler - the scheduler will re-fire itself in the next round of cron.
• Currently stopped at manual_approval - waiting for approval means that it is originally blocked.
• The workflow definition file was modified during the outage - the platform detected a hash mismatch and is unsure if the new version is compatible.

Manual "Retry from a Step" (Available at Any Time)

Regardless of whether the workflow is configured with recovery or not, the execution details page always provides a "Retry from failed step" button:

1. Open an INTERRUPTED / FAILED execution;
2. Click "Retry from this step" next to the failed/interrupted step;
3. The result of the previously COMPLETED step will be reused; this step starts to run again.

Even if the step isn't idempotent, you can manually retry it - because that's when you as a human consciously decide it's "worth rerunning".

Continue Running Limit

Prevent the system from restarting the same execution in an "infinite loop": each execution will automatically continue running up to 3 times. If it still fails, stop and wait for you to deal with it. You can override (1-20) in recovery.maxAutoResumeAttempts.

Audit

Audit events will be recorded every time automatic continuation is performed:

• execution.auto_resume — Successfully continued running
• execution.auto_resume.skipped — According to the rules, you should not continue running, with reason

Admin can filter and view the audit logs by these two event_types.

Practical Suggestions

• Pure data pipeline (ASA daily, RSS summary): full recovery + idempotent, rest assured that it will continue running automatically.
• If there are external side effects (deductions, sending messages, placing orders): do not open auto resume. Interrupting human decision-making.
• Add manual_approval before the key side effect steps - approval itself is a "natural blocking point" for discontinuous running.

Next

• Execution Monitoring and Logging — How to view execution in INTERRUPTED state
• Best Practices — Comprehensive suggestions for idempotent annotation/splitting/modularization