Retries, Failures, and Skipped Steps

This guide explains how Duvo handles things that go wrong during a Job — what the system retries automatically, what counts as a permanent failure, how to write SOPs that control recovery behavior, and how to prevent duplicate actions when a Job runs more than once.

---

How Duvo Recovers from Transient Problems

When a Job hits a problem that is likely temporary — a network timeout, an overloaded external service, a rate-limit response — Duvo automatically returns the Job to the queue and retries it. You do not need to do anything. The Job will appear as Running or briefly as Pending in your Jobs List while the retry is in progress.

What counts as a transient problem:

• Connection timeouts and network errors (ETIMEDOUT, ECONNRESET, socket hang up)
• HTTP 429 Too Many Requests (rate limiting)
• HTTP 502, 503, 504 (gateway errors and temporary service unavailability)
• Brief overload on the Duvo infrastructure

How retries work:

Duvo uses exponential backoff — each successive attempt waits longer than the previous one, up to a maximum of 60 seconds. The system tries up to a configured number of times before giving up. If the problem resolves during that window (which it usually does for transient errors), the Job continues normally.

You will not see a Failed status for Jobs that recovered through automatic retry.

---

When a Job Fails Permanently

Some failures cannot be recovered by retrying. Duvo marks the Job as Failed immediately in these cases:

Failure type	What it means	What to do
Connection could not be resolved	A Connection the assignment needs was disconnected or its credentials expired before the Job started	Go to Connections, re-authorize the affected Connection, then retry the Job
Job content is too large	The data being sent to the assignment exceeds the system limit	Break the task into smaller batches in your SOP
Invalid input (HTTP 422)	The Job was started with data the assignment cannot process	Fix the input data and start a new Job
Max retries exhausted	A transient error persisted through all automatic retry attempts	Check whether the external service is experiencing an outage; retry the Job once the service recovers

You can manually retry a failed Job from the Jobs List — click the row to open the Job, then use the retry action. For case-based workflows, see Retrying and Updating Cases.

---

Understanding Job Statuses

Status	What is happening
Running	The assignment is actively processing
Needs Input	The assignment is waiting for a human response before it can continue
Postponed	A case has been deliberately scheduled for a later time
Completed	The assignment finished and explicitly resolved the case or Job
Failed	The Job ended without completing — either an error occurred or the assignment finished without marking the case as done
Stopped	A team member or the API stopped the Job manually

A Job that ends without the assignment explicitly completing, postponing, or handing over the underlying case is automatically marked Failed. This means every SOP that processes cases should clearly instruct the assignment when and how to resolve each case.

---

SOP Patterns for Controlling Recovery Behavior

The assignment follows your SOP instructions when deciding how to handle errors it encounters mid-run (as distinct from the system-level retry logic above, which handles failures before or during dispatch). Use these patterns to tell the assignment how to behave when something goes wrong.

Retry an action a limited number of times

If sending the email fails, wait 30 seconds and try again.
Retry up to 3 times before marking the case as failed with the reason.

Fail fast on a specific error

If the customer record cannot be found in Salesforce, stop immediately.
Mark the case as failed with the note: "Customer not found — requires manual lookup."
Do not attempt to create any records.

Postpone and come back later

Use this when the condition needed to proceed may resolve on its own:

If the report file is not yet available in Google Drive, postpone the case for 2 hours.
On the next attempt, check again before proceeding.

When the assignment postpones a case, its status changes to Postponed with the scheduled retry time. At that time, Duvo automatically picks it up again.

Escalate to a human instead of failing

Use HITL when you want a person to decide what to do rather than having the Job fail silently:

If the invoice total is outside the expected range ($0–$50,000), do not process it.
Request human review. Title: "Invoice out of range — [vendor] — $[amount]".
Description: include the full extracted invoice data and the specific value that triggered this check.
Wait for the human to either approve with corrections or reject.

See Designing Human-in-the-Loop Workflows for guidance on choosing between approval gates and automatic failures.

---

Skip Semantics: Skipped vs Failed vs Postponed

These three outcomes look similar but mean different things:

Outcome	What triggers it	What happens next
Skipped step	The assignment’s SOP includes a condition that was not met (e.g., “only send a notification if the amount is above $1,000”)	The Job continues; the skipped step simply does not execute
Failed	The Job ends without the assignment completing, postponing, or handing over the case; or the assignment explicitly marks the case as failed	The case moves to Failed status; no further processing until manually retried
Postponed	The assignment explicitly postpones the case to a future time	The case moves to Postponed status and is automatically retried at the scheduled time

When an operator rejects a HITL request

When an operator clicks Reject on an approval request, the assignment receives a rejection signal. What happens next depends entirely on how your SOP handles it. Duvo does not automatically halt or retry the Job — your SOP must define the response.

Common patterns:

If the approval is rejected, ask the operator: (a) Revise and resubmit, (b) Cancel this item.

If rejected twice in a row for the same case, stop processing and mark the case as failed
with the note: "Halted after two rejections — requires manual review."

Without explicit instructions for the rejection path, the assignment may loop, stall, or make an unsafe assumption. Always define what a rejection means for your workflow.

When condition data is missing

If the assignment reaches a conditional step but the data needed to evaluate the condition is absent:

If the vendor's payment terms are not specified in their record, treat them as Net 30
and continue. Add a note to the case: "Payment terms defaulted to Net 30 — verify with vendor."

Or:

If the customer ID is missing from the email, request HITL before proceeding.
Do not guess the customer — an incorrect association will create a bad record.

---

Idempotency: Preventing Duplicate Actions on Retry

When an assignment retries a step after an earlier attempt failed, there is a risk of performing an action twice — sending two emails, creating two records, charging a customer twice. Use these patterns to prevent that.

The check-before-write pattern

Always check whether the action has already been completed before performing it:

Before creating the invoice in NetSuite, search for an existing invoice
with the same PO reference number and vendor. If one already exists, skip
creation and log: "Invoice already exists — [invoice ID]. Skipping."
Only create a new record if none is found.

Tag actions with a reference ID

Use a stable identifier from your source data (PO number, order ID, ticket ID) to make each action uniquely identifiable:

When sending the confirmation email, set the subject to include the order reference:
"Order confirmed — [PO-2024-0847]". If this email has already been sent (check the
sent items for a subject matching this reference), skip sending and log that the
confirmation was already delivered.

Per-connection idempotency patterns

Connection	Safe write pattern
Gmail	Before sending, search sent items for an email with the same subject and recipient. Send only if no match.
Google Sheets	Before appending a row, search the sheet for a row with the same identifier column. Append only if not found.
Snowflake	Use MERGE INTO ... WHEN NOT MATCHED THEN INSERT rather than plain INSERT to avoid duplicate rows.
Slack	When posting a follow-up, reply in an existing thread (thread_ts) rather than posting a new top-level message.
HubSpot	Use upsert (search by email or ID, then update or create) rather than creating a new contact each time.
Stripe	Pass the order reference or case ID as the idempotency_key header when creating charges or payment intents.

---

Where to Look When Something Goes Wrong

Jobs List

The Jobs List (Past Jobs in the sidebar) shows the status of every Job across all your assignments. Use the Needs attention quick filter to surface Failed and Needs Input Jobs immediately.

For each failed Job, the evaluation badge shows the severity of any issues found. Click the badge to see a breakdown of what went wrong and why.

Case timeline

For cases processed through Case Queue, open the case detail view to see the full processing history — which assignment handled each stage, when it ran, and what the outcome was. If a case has been retried multiple times, each attempt appears in the timeline.

Reading a failure

Common failure patterns and what they indicate:

Failure pattern	Likely cause	What to fix
Connection error early in the run	A Connection was disconnected or expired	Re-authorize the Connection
Timeout on an external API call	Rate limiting or the external service is slow	Add a retry instruction to your SOP or reduce batch size
Assignment completed but case still Failed	SOP does not explicitly mark the case as completed	Add a completion step to your SOP
Same step fails on every attempt	A hard data error (missing field, wrong format)	Fix the input data or add a validation gate in the SOP
Evaluation shows repeated issues	The assignment is consistently making the same mistake	Refine the SOP for the failing step

---

Related

• Designing Human-in-the-Loop Workflows — Escalation patterns, fallbacks, and approval thresholds
• Case Queue — Case statuses, postpone behavior, and manual retry
• Jobs List — Monitoring status across all assignments