Durable jobs

1. When the user submits, authenticate and authorize in the API.
2. Persist a unique owner-scoped job identity.
3. Launch a private workflow asynchronously and return the saved job ID and status.
4. The workflow owns provider submission, bounded polling with durable waits, retry and backoff, transfer to project storage, and terminal persistence.
5. The browser only observes application state.

A normal job uses one workflow run, however many provider polls it makes. Read the live workflow graph guide for control nodes; do not invent wait or retry syntax.

• Use secret references in HTTP and connector nodes. Keep provider credentials out of the frontend and workflow outputs.
• Persist the external task ID before progressing.
• Copy expiring provider output into durable storage before marking success, and serve owner-authorized stored objects.

Do not add an unconditional cron that calls a backend reconciliation endpoint. At one minute that is 43,200 executions per 30 days with no jobs at all, and the calls can stop an API from sleeping. Provider polling inside an existing durable run needs no scheduler.

Cron suits user-requested time-based work, or a justified recovery sweep with an approved interval, scope and cost. The optional starter reconciler reads existing workflow results. It is not the provider pipeline, and it is no recovery guarantee for a sleeping API.

• Use a database unique constraint on owner plus request key, and compare payloads. A duplicate returns the existing job; a conflicting payload is rejected.
• The gateway’s `runName` is a label, not an idempotency key. Claim the job transactionally in the workflow before paid side effects. Replays read the saved task or result instead of creating another provider task.
• An uncertain provider response is not a safe retry. Use provider idempotency or lookup by a stable reference. Otherwise mark the submission uncertain, keep its identity and expose a recovery path; do not charge again.
• Completion updates and active-slot release must be transactional and replay-safe. Failed, cancelled, exhausted and expired jobs release capacity.