> ## Documentation Index
> Fetch the complete documentation index at: https://docs.agenticenv.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Durable Agent (Temporal)
> Test Temporal durability with worker crashes, process restarts, and mid-run recovery scenarios
The **`durable_agent/temporal`** example is an interactive **Temporal lab**, not a minimal hello-world. It runs the agent client and Temporal worker in **separate processes**, streams events over the workflow's event stream, and includes a REPL so you can deliberately break things — kill workers, crash processes, miss approvals — and observe how Temporal and the SDK behave.
Use it after [Agent Worker](/examples/agent-worker) when you need to **feel** durability guarantees. For a zero-infrastructure single-process lab, see [Durable Agent (Local)](/examples/durable-agent-local); for the single-process Restate lab, see [Durable Agent (Restate)](/examples/durable-agent-restate).
Source: [`examples/durable_agent/temporal/`](https://github.com/agenticenv/agent-sdk-go/tree/main/examples/durable_agent/temporal)
For hands-on step-by-step exercises, follow **[`examples/durable_agent/temporal/README.md`](https://github.com/agenticenv/agent-sdk-go/blob/main/examples/durable_agent/temporal/README.md)**.
## Architecture
| Process | Entry | SDK pattern |
| - | - | - |
| **Worker** | `durable_agent/temporal/worker` | `NewAgentWorker` — polls the Temporal task queue and executes workflows and activities |
| **Agent (client)** | `durable_agent/temporal/agent` | `NewAgent` with `DisableLocalWorker` — starts runs, consumes stream |
Both processes share the same agent options from `durable_agent/temporal/opts/opts.go` so the SDK **fingerprint** matches — name, LLM client, tools, approval policy, etc.
The agent uses **`Stream`**. The REPL prints event types as they arrive:
* `TEXT_MESSAGE_CONTENT` — token deltas
* Tool lifecycle events
* `CUSTOM` — tool and delegation **approvals** (respond with `AgentStream.Approve` in the sample)
* `RUN_FINISHED` / errors
Events are delivered by the runtime as a durable ordered stream. Each event carries a monotonic offset.
The example persists `runID` and the last event offset to `/tmp/durable_agent_temporal_runstate.json` on every event. Kill the agent mid-stream with `pkill -SIGKILL` and restart — the REPL detects the saved state, asks whether to reconnect, and resumes via `GetAgentStream` + `WithOffset` from the exact offset. See the [README](https://github.com/agenticenv/agent-sdk-go/blob/main/examples/durable_agent/temporal/README.md) and [Durable Execution](/advanced/durable-execution) for the full protocol.
## Before you start
Run all commands from the **`examples/`** directory. Complete [Configuration](/examples/configuration) first (`.env`, API key, env load order).
Add to `examples/.env` (defaults match local Docker compose; override for Temporal Cloud or a custom cluster):
```bash theme={null}
AGENT_RUNTIME=temporal
TEMPORAL_HOST=127.0.0.1
TEMPORAL_PORT=7233
TEMPORAL_NAMESPACE=default
TEMPORAL_TASKQUEUE=agent-sdk-go-durable-agent
```
Both `agent` and `worker` read the same values. Override any `TEMPORAL_*` variable for your environment.
Start Temporal:
```bash theme={null}
task infra:temporal:up && task infra:temporal:wait
```
With the local Docker dev server, open [http://localhost:8233](http://localhost:8233) (**Temporal Web UI**) while the worker and agent run. Inspect workflows and activities in namespace `default`. Helpful when stepping through the scenarios below.
Stop Temporal when finished:
```bash theme={null}
task infra:temporal:down
```
| Rule | Why |
| - | - |
| **Start the worker before typing a prompt** | Interactive mode checks for available workers at **prompt submission**, not at agent startup |
| **Worker poller check: \~15 seconds** | `AgentModeInteractive` — fails fast if no worker polls the queue before the stream starts |
| **Run timeout in this example: 3 minutes** | Set in `durable_agent/temporal/opts/opts.go` via `WithTimeout(3 * time.Minute)` |
| **Autonomous mode skips worker pre-check** | `AgentModeAutonomous` queues in Temporal until a worker appears — different failure mode on misconfig |
Minimal smoke test (from `examples/`):
**Terminal 1 — worker:**
```bash theme={null}
go run ./durable_agent/temporal/worker
```
**Terminal 2 — agent REPL:**
```bash theme={null}
go run ./durable_agent/temporal/agent
```
One-shot (no REPL):
```bash theme={null}
go run ./durable_agent/temporal/agent "Hello from remote agent!"`
```
Type `exit`, `quit`, or `bye` to leave the REPL. For hands-on scenarios, see the [README](https://github.com/agenticenv/agent-sdk-go/blob/main/examples/durable_agent/temporal/README.md).
## Durability scenarios
Each scenario below lists **what to try** and **what you should learn**. Step-by-step instructions: [README](https://github.com/agenticenv/agent-sdk-go/blob/main/examples/durable_agent/temporal/README.md).
### 1. Baseline — worker first
Start worker, then agent, send `Hello from remote agent!`.
**Learn:** End-to-end remote path works — stream completes, REPL ready for the next turn.
### 2. Agent without worker (intentional timeout)
Start agent **with no worker**, send a prompt. With no pollers on the task queue, the SDK checks Temporal before starting the stream (\~**15 seconds**). Typical error:
```text theme={null}
[error] failed to start stream: no workers available on task queue agent-sdk-go-durable-agent_remote-worker
```
If a worker **was** running recently, Temporal may still list stale pollers briefly — the stream can start, the workflow queues, and you hit the run timeout instead (**3 minutes**): `[error] request timed out (approval expired or deadline exceeded)`.
Then start the worker and **resend** the same prompt — run succeeds.
**Learn:** Interactive mode **fails fast** when no worker is polling — better than hanging forever. After the worker is up, resend the prompt and the run succeeds.
### 3. Kill worker between runs
**3a — Graceful stop:** Complete a run, Ctrl+C worker, restart worker, send another prompt.
**3b — Crash:** Complete a run, kill the worker (`pkill -f 'go run ./durable_agent/temporal/worker|go-build/.*/worker'`), restart worker, send another prompt.
**Learn:** Planned or crash worker shutdown **between runs** does not lose completed work. Temporal history already recorded finished activities; the restarted worker does not re-execute them.
### 4. Kill worker during an LLM call (mid-stream)
Send a long prompt (e.g. 7-day Japan travel plan). While tokens stream, stop the worker — use `pkill` mid-stream (graceful Ctrl+C may wait for the in-flight activity to finish; see README scenario 4).
**4a — Worker stays down:** Stream pauses silently until the run timeout (**3 minutes** in this example), then `[error] request timed out (approval expired or deadline exceeded)`. Restart worker to resume normal operation.
**4b — Worker restarts before timeout:** Stop worker mid-stream, restart within **3 minutes**. Temporal reschedules the in-flight LLM **activity**; stream **resumes on the same agent process** without resending the prompt.
**Learn:** Core durability — the run continues when the client still lives. **Caveat:** on worker restart the LLM call retries from the start; the SDK discards stale tokens so subscribers receive clean delivery, but there may be a brief gap before new tokens arrive. Final conversation content is one complete result.
### 5. Agent restart or crash
**5a — Graceful agent exit:** Finish a run, type `bye`, restart agent, new prompt works against same worker.
**5b — Agent crash between runs:** Kill the agent (`pkill -f 'go run ./durable_agent/temporal/agent|go-build/.*/agent'`) after a completed run; worker keeps polling; new agent process connects and runs immediately.
**5c — Agent crash mid-LLM call:** Kill the agent while tokens stream (`pkill -f 'go run ./durable_agent/temporal/agent|go-build/.*/agent'`); **worker completes the run in Temporal** even though the user saw nothing. Restart agent — follow-up prompt has **no conversation memory** in this example (no `WithConversation` wired).
**Learn:** Worker survives agent death; work can finish server-side. **This example does not persist chat history** — a follow-up like "What was the first destination?" gets no context. For production UIs where users expect continuity after reconnect, see [Agent Chat](/reference-apps/agent-chat) (Postgres + SSE + durable workflows).
### 6. Two workers, one queue
Run two `durable_agent/temporal/worker` processes on the same task queue. Send prompts; stop one worker after a reply or during a long prompt while the other keeps polling; send another prompt.
**Learn:** Temporal load-balances across workers. Losing one worker mid-session does not drop an in-flight run if another worker polls the same queue.
### 7. Task queue mismatch
Point the **worker** at a different queue via env while the agent keeps the default from `.env` (see README — stop workers on the default queue first):
```bash theme={null}
TEMPORAL_TASKQUEUE=wrong-queue-durable-agent go run ./durable_agent/temporal/worker
```
Send a prompt from the agent. Outcome depends on timing (same as scenario 2):
* **Typical:** `[error] failed to start stream: no workers available on task queue agent-sdk-go-durable-agent_remote-worker` (\~15s poller check).
* **Stale pollers on the agent queue:** stream may start, then `[error] request timed out (approval expired or deadline exceeded)` after **3 minutes**.
**Learn:** Misconfiguration surfaces clearly — not silent corruption. Restart both processes with matching `TEMPORAL_TASKQUEUE` (or remove the override).
**Tip:** Under `AgentModeAutonomous`, the immediate worker check is skipped — mismatch may queue until timeout instead of failing fast. Always align task queue (and shared `opts`) between agent and worker before deploy.
## Approvals in this example
`agent/main.go` handles **`CUSTOM`** approval events — parse with `ParseCustomEventApproval` / `ParseCustomEventDelegation` and call `AgentStream.Approve` with the token (same pattern as [Approvals](/features/approvals)). Shared opts in `opts/opts.go` do not register tools by default, so simple prompts usually skip this path.
If you add tools that require approval and ignore prompts, tools are skipped with a clear message rather than hanging indefinitely.
## What this example does not cover
| Topic | Where to go |
| - | - |
| Minimal split client/worker | [Agent Worker](/examples/agent-worker) |
| Conversation across restarts | [Agent Chat](/reference-apps/agent-chat) or [Conversation](/examples/conversation) with Redis |
| Production checklist | [Readiness](/production/readiness) |
## Learn more
Server-side durability and client-side stream recovery
Production split-process pattern
Single-process GetAgentStream demo
Architecture and event delivery model
Zero-infrastructure crash/reconnect lab, durable by default
This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.