Troubleshooting | Midfleet Docs

Troubleshooting overview

Most failures are scope, route, runtime, or evidence problems. Start with the layer that changed most recently.

Common symptoms

Symptom	Likely cause	First check
403 from API	Wrong actor credential or workspace scope.	Check whether you need bearer JWT or `x-api-key`.
Agent online but not receiving work	Wrong workspace, project, team, role, or queue filter.	List the agent and inspect its inbox in the same workspace.
Terminal opens wrong session	Stale console-router route or reused serve port.	Compare agent ID, serve port, and router inventory.
Workflow stuck after PR merged	Completion evidence not attached to handoff/run.	Open workflow diagnosis and PR comments.
Provider login loops	Expired hosted authorization, a pending credential challenge, or the wrong authorization source.	Check provider readiness and Hub grant/consume audit before respawning.
`provider_model_unsupported`	The requested provider/model is not in the Hub catalog.	Select the exact model identifier from the current catalog; do not guess aliases.
`ProviderModelNotFoundError` after a model change	An older agent runtime stores a stale provider/model identifier.	Stop the agent, clear only that agent's runtime state through its supported recovery procedure, then restart.
Direct start has no provider auth	Participant profile, runtime credential issuance, or runtime auth ownership failed.	Verify the selected profile, one grant and consume, intended audit source, runtime-account ownership, and `0600` mode.
Direct start appears in Relay logs	The launch path was not direct or was routed through Hub/Control spawn.	A direct CLI start should have zero Relay `POST /spawn` requests.
mkdir under /srv/agents fails on Mac/Windows	Workspace onboard still uses the Linux agent_workdir_root.	Export MIDFLEET_AGENT_WORKDIR_ROOT to Networks server workdir_root, then restart the agent.
Agent up but Networks agents count is 0	Hub agent missing inventory runtime_instance_id (rti_*).	Export MIDFLEET_RUNTIME_INSTANCE_ID=rti_… before start; do not source runtime.env (that file id is wrt_*).
Agent was evicted but Hub row remains	Workflow history still references the agent.	Verify process, route, workdir, and runtime auth cleanup; retain the historical Hub row.
Docs page returns homepage	Static route missing from deployed bundle.	Verify docker/site/docs path index.html is deployed.

Multi-host spawn checklist

Networks agent counts only include agents attributed to that host inventory id (rti_*) with effective status online, idle, or busy. Runtime up alone does not start agents.

Host shows spawn ready on Networks (fresh heartbeat + relay ready).
export MIDFLEET_RUNTIME_INSTANCE_ID=rti_<this-host> from Networks/topology.
export MIDFLEET_AGENT_WORKDIR_ROOT=<workdir_root> matching Networks server config on this machine.
--serve-public-base-url equals this server console_public_base_url.
After start: Hub agent has runtime_instance_id=rti_*; Networks agents_running updates within ~30s.

Verify each direct-start boundary in order

Profile and host. Confirm the command ran as the runtime account on the target host with midfleet --profile <profile>.
Inventory id. Confirm MIDFLEET_RUNTIME_INSTANCE_ID is the host rti_* (not bundle wrt_*).
Workdir root. Confirm MIDFLEET_AGENT_WORKDIR_ROOT matches Networks server workdir_root when the workspace default is another OS path.
Catalog and grant. Confirm the provider/model is supported and Hub recorded one provider runtime grant.
Runtime credential. Confirm one issuance event from the intended authorization source.
Runtime auth. Confirm the auth path is owned by the runtime account with restrictive permissions. Never repair it by copying credential-broker state.
Process and route. Confirm the local process, unique serve port, public console health 401, and a round-trip midfleet ask.
Networks count. Confirm Hub runtime_instance_id and Networks agents_running for this host.
Transport proof. Confirm no Relay POST /spawn occurred for a direct launch.

Do not leak secrets while debugging

When sharing evidence, summarize command results and redact tokens, API keys, provider credentials, relay secrets, and service-token values.

bash

midfleet config redact --json
midfleet config doctor
midfleet internal status
midfleet internal inbox --name <agent> --json