# Troubleshoot a HypAware installation Canonical: https://hypaware.ai/docs/troubleshooting Check the installation's status and version: ```sh hyp status hyp version ``` ```output HypAware · Healthy Daemon Running Storage 37 MB · 90-day retention Sharing No organization sync Clients Status Data claude Recording (otel) Local only codex Not recording Local only To record codex again: hyp client attach codex No issues detected. More details: hyp status --verbose hypaware 1.41.0 node: v26.10.0 platform: darwin arm64 ``` `hyp status` reports the background service, clients, storage, and problems, with guidance to help resolve them. Use `hyp status --verbose` for more detail or `hyp status --json` for scripts. ## No new recordings ```sh hyp client status hyp daemon status hyp cache status ``` 1. Check that the intended source is configured. Add it through `hyp setup` if needed. 2. Check that the client is attached. Preview and apply its attach using `hyp attach CLIENT --dry-run` and `hyp attach CLIENT`. 3. Check the daemon is running. Use `hyp daemon start` for an installed service, or follow [headless setup](https://hypaware.ai/docs/integrations/ci-and-headless/index.md) when there is no service manager. 4. Start a new client process and complete a turn so it uses the managed settings. Ollama attach enables recording and prints recipes without editing client files or shell/service `OLLAMA_HOST`; use its [explicit CLI host or SDK constructor](https://hypaware.ai/docs/integrations/ollama/index.md#route-a-cli-or-sdk-client). 5. Check `hyp privacy show /absolute/path/to/project` for an ignore marking. For Claude Code, `hyp client status claude` reports the configured and live listener endpoints, recent telemetry, and transcript activity. If the endpoints differ, run `hyp attach claude`, restart the daemon, and start a new Claude Code process. Check the [Claude Code requirements](https://hypaware.ai/docs/integrations/claude-code/index.md#requirements) for minimum versions. For transcript-only clients, allow time for the scheduled recovery sweep. For Factory Droid, restart Droid after attaching so it loads the native hooks. Check `hyp client status droid`, then preview recovery with `hyp backfill droid --dry-run`. The collector must have access to the same Factory home as Droid. See [Droid troubleshooting and limits](https://hypaware.ai/docs/integrations/droid/index.md#troubleshooting-and-limits) for legacy hook files, missing transcripts, and capture bounds. Detaching a gateway-routed client restores its own connection settings if a stopped gateway is preventing normal use where that integration manages them. Ollama detach leaves the capture host unchanged. With its collector stopped, launch the CLI at the preserved direct root or construct a new SDK object with that root using [direct recovery](https://hypaware.ai/docs/integrations/ollama/index.md#stop-and-resume-ollama-recording). ## Goose records nothing Goose capture reads Goose's session database, so `hyp status` is where its health shows. `hyp client status` does not list Goose. `hyp status` prints a Goose line only when something is wrong. The ready messages, the last two rows below, show in `hyp status --json` as the `goose` entry's `health.message`. | Goose status says | What to do | | --- | --- | | `cannot read the database at ` | The daemon's user cannot read the file. Run HypAware as the same user as Goose. | | `failed to open goose sessions.db at ` | The file exists but SQLite could not open it, for example because it is damaged. Check that Goose itself can still open its sessions. | | `database busy, retrying next poll` | Goose was writing when HypAware read. It retries on the next poll; if this persists, check for a stuck Goose process. | | `goose sessions.db has no . column` | Your Goose version uses a database schema this HypAware release does not support. Nothing is recorded until you upgrade HypAware to a release that supports it. | | `goose source requires Node >= 22.12` | Goose capture reads the database with Node's built-in SQLite. Upgrade the Node.js that runs HypAware, then run `hyp daemon restart`. | | `no goose installation detected at ` | HypAware found no database at that path. If you set `GOOSE_PATH_ROOT` or keep the database elsewhere, set `sessions_db` as shown on the [Goose page](https://hypaware.ai/docs/integrations/goose/index.md#use-a-different-database-location), then run `hyp daemon restart`. Sessions from Goose before 1.10.0 are not supported. | | `polling every 60000ms, no sessions yet` | Capture is ready. Finish a Goose turn and wait up to a minute. | New messages arrive within a minute. A change Goose makes to a session it already wrote, such as rewriting earlier messages, arrives with the scheduled import, up to 15 minutes later. Run `hyp backfill goose` to pick it up at once. ## Crush records nothing Crush capture reads each project's Crush database, so `hyp status` is where its health shows. `hyp client status` does not list Crush. `hyp status` prints a Crush line only when something is wrong. The ready messages, the last four rows below, show in `hyp status --json` as the `crush` entry's `health.message`. | Crush status says | What to do | | --- | --- | | `cannot read the crush index at ` | `projects.json` exists but could not be read or parsed. HypAware keeps reading the projects it last found there. Check that Crush itself still starts; Crush rewrites the index when it opens a project. | | ` crush projects skipped over the 1,024 limit` | HypAware reads at most 1,024 project databases. Remove projects you no longer use from Crush's index, or list the ones you want in `data_dirs`. | | `crush.db has no
table ( crush databases)` or `crush.db has no
. column ( crush databases)` | Your Crush version uses a database schema this HypAware release does not support. A table named `goose_db_version` is Crush's own migration record, not a Goose problem. Nothing is recorded from those databases until you upgrade HypAware to a release that supports it. | | `cannot read crush databases` | The daemon's user cannot open those databases, or the Node.js that runs HypAware has no built-in SQLite. Or a database is damaged; check that Crush still opens that project. Run HypAware as the same user as Crush, and use Node 22.12 or later; `hyp backfill crush` prints `crush capture requires Node >= 22.12` in the second case. Then run `hyp daemon restart`. | | `crush poll failed: ` | HypAware retries on the next poll. If it persists, run `hyp daemon restart`. | | ` crush databases busy, retrying next poll` | Crush was writing when HypAware read. It retries on the next poll; if this persists, check for a stuck Crush process. | | ` sessions appear in more than one crush database` | A project's `.crush` directory was copied, so two databases hold the same session. HypAware records each such session from the first database it read. Remove the copy, or mark it with `hyp privacy ignore `. | | `no crush installation detected at ` | HypAware found no index at that path. If you set `CRUSH_GLOBAL_DATA` or `XDG_DATA_HOME`, set `data_dirs` as shown on the [Crush page](https://hypaware.ai/docs/integrations/crush/index.md#use-a-different-data-location), then run `hyp daemon restart`. | | `crush index at lists no projects yet` | Capture is ready. Start Crush in a project. | | `polling crush projects every 60000ms, no sessions yet` | Capture is ready. Finish a Crush turn and wait up to a minute. Older sessions, and the first session in a project HypAware has not read before, arrive with the scheduled import within 15 minutes, or now with `hyp backfill crush`. | | `crush ongoing capture disabled (crush.enabled = false)` | Capture is off in your configuration. Set `crush.enabled` to `true`, then run `hyp daemon restart`. | A status that ends `; listed projects have no crush.db` names index entries or `data_dirs` whose database is gone, such as a deleted project. They are skipped and need no action. New messages arrive within a minute. A change Crush makes to a message it already finished arrives with the scheduled import, up to 15 minutes later. Run `hyp backfill crush` to pick it up at once. Reading a database can leave an empty `crush.db-wal` and a 32 KiB `crush.db-shm` in the project's `.crush/` directory until Crush next closes it cleanly. These are SQLite's own index files and hold no content. ## Mastra Code records nothing Mastra Code capture reads Mastra Code's `mastra.db`, so `hyp status` is where its health shows. `hyp client status` does not list Mastra Code. `hyp status` prints a Mastra Code line only when something is wrong. The ready messages, the last three rows below, show in `hyp status --json` as the `mastracode` entry's `health.message`. | Mastra Code status says | What to do | | --- | --- | | `cannot read the mastracode database at ` | The daemon's user cannot read the file, or SQLite could not open it, for example because it is damaged. Run HypAware as the same user as Mastra Code, and check that Mastra Code itself still opens its threads. | | `mastracode database busy, retrying next poll` | Mastra Code was writing when HypAware read. It retries on the next poll; if this persists, check for a stuck Mastra Code process. | | `mastra.db has no
table` or `mastra.db has no
. column` | Your Mastra Code version uses a database schema this HypAware release does not support. Nothing is recorded until you upgrade HypAware to a release that supports it. | | `SQLite cannot read Mastra Code's JSONB metadata (needs 3.45 or newer)` | The Node.js that runs HypAware has a built-in SQLite too old to read Mastra Code's thread metadata. Upgrade that Node.js, then run `hyp daemon restart`. | | `mastracode source requires Node >= 22.13 (node:sqlite builtin not available)` | Mastra Code capture reads the database with Node's built-in SQLite. Upgrade the Node.js that runs HypAware, then run `hyp daemon restart`. | | `MASTRA_DB_URL is set; remote stores are not supported` | Mastra Code writes to a remote Turso or Postgres store, which HypAware cannot read. Nothing is recorded while `MASTRA_DB_URL` is set in the daemon's environment. | | `mastracode poll failed: ` | HypAware retries on the next poll. If it persists, run `hyp daemon restart`. | | `session exclusions could not be read`, `session exclusions could not be loaded`, or `Session exclusions could not be loaded; capture is disabled. ...` | HypAware could not read its saved session exclusions under `/hypaware/session-ignores/`, so it records nothing rather than risk recording an ignored thread. `hyp backfill mastracode` fails with `Mastra Code session exclusions are unreadable` for the same reason. Check that the daemon's user can read that directory and its files, then run `hyp daemon restart`. | | `no mastracode installation detected at ` or `no mastra.db at the resolved path` | HypAware found no database at that path. If you set `MASTRA_DB_PATH`, `MASTRA_APP_DATA_DIR` or (on Linux) `XDG_DATA_HOME`, or moved the database, set `db_path` as shown on the [Mastra Code page](https://hypaware.ai/docs/integrations/mastracode/index.md#use-a-different-database-location), then run `hyp daemon restart`. | | `polling mastracode every 60000ms, no threads yet` or `polling mastracode every 60000ms` | Capture is ready. The `, no threads yet` ending drops once the poll has recorded a row since the daemon started, and comes back after a restart until the next new row. With no rows yet, finish a Mastra Code turn and wait up to a minute. | | `mastracode ongoing capture disabled (mastracode.enabled = false)` | Capture is off in your configuration. Set `mastracode.enabled` to `true`, then run `hyp daemon restart`. | New messages arrive within a minute. A change Mastra Code makes to a message it already wrote arrives with the scheduled import, up to 15 minutes later. Run `hyp backfill mastracode` to pick it up at once. ## Ollama capture is unconfirmed ```sh hyp ollama setup hyp client status ollama hyp status --verbose ``` Check the **Direct upstream**, service/model inventory and **Capture root (live)** separately. An executable on PATH does not prove a service or model is ready. Use a model already listed and the supported CLI/SDK recipes in [clients](https://hypaware.ai/docs/integrations/ollama/index.md#route-a-cli-or-sdk-client). A configured or stale capture root is not a live route. After saved endpoint changes with an unconfirmed restart, run `hyp daemon restart` and check setup again. | Result or reason | Next step | | --- | --- | | Ready, no traffic | Route the next client to the printed live `/ollama` root, or explicitly run `hyp ollama verify --model NAME`. | | Observed, awaiting persistence | Give the processor time to finish; use verify for a fresh committed-pair check. | | Persisted stamp | The append resolved. Verify or query the particular request to establish current visibility. | | Off or historical evidence | Successful detach keeps the running host forwarding without capture. Use `hyp client attach ollama` to resume new requests. | | `unsupported_shape` | Use native chat/generate with `think=False`, no nonempty thinking/tools, and supported fields. Unsupported exchanges are omitted in full. | | `malformed_stream`, `missing_terminal`, `trailing_record` | The stream must finish with exactly one completed terminal and no later nonblank records. Retry the request explicitly after fixing its producer. | | `http_error`, `upstream_unavailable`, `transport_error` | Check the direct Ollama service and saved custom upstream with setup. Direct inference and capture are separate outcomes. | | `capture_limit`, `processor_unavailable`, `append_failure` | Check the ordinary daemon log and processing health. Lost capture is not replayed. | | `policy_unreadable` | Restore readable, valid policy/configuration. Verification does not bypass policy. | | Unconfirmed attach/detach | Follow the command's recovery guidance and retry for live confirmation. A saved setting alone is not a completed barrier. | `hyp ollama verify --model NAME --json` returns machine-readable status, reason and whether HTTP completed; its prompt/export disclosure goes to stderr. A `persistence_timeout` means inference completed but storage confirmation timed out. The live collector may finish; settings are unchanged. A later check can report `settlement_busy` until the original storage operation actually settles. The command never retries inference automatically. If the actual collector stops or its processor is lost, confirmation can be interrupted and remains unconfirmed. Verification refreshes the shared current gateway spool, including earlier and other-client pending rows, with normal storage/sink hooks. Its cost depends on that backlog. The disposable query worker only reads committed data with normal caller policy and disables legacy query-config migration. A receipt, spool row, HTTP result, old saved stamp or hidden row cannot establish verification success. Default status carries bounded capture evidence and finite recent failure reasons. These summaries can lag until the next processor status update. Check the ordinary daemon log for a safe failure reason while waiting; it also carries projection, append and lifecycle events. No prompt or response text is needed to diagnose these reasons. Optional dev telemetry can supply structured run/exchange correlation, but is not required for the ordinary status/log path. See [diagnostic logs](#find-diagnostic-logs). When the collector has stopped, use the preserved direct-host CLI command or a new `Client(host=...)` from [direct recovery](https://hypaware.ai/docs/integrations/ollama/index.md#stop-and-resume-ollama-recording). This restores inference without recording. Off/stopped historical timestamps do not claim current health. Ollama has no history import or outage replay, and its unknown directory means directory exclusions cannot protect these API requests. ## History or search results are missing An overview covers the dates printed in its heading. A search may be limited by dates, hit count, searchable columns, privacy, or local retention. - Confirm the correct `HYP_HOME` and local versus remote target. - Read stderr for withheld-row, freshness, and truncation notices. - Use `hyp backfill list` and a provider-specific backfill dry run to check whether older transcripts are available. Cache refresh is not a backfill. - Refresh the affected local dataset with `hyp cache refresh ai_gateway_messages` when freshness warnings call for it. - Use `--output ./results.json` with `--format json` for complete query output; this does not lift your query's own `LIMIT`. See [querying](https://hypaware.ai/docs/querying/index.md) for searches, session drill-downs, and bounded SQL. ## History import failed `hyp status` shows this problem when a provider's scheduled history import failed on its last run: ```output codex history import failed on its last run, so some history may be missing. Rows read before the failure may have been imported; the command below shows the cause. The next scheduled run retries, and this clears once a run succeeds. ``` Rows read before the failure may have been imported. Every scheduled run retries, and the warning clears on its own once a run succeeds. To see the error, and the failing file if there is one: ```sh hyp backfill codex --dry-run ``` If the error is a permission problem, fix it and the next run picks the file up. If the same file keeps failing, the file cannot be imported as it is, for example a Codex session file over its [size limits](https://hypaware.ai/docs/integrations/codex/index.md#large-session-files). Moving that file out of the session directory clears the warning on the next run, but HypAware will not import it and Codex will no longer list that session, so keep the file somewhere safe. A Claude import that cannot read OpenCode's session list stops before it writes anything, so it never records OpenCode's Claude sessions as your own. `hyp backfill claude --dry-run` shows why. If it says Node 22.13 or newer is needed, upgrade Node; otherwise fix the problem it names, such as the permissions on OpenCode's data directory. ## An export or team sync is missing ```sh hyp status hyp sync --dry-run hyp remote list ``` Check that a sink exists, its destination is correct, and the rows are eligible to leave. An attended first-sync review hold, local-only folder or client marking, or export retry can explain why local data is not yet in HypAware Cloud. After reviewing the destination plan, use `hyp sync` to attempt delivery and inspect each sink's result. It can partially succeed and still return failure. Cloud sync gives each upload chunk a 330-second limit, including authentication, the request, response reading and retry waits. Retries share that limit; a healthy sync with several chunks can take longer overall. Timeout or stop cancels the current Cloud request and waits. Unacknowledged rows remain eligible for a later export while they remain in the local cache. Check connectivity and the destination's failure reason, then retry with `hyp sync` after fixing it. An export warning clears only when that same configured destination records a full successful completion strictly after its failure was recorded. Starting an upload, acknowledging some rows, queuing a follow-up or succeeding at another destination does not clear it. The warning can remain after 24 hours; equal timestamps do not establish recovery. Ordinary daemon logs record failure and later recovery separately. See [diagnostic logs](#find-diagnostic-logs). The verbose status `recent errors` count describes retained history from the last 24 hours, including recovered failures. It is not a count of unsent rows or every failure the installation has ever had. A warning can clear while this count stays nonzero. Deleting failure records or letting them age does not deliver data; retries use the cache and saved export progress. See [storage paths](https://hypaware.ai/docs/configuration/index.md#know-where-data-lives) for the diagnostic cap and cleanup limits. Remote sign-in with `--no-forward` grants query access without setting up sync. `hyp leave` stops sync but keeps local recordings. See [team setup](https://hypaware.ai/docs/team-setup/index.md) and [privacy controls](https://hypaware.ai/docs/privacy/index.md). ### Rows skipped by exports before v1.42.0 Before v1.42.0, a `hyp` command that refreshed the cache while the daemon was also writing could commit rows below the point a destination had already exported to. Those rows stay in the local cache but no scheduled export sends them ([issue 2225](https://github.com/hyparam/hypaware/issues/2225)). v1.42.0 stops new skips once the daemon restarts on it; `hyp daemon restart` makes sure. Nothing finds the rows already skipped, so recovery is a manual step for each destination. It reaches only rows the local cache still holds: rows that retention has removed are gone from this host. For HypAware Cloud, `hyp sync --history ` resends one client's retained `ai_gateway_messages` and leaves normal export progress alone. The server drops rows it already has, so a rerun is safe. Run it for each client that records on this host, and read the preview first: the client must be syncing, and the replay also sends rows it held back while it was local-only. Folders marked local-only stay withheld. Other datasets, such as traces and logs, have no Cloud replay. A local folder or S3 destination has no replay, and it does not drop rows it already has. To export one dataset again in full, stop the daemon, move that dataset's earlier export files out of the destination, remove the destination's saved position for the dataset, then start the daemon and sync: ```sh hyp daemon stop mv / / rm -r ~/.hyp/hypaware/plugins/@hypaware/local-fs/sink-instances//watermarks/ hyp daemon start hyp sync ``` `` is the destination's name in `hyp sync --dry-run`, `` is its `dir` setting, and `` is any folder outside ``. The `~/.hyp` path is the default `HYP_HOME`; use yours if you set another. For S3, move the objects under `//` in the destination's bucket to a prefix outside the destination instead, and the saved position is under `@hypaware/s3` instead of `@hypaware/local-fs`. Move the earlier files before the export, don't skip it. A file is named only by the range of rows it holds, so the re-export can reuse an earlier file's name and replace it, and if retention has since removed some of that file's rows from the local cache, they are lost. After the export, the destination holds every row the local cache still retains, once. The moved files are the only copy of rows that retention has removed, so keep them; a row in both places is the same row, and can be dropped where the data is read, for example by `part_id` for `ai_gateway_messages`. Rows held back as local-only stay held back. ## A command is missing or rejects an option ```sh hyp version hyp --help hyp plugin list hyp query --help ``` Plugin-owned commands appear only while the owning plugin is active. Entering a known inactive command prints a repair instruction. If a flag in these docs is unavailable, check the installed version's command help and the [update procedure](https://hypaware.ai/docs/cli/index.md#update-hypaware). ## The CLI and daemon run different versions ```sh hyp update hyp status ``` `hyp update` checks for a newer release and restarts an installed daemon after updating. If installation succeeded but restart failed, run `hyp daemon restart`. A foreground daemon needs to be relaunched by its owner or supervisor. Source checkouts and npx-cache copies do not self-update. For a missing global binary or broken service definition, follow [recovery](https://hypaware.ai/docs/cli/index.md#reinstall-or-recover-the-current-version). ## Disk usage or maintenance problems ```sh hyp cache status hyp cache maintain ai_gateway_messages --dry-run ``` Check [retention and storage paths](https://hypaware.ai/docs/configuration/index.md) to distinguish the query cache, exported files, retry state, and logs. Cache retention does not reclaim exported copies. Inspect a maintenance dry run before applying it, and use `hyp privacy purge` only when you intend to delete recorded data. ## A query or graph projection refuses for memory Execution memory is bounded, and work that outgrows its budget refuses instead of returning a partial answer. The refusal names the budget it hit and, where one applies, the variable that raises it. Each is read from the environment of the process doing the work, in MB: - `HYP_QUERY_MAX_HEAP_MB` (default 1024) bounds a query you run. Work that carries its own budget, such as a graph read or projection, ignores it. - `HYP_GRAPH_PROJECTION_MAX_HEAP_MB` (default 3072) bounds a graph projection, which scans whole source tables. Narrow the work first: add a `WHERE` or date filter, a `LIMIT`, or aggregate instead of selecting raw rows. Raise a budget only when the work genuinely needs the memory, and give it a positive value. A value that is blank or not a number is ignored and the default applies. At zero or below the two differ: `HYP_QUERY_MAX_HEAP_MB` turns the query heap guard off rather than tightening it, while `HYP_GRAPH_PROJECTION_MAX_HEAP_MB` is ignored and the projection default applies. For a command you run yourself, set it in the same shell: ```sh HYP_GRAPH_PROJECTION_MAX_HEAP_MB=6144 hyp graph project ``` The installed daemon does not read your shell. It runs scheduled work, such as the projection after a GitHub poll, in a child of the service process, and that child inherits the service environment. `hyp daemon install` writes no environment block of its own, so set the variable where your service manager builds that environment, then restart the daemon so it starts with the new value. On macOS (launchd): ```sh launchctl setenv HYP_GRAPH_PROJECTION_MAX_HEAP_MB 6144 hyp daemon restart ``` On Linux (systemd user service): ```sh systemctl --user set-environment HYP_GRAPH_PROJECTION_MAX_HEAP_MB=6144 hyp daemon restart ``` Either setting applies only to processes started afterwards, which is what the restart is for, and both are forgotten at logout or reboot. To keep a value, set it where the login session environment is built: an `~/.config/environment.d/hypaware.conf` entry under systemd, or a login-time `launchctl setenv` under launchd. Remove one with `launchctl unsetenv HYP_GRAPH_PROJECTION_MAX_HEAP_MB` or `systemctl --user unset-environment HYP_GRAPH_PROJECTION_MAX_HEAP_MB`, then restart the daemon again. A foreground `hyp daemon run` takes the environment of the shell that started it, so it needs neither step. ## Find diagnostic logs With the default `HYP_HOME`, start with the structured log at `~/.hyp/hypaware/logs/daemon.log`. Service stdout and stderr are in `daemon.out.log` and `daemon.err.log` in the same directory. Processing work has its own `~/.hyp/hypaware/processing/logs/daemon.log`. Use the paths reported for your installation if you configured another home. On Windows, the Scheduled Task does not capture the daemon's output, so there is no `daemon.out.log` or `daemon.err.log`; `daemon.log` is still written. Run `hyp daemon status` to see whether the daemon is running. To check the task itself, run this in PowerShell or Command Prompt, replacing `` with the SID that `whoami /user` prints (it starts with `S-1-5-`): ```powershell schtasks /Query /TN \hypaware\ /V /FO LIST ``` Its `Status` line shows whether the task is running, and `Last Result` shows how the daemon last exited. An OTLP exporter refused with `421 Misdirected Request` is pointed at a name other than `localhost` or `127.0.0.1`; point it at one of those instead. [Product telemetry](https://hypaware.ai/docs/privacy/index.md#product-telemetry) has separate controls; it is automatic for enrolled organizations, defaults off on standalone installations, and is not the captured conversations or the local diagnostic log. An OTLP exporter refused with `413 Content Too Large` sent a request whose body is larger than 64 MiB after decompression. HypAware keeps none of that request's records, and the exporter does not retry it. The daemon log has no entry for this refusal, so the error appears only on the sending side. Lower the exporter's batch size so each request stays under the limit: SDKs that read the standard variables use `OTEL_BLRP_MAX_EXPORT_BATCH_SIZE` for logs and `OTEL_BSP_MAX_EXPORT_BATCH_SIZE` for traces. The limit is fixed and has no configuration setting. When reporting a problem, include the version, failing command, exit status, relevant status repair lines, and a short log excerpt around the failure. Review excerpts for credentials, paths, and captured content before sharing.