HypAware
View as Markdown

Troubleshoot a HypAware installation

Check the installation's status and version:

Terminal
hyp status
hyp version
Output
HypAware · Healthy

  Daemon     Running
  Storage    37 MB · 90-day retention
  Sharing    No organization sync

  Clients                 Status                           Data
  claude                  Recording (otel)                 Local only
  codex                   Not recording                    Local only

  To record codex again: hyp client attach codex

No issues detected.

More details: hyp status --verbose
hypaware 1.41.0
  node:     v26.10.0
  platform: darwin arm64

hyp status reports the background service, clients, storage, and problems, with guidance to help resolve them. Use hyp status --verbose for more detail or hyp status --json for scripts.

No new recordings

Terminal
hyp client status
hyp daemon status
hyp cache status
  1. Check that the intended source is configured. Add it through hyp setup if needed.
  2. Check that the client is attached. Preview and apply its attach using hyp attach CLIENT --dry-run and hyp attach CLIENT.
  3. Check the daemon is running. Use hyp daemon start for an installed service, or follow headless setup when there is no service manager.
  4. Start a new client process and complete a turn so it uses the managed settings. Ollama attach enables recording and prints recipes without editing client files or shell/service OLLAMA_HOST; use its explicit CLI host or SDK constructor.
  5. Check hyp privacy show /absolute/path/to/project for an ignore marking.

For Claude Code, hyp client status claude reports the configured and live listener endpoints, recent telemetry, and transcript activity. If the endpoints differ, run hyp attach claude, restart the daemon, and start a new Claude Code process. Check the Claude Code requirements for minimum versions. For transcript-only clients, allow time for the scheduled recovery sweep.

For Factory Droid, restart Droid after attaching so it loads the native hooks. Check hyp client status droid, then preview recovery with hyp backfill droid --dry-run. The collector must have access to the same Factory home as Droid. See Droid troubleshooting and limits for legacy hook files, missing transcripts, and capture bounds.

Detaching a gateway-routed client restores its own connection settings if a stopped gateway is preventing normal use where that integration manages them. Ollama detach leaves the capture host unchanged. With its collector stopped, launch the CLI at the preserved direct root or construct a new SDK object with that root using direct recovery.

Goose records nothing

Goose capture reads Goose's session database, so hyp status is where its health shows. hyp client status does not list Goose. hyp status prints a Goose line only when something is wrong. The ready messages, the last two rows below, show in hyp status --json as the goose entry's health.message.

Goose status says What to do
cannot read the database at <path> The daemon's user cannot read the file. Run HypAware as the same user as Goose.
failed to open goose sessions.db at <path> The file exists but SQLite could not open it, for example because it is damaged. Check that Goose itself can still open its sessions.
database busy, retrying next poll Goose was writing when HypAware read. It retries on the next poll; if this persists, check for a stuck Goose process.
goose sessions.db has no <table>.<column> column Your Goose version uses a database schema this HypAware release does not support. Nothing is recorded until you upgrade HypAware to a release that supports it.
goose source requires Node >= 22.12 Goose capture reads the database with Node's built-in SQLite. Upgrade the Node.js that runs HypAware, then run hyp daemon restart.
no goose installation detected at <path> HypAware found no database at that path. If you set GOOSE_PATH_ROOT or keep the database elsewhere, set sessions_db as shown on the Goose page, then run hyp daemon restart. Sessions from Goose before 1.10.0 are not supported.
polling <path> every 60000ms, no sessions yet Capture is ready. Finish a Goose turn and wait up to a minute.

New messages arrive within a minute. A change Goose makes to a session it already wrote, such as rewriting earlier messages, arrives with the scheduled import, up to 15 minutes later. Run hyp backfill goose to pick it up at once.

Crush records nothing

Crush capture reads each project's Crush database, so hyp status is where its health shows. hyp client status does not list Crush. hyp status prints a Crush line only when something is wrong. The ready messages, the last four rows below, show in hyp status --json as the crush entry's health.message.

Crush status says What to do
cannot read the crush index at <path> projects.json exists but could not be read or parsed. HypAware keeps reading the projects it last found there. Check that Crush itself still starts; Crush rewrites the index when it opens a project.
<N> crush projects skipped over the 1,024 limit HypAware reads at most 1,024 project databases. Remove projects you no longer use from Crush's index, or list the ones you want in data_dirs.
crush.db has no <table> table (<N> crush databases) or crush.db has no <table>.<column> column (<N> crush databases) Your Crush version uses a database schema this HypAware release does not support. A table named goose_db_version is Crush's own migration record, not a Goose problem. Nothing is recorded from those databases until you upgrade HypAware to a release that supports it.
cannot read <N> crush databases The daemon's user cannot open those databases, or the Node.js that runs HypAware has no built-in SQLite. Or a database is damaged; check that Crush still opens that project. Run HypAware as the same user as Crush, and use Node 22.12 or later; hyp backfill crush prints crush capture requires Node >= 22.12 in the second case. Then run hyp daemon restart.
crush poll failed: <kind> HypAware retries on the next poll. If it persists, run hyp daemon restart.
<N> crush databases busy, retrying next poll Crush was writing when HypAware read. It retries on the next poll; if this persists, check for a stuck Crush process.
<N> sessions appear in more than one crush database A project's .crush directory was copied, so two databases hold the same session. HypAware records each such session from the first database it read. Remove the copy, or mark it with hyp privacy ignore <dir>.
no crush installation detected at <path> HypAware found no index at that path. If you set CRUSH_GLOBAL_DATA or XDG_DATA_HOME, set data_dirs as shown on the Crush page, then run hyp daemon restart.
crush index at <path> lists no projects yet Capture is ready. Start Crush in a project.
polling <N> crush projects every 60000ms, no sessions yet Capture is ready. Finish a Crush turn and wait up to a minute. Older sessions, and the first session in a project HypAware has not read before, arrive with the scheduled import within 15 minutes, or now with hyp backfill crush.
crush ongoing capture disabled (crush.enabled = false) Capture is off in your configuration. Set crush.enabled to true, then run hyp daemon restart.

A status that ends ; <N> listed projects have no crush.db names index entries or data_dirs whose database is gone, such as a deleted project. They are skipped and need no action.

New messages arrive within a minute. A change Crush makes to a message it already finished arrives with the scheduled import, up to 15 minutes later. Run hyp backfill crush to pick it up at once.

Reading a database can leave an empty crush.db-wal and a 32 KiB crush.db-shm in the project's .crush/ directory until Crush next closes it cleanly. These are SQLite's own index files and hold no content.

Mastra Code records nothing

Mastra Code capture reads Mastra Code's mastra.db, so hyp status is where its health shows. hyp client status does not list Mastra Code. hyp status prints a Mastra Code line only when something is wrong. The ready messages, the last three rows below, show in hyp status --json as the mastracode entry's health.message.

Mastra Code status says What to do
cannot read the mastracode database at <path> The daemon's user cannot read the file, or SQLite could not open it, for example because it is damaged. Run HypAware as the same user as Mastra Code, and check that Mastra Code itself still opens its threads.
mastracode database busy, retrying next poll Mastra Code was writing when HypAware read. It retries on the next poll; if this persists, check for a stuck Mastra Code process.
mastra.db has no <table> table or mastra.db has no <table>.<column> column Your Mastra Code version uses a database schema this HypAware release does not support. Nothing is recorded until you upgrade HypAware to a release that supports it.
SQLite <version> cannot read Mastra Code's JSONB metadata (needs 3.45 or newer) The Node.js that runs HypAware has a built-in SQLite too old to read Mastra Code's thread metadata. Upgrade that Node.js, then run hyp daemon restart.
mastracode source requires Node >= 22.13 (node:sqlite builtin not available) Mastra Code capture reads the database with Node's built-in SQLite. Upgrade the Node.js that runs HypAware, then run hyp daemon restart.
MASTRA_DB_URL is set; remote stores are not supported Mastra Code writes to a remote Turso or Postgres store, which HypAware cannot read. Nothing is recorded while MASTRA_DB_URL is set in the daemon's environment.
mastracode poll failed: <kind> HypAware retries on the next poll. If it persists, run hyp daemon restart.
session exclusions could not be read, session exclusions could not be loaded, or Session exclusions could not be loaded; capture is disabled. ... HypAware could not read its saved session exclusions under <HYP_HOME>/hypaware/session-ignores/, so it records nothing rather than risk recording an ignored thread. hyp backfill mastracode fails with Mastra Code session exclusions are unreadable for the same reason. Check that the daemon's user can read that directory and its files, then run hyp daemon restart.
no mastracode installation detected at <path> or no mastra.db at the resolved path HypAware found no database at that path. If you set MASTRA_DB_PATH, MASTRA_APP_DATA_DIR or (on Linux) XDG_DATA_HOME, or moved the database, set db_path as shown on the Mastra Code page, then run hyp daemon restart.
polling mastracode every 60000ms, no threads yet or polling mastracode every 60000ms Capture is ready. The , no threads yet ending drops once the poll has recorded a row since the daemon started, and comes back after a restart until the next new row. With no rows yet, finish a Mastra Code turn and wait up to a minute.
mastracode ongoing capture disabled (mastracode.enabled = false) Capture is off in your configuration. Set mastracode.enabled to true, then run hyp daemon restart.

New messages arrive within a minute. A change Mastra Code makes to a message it already wrote arrives with the scheduled import, up to 15 minutes later. Run hyp backfill mastracode to pick it up at once.

Ollama capture is unconfirmed

Terminal
hyp ollama setup
hyp client status ollama
hyp status --verbose

Check the Direct upstream, service/model inventory and Capture root (live) separately. An executable on PATH does not prove a service or model is ready. Use a model already listed and the supported CLI/SDK recipes in clients. A configured or stale capture root is not a live route. After saved endpoint changes with an unconfirmed restart, run hyp daemon restart and check setup again.

Result or reason Next step
Ready, no traffic Route the next client to the printed live /ollama root, or explicitly run hyp ollama verify --model NAME.
Observed, awaiting persistence Give the processor time to finish; use verify for a fresh committed-pair check.
Persisted stamp The append resolved. Verify or query the particular request to establish current visibility.
Off or historical evidence Successful detach keeps the running host forwarding without capture. Use hyp client attach ollama to resume new requests.
unsupported_shape Use native chat/generate with think=False, no nonempty thinking/tools, and supported fields. Unsupported exchanges are omitted in full.
malformed_stream, missing_terminal, trailing_record The stream must finish with exactly one completed terminal and no later nonblank records. Retry the request explicitly after fixing its producer.
http_error, upstream_unavailable, transport_error Check the direct Ollama service and saved custom upstream with setup. Direct inference and capture are separate outcomes.
capture_limit, processor_unavailable, append_failure Check the ordinary daemon log and processing health. Lost capture is not replayed.
policy_unreadable Restore readable, valid policy/configuration. Verification does not bypass policy.
Unconfirmed attach/detach Follow the command's recovery guidance and retry for live confirmation. A saved setting alone is not a completed barrier.

hyp ollama verify --model NAME --json returns machine-readable status, reason and whether HTTP completed; its prompt/export disclosure goes to stderr. A persistence_timeout means inference completed but storage confirmation timed out. The live collector may finish; settings are unchanged. A later check can report settlement_busy until the original storage operation actually settles. The command never retries inference automatically. If the actual collector stops or its processor is lost, confirmation can be interrupted and remains unconfirmed.

Verification refreshes the shared current gateway spool, including earlier and other-client pending rows, with normal storage/sink hooks. Its cost depends on that backlog. The disposable query worker only reads committed data with normal caller policy and disables legacy query-config migration. A receipt, spool row, HTTP result, old saved stamp or hidden row cannot establish verification success.

Default status carries bounded capture evidence and finite recent failure reasons. These summaries can lag until the next processor status update. Check the ordinary daemon log for a safe failure reason while waiting; it also carries projection, append and lifecycle events. No prompt or response text is needed to diagnose these reasons. Optional dev telemetry can supply structured run/exchange correlation, but is not required for the ordinary status/log path. See diagnostic logs.

When the collector has stopped, use the preserved direct-host CLI command or a new Client(host=...) from direct recovery. This restores inference without recording. Off/stopped historical timestamps do not claim current health. Ollama has no history import or outage replay, and its unknown directory means directory exclusions cannot protect these API requests.

History or search results are missing

An overview covers the dates printed in its heading. A search may be limited by dates, hit count, searchable columns, privacy, or local retention.

  • Confirm the correct HYP_HOME and local versus remote target.
  • Read stderr for withheld-row, freshness, and truncation notices.
  • Use hyp backfill list and a provider-specific backfill dry run to check whether older transcripts are available. Cache refresh is not a backfill.
  • Refresh the affected local dataset with hyp cache refresh ai_gateway_messages when freshness warnings call for it.
  • Use --output ./results.json with --format json for complete query output; this does not lift your query's own LIMIT.

See querying for searches, session drill-downs, and bounded SQL.

History import failed

hyp status shows this problem when a provider's scheduled history import failed on its last run:

codex history import failed on its last run, so some history may be missing. Rows read before the failure may have been imported; the command below shows the cause. The next scheduled run retries, and this clears once a run succeeds.

Rows read before the failure may have been imported. Every scheduled run retries, and the warning clears on its own once a run succeeds. To see the error, and the failing file if there is one:

Terminal
hyp backfill codex --dry-run

If the error is a permission problem, fix it and the next run picks the file up. If the same file keeps failing, the file cannot be imported as it is, for example a Codex session file over its size limits. Moving that file out of the session directory clears the warning on the next run, but HypAware will not import it and Codex will no longer list that session, so keep the file somewhere safe.

A Claude import that cannot read OpenCode's session list stops before it writes anything, so it never records OpenCode's Claude sessions as your own. hyp backfill claude --dry-run shows why. If it says Node 22.13 or newer is needed, upgrade Node; otherwise fix the problem it names, such as the permissions on OpenCode's data directory.

An export or team sync is missing

Terminal
hyp status
hyp sync --dry-run
hyp remote list

Check that a sink exists, its destination is correct, and the rows are eligible to leave. An attended first-sync review hold, local-only folder or client marking, or export retry can explain why local data is not yet in HypAware Cloud. After reviewing the destination plan, use hyp sync to attempt delivery and inspect each sink's result. It can partially succeed and still return failure.

Cloud sync gives each upload chunk a 330-second limit, including authentication, the request, response reading and retry waits. Retries share that limit; a healthy sync with several chunks can take longer overall. Timeout or stop cancels the current Cloud request and waits. Unacknowledged rows remain eligible for a later export while they remain in the local cache. Check connectivity and the destination's failure reason, then retry with hyp sync after fixing it.

An export warning clears only when that same configured destination records a full successful completion strictly after its failure was recorded. Starting an upload, acknowledging some rows, queuing a follow-up or succeeding at another destination does not clear it. The warning can remain after 24 hours; equal timestamps do not establish recovery. Ordinary daemon logs record failure and later recovery separately. See diagnostic logs.

The verbose status recent errors count describes retained history from the last 24 hours, including recovered failures. It is not a count of unsent rows or every failure the installation has ever had. A warning can clear while this count stays nonzero. Deleting failure records or letting them age does not deliver data; retries use the cache and saved export progress. See storage paths for the diagnostic cap and cleanup limits.

Remote sign-in with --no-forward grants query access without setting up sync. hyp leave stops sync but keeps local recordings. See team setup and privacy controls.

Rows skipped by exports before v1.42.0

Before v1.42.0, a hyp command that refreshed the cache while the daemon was also writing could commit rows below the point a destination had already exported to. Those rows stay in the local cache but no scheduled export sends them (issue 2225). v1.42.0 stops new skips once the daemon restarts on it; hyp daemon restart makes sure. Nothing finds the rows already skipped, so recovery is a manual step for each destination. It reaches only rows the local cache still holds: rows that retention has removed are gone from this host.

For HypAware Cloud, hyp sync --history <client> resends one client's retained ai_gateway_messages and leaves normal export progress alone. The server drops rows it already has, so a rerun is safe. Run it for each client that records on this host, and read the preview first: the client must be syncing, and the replay also sends rows it held back while it was local-only. Folders marked local-only stay withheld. Other datasets, such as traces and logs, have no Cloud replay.

A local folder or S3 destination has no replay, and it does not drop rows it already has. To export one dataset again in full, stop the daemon, move that dataset's earlier export files out of the destination, remove the destination's saved position for the dataset, then start the daemon and sync:

Terminal
hyp daemon stop
mv <dir>/<dataset> <kept>/<dataset>
rm -r ~/.hyp/hypaware/plugins/@hypaware/local-fs/sink-instances/<instance>/watermarks/<dataset>
hyp daemon start
hyp sync <instance>

<instance> is the destination's name in hyp sync --dry-run, <dir> is its dir setting, and <kept> is any folder outside <dir>. The ~/.hyp path is the default HYP_HOME; use yours if you set another. For S3, move the objects under <prefix>/<dataset>/ in the destination's bucket to a prefix outside the destination instead, and the saved position is under @hypaware/s3 instead of @hypaware/local-fs.

Move the earlier files before the export, don't skip it. A file is named only by the range of rows it holds, so the re-export can reuse an earlier file's name and replace it, and if retention has since removed some of that file's rows from the local cache, they are lost. After the export, the destination holds every row the local cache still retains, once. The moved files are the only copy of rows that retention has removed, so keep them; a row in both places is the same row, and can be dropped where the data is read, for example by part_id for ai_gateway_messages. Rows held back as local-only stay held back.

A command is missing or rejects an option

Terminal
hyp version
hyp --help
hyp plugin list
hyp query --help

Plugin-owned commands appear only while the owning plugin is active. Entering a known inactive command prints a repair instruction. If a flag in these docs is unavailable, check the installed version's command help and the update procedure.

The CLI and daemon run different versions

Terminal
hyp update
hyp status

hyp update checks for a newer release and restarts an installed daemon after updating. If installation succeeded but restart failed, run hyp daemon restart. A foreground daemon needs to be relaunched by its owner or supervisor. Source checkouts and npx-cache copies do not self-update.

For a missing global binary or broken service definition, follow recovery.

Disk usage or maintenance problems

Terminal
hyp cache status
hyp cache maintain ai_gateway_messages --dry-run

Check retention and storage paths to distinguish the query cache, exported files, retry state, and logs. Cache retention does not reclaim exported copies. Inspect a maintenance dry run before applying it, and use hyp privacy purge only when you intend to delete recorded data.

A query or graph projection refuses for memory

Execution memory is bounded, and work that outgrows its budget refuses instead of returning a partial answer. The refusal names the budget it hit and, where one applies, the variable that raises it. Each is read from the environment of the process doing the work, in MB:

  • HYP_QUERY_MAX_HEAP_MB (default 1024) bounds a query you run. Work that carries its own budget, such as a graph read or projection, ignores it.
  • HYP_GRAPH_PROJECTION_MAX_HEAP_MB (default 3072) bounds a graph projection, which scans whole source tables.

Narrow the work first: add a WHERE or date filter, a LIMIT, or aggregate instead of selecting raw rows. Raise a budget only when the work genuinely needs the memory, and give it a positive value. A value that is blank or not a number is ignored and the default applies. At zero or below the two differ: HYP_QUERY_MAX_HEAP_MB turns the query heap guard off rather than tightening it, while HYP_GRAPH_PROJECTION_MAX_HEAP_MB is ignored and the projection default applies.

For a command you run yourself, set it in the same shell:

Terminal
HYP_GRAPH_PROJECTION_MAX_HEAP_MB=6144 hyp graph project

The installed daemon does not read your shell. It runs scheduled work, such as the projection after a GitHub poll, in a child of the service process, and that child inherits the service environment. hyp daemon install writes no environment block of its own, so set the variable where your service manager builds that environment, then restart the daemon so it starts with the new value.

On macOS (launchd):

Terminal
launchctl setenv HYP_GRAPH_PROJECTION_MAX_HEAP_MB 6144
hyp daemon restart

On Linux (systemd user service):

Terminal
systemctl --user set-environment HYP_GRAPH_PROJECTION_MAX_HEAP_MB=6144
hyp daemon restart

Either setting applies only to processes started afterwards, which is what the restart is for, and both are forgotten at logout or reboot. To keep a value, set it where the login session environment is built: an ~/.config/environment.d/hypaware.conf entry under systemd, or a login-time launchctl setenv under launchd. Remove one with launchctl unsetenv HYP_GRAPH_PROJECTION_MAX_HEAP_MB or systemctl --user unset-environment HYP_GRAPH_PROJECTION_MAX_HEAP_MB, then restart the daemon again. A foreground hyp daemon run takes the environment of the shell that started it, so it needs neither step.

Find diagnostic logs

With the default HYP_HOME, start with the structured log at ~/.hyp/hypaware/logs/daemon.log. Service stdout and stderr are in daemon.out.log and daemon.err.log in the same directory. Processing work has its own ~/.hyp/hypaware/processing/logs/daemon.log. Use the paths reported for your installation if you configured another home.

On Windows, the Scheduled Task does not capture the daemon's output, so there is no daemon.out.log or daemon.err.log; daemon.log is still written. Run hyp daemon status to see whether the daemon is running. To check the task itself, run this in PowerShell or Command Prompt, replacing <SID> with the SID that whoami /user prints (it starts with S-1-5-):

schtasks /Query /TN \hypaware\<SID> /V /FO LIST

Its Status line shows whether the task is running, and Last Result shows how the daemon last exited.

An OTLP exporter refused with 421 Misdirected Request is pointed at a name other than localhost or 127.0.0.1; point it at one of those instead. Product telemetry has separate controls; it is automatic for enrolled organizations, defaults off on standalone installations, and is not the captured conversations or the local diagnostic log.

An OTLP exporter refused with 413 Content Too Large sent a request whose body is larger than 64 MiB after decompression. HypAware keeps none of that request's records, and the exporter does not retry it. The daemon log has no entry for this refusal, so the error appears only on the sending side. Lower the exporter's batch size so each request stays under the limit: SDKs that read the standard variables use OTEL_BLRP_MAX_EXPORT_BATCH_SIZE for logs and OTEL_BSP_MAX_EXPORT_BATCH_SIZE for traces. The limit is fixed and has no configuration setting.

When reporting a problem, include the version, failing command, exit status, relevant status repair lines, and a short log excerpt around the failure. Review excerpts for credentials, paths, and captured content before sharing.

Was this page useful?

↑↓ navigate↵ openesc close