ka2a documentation

Error reference

Developer preview. Not yet production ready. This page is rendered from docs/error-reference.md of the ka2a repository at revision 96fb45e5e5f6693e779837998c3aa9581d6a37f2. It describes the behavior of that revision.

Contents

This reference lists every error code, failure class, hint and reason token that ka2a reports. Each token is stable: a later revision does not change its meaning. A later revision can add tokens, so treat an unknown token as a failure that needs attention.

A test (TestErrorReferenceListsEveryCodeAndReason, internal/cli) keeps this reference complete: it reads the constants and the reason literals from the source and fails when a token is missing from its section.

"Retry" tells what a caller does with the same request:

yes: try the same call again later, with a backoff.
same key: submit again with the same operation key (SubmitOptions.OperationID). The node returns the original operation and sends no new request. This is always safe.
no: the same call fails again. Change the input, the configuration or the state first.

Library error codes

ka2a.CodeOf(err) returns one of these codes. errors.Is(err, ka2a.ErrHeld) and the other sentinel errors match them too. The command line prints the same code in its message and in the code member of ka2a.error/1.

Code	Meaning	Retry	Action
`invalid_argument`	The profile refuses an argument, for example an operation ID that is not a lowercase UUIDv4.	no	Fix the call.
`invalid_config`	A configuration, catalog, key file or mailbox topic cannot be used.	no	Fix the configuration. Run `ka2a doctor` and `ka2a catalog validate`.
`unsupported_operation`	The operation is outside the declared profile.	no	Use an operation of the profile.
`unsupported_feature`	An optional A2A feature is outside the profile, for example push notifications.	no	Remove the feature from the request.
`identity_conflict`	The operation ID names another logical request.	no	Use a new operation ID for a new request. See identity_conflict.
`unauthorized`	The trusted catalog has no grant for this source, target and operation.	no	Add the grant to the catalog and restart the nodes.
`unknown_peer`	The trusted catalog does not list the peer.	no	Add the peer to the catalog and restart the node.
`storage_pressure`	New work would exceed the retained-state budget. Nothing is stored.	yes	Resolve held work, let retention run or raise `Limits.RetainedBytes`. See Storage pressure.
`storage_unavailable`	An I/O, lock or capacity failure of the local store. Nothing is admitted.	yes	Retry with a backoff. If it stays, check the disk and the file system of the state directory.
`store_busy`	Another process owns the state directory.	yes, after the owner stops	Stop the other owner, or use the running node. The command line exits with 5.
`state_invalid`	The state directory cannot be used safely: unsafe permissions, another endpoint, a foreign schema, or lost or damaged state.	no	Fix the permissions or the path. Restore from a backup when the state is lost.
`quarantined`	The store waits for operator reconciliation after a restore. No handler starts.	no	Reconcile and run `ka2a quarantine release`. See Quarantine after a restore.
`not_ready`	The call came before the node finished its start.	yes	Wait for `Node.Ready` or the `ready` event.
`closed`	The call came after `Close`.	no	Open a new node.
`already_running`	A second `Run` call on one node.	no	Call `Run` once per node.
`unknown_outcome`	A wait or a call ended before its outcome was known. This is an observation, not a remote failure: the admitted work continues.	same key	Wait again, or read the operation with `Client.Lookup`. Do not send a new request with a new key. See Unknown outcomes and exit code 6.
`held`	The node holds the exchange because its outcome is unknown and needs attention.	no	An operator decides with `ka2a held release` or `ka2a held abandon`. See Held work.
`abandoned`	An operator abandoned the exchange. Its outcome stays unknown.	no	Check the remote effect out of band before a new request.
`expired`	A record expired, or retention purged the identity. An expired identity is never refreshed.	no	Send a new request with a new operation ID.
`not_found`	The operation or task is unknown.	no	Check the ID. Retention can have purged it.
`publish_rejected`	The broker refused the request permanently, for example a missing Write ACL or a record that is too large. The exchange is held.	no	Fix the cause (see Publish reasons), then decide on the held exchange.
`invalid_transition`	The state change is not permitted, for example a change of a terminal task.	no	Read the current state first.
`internal`	A defect or an unclassified failure.	yes, once	Report it with the log line. The detail never holds a body or a key.

Command-line error codes

The command line prints a library code when the failure has one. Otherwise it prints one of these codes, or a code of the store, snapshot and catalog.

Code	Meaning	Exit
`usage`	Unknown command, flag or argument.	2
`problems`	The checks of the command found problems. The report names them.	3
`not_available`	The command is not available in this build.	4
`store_busy`	A running node owns the state directory.	5
`canceled`	The command was interrupted or its deadline passed.	1
`failed`	An unclassified failure. The message explains it.	1

Request outcomes

ka2a send, task get, tasks list and task cancel print an outcome and exit with its code. The request stays admitted after exit code 6.

Outcome	Meaning	Exit
`result`	The peer answered with a result.	0
`error`	The peer answered with an A2A error.	1
`pending`	No response arrived within `--wait`. The request can still complete.	6
`held`	The exchange is held: its outcome is unknown and needs an operator.	6
`expired`	The request expired before a response.	1
`publish_rejected`	The broker refused the request permanently.	1
`abandoned`	An operator abandoned the exchange.	1

Exit codes

Code	Meaning
0	Success.
1	The command failed. The message names the error code.
2	Usage error: unknown command, flag or argument.
3	The checks found problems.
4	The command is not available in this build.
5	A running node owns the state directory.
6	A request is admitted, but its outcome is not known yet. It is not a failure.

Store, snapshot and catalog error codes

The command line can print these codes of the local store, of pkg/observe (snapshots) and of the catalog and key checks. pkg/ka2a maps them to a library code.

Code	Meaning	Action
`invalid_input`	A malformed or out-of-bounds argument.	Fix the argument.
`store_busy`	Another owner holds the state directory.	Stop the other owner.
`unsafe_storage`	Unsafe permissions, links or file system type of the state directory.	Give the directory to the service user with mode 0700, without links.
`pragma_mismatch`	A SQLite connection setting did not take effect.	Report it. The file system can be unsupported.
`schema_too_new`	The store comes from a newer ka2a binary.	Use the newer binary. See the upgrade notes.
`schema_too_old`	Only the owner can migrate this store.	Start the owner (`ka2a run`) once with the new binary.
`schema_mismatch`	The database is not a ka2a store.	Check the path.
`identity_mismatch`	The store belongs to another endpoint.	Check `--state-dir`, `--domain` and `--endpoint`.
`state_missing`	The endpoint state is lost or incomplete, or the directory has no store.	Run `ka2a init`, or restore from a backup.
`quarantined`	The store waits for operator reconciliation.	See Quarantine after a restore.
`recovery_pending`	Dispatch before the restart recovery ran.	Internal ordering. Report it.
`identity_conflict`	An identity is reused with other content.	See identity_conflict.
`correlation_mismatch`	A response for another peer or parent.	The response is rejected. Check the peer.
`response_conflict`	A second, different final response.	The first response stays. Check the peer.
`expired`	An expired record or a purged identity.	Use a new operation.
`storage_pressure`	New admission would exceed the budget.	See Storage pressure.
`storage_full`	SQLite or the file system is full.	Free disk space. The node retries.
`storage_unavailable`	An I/O, lock or other storage failure.	Retry. Check the disk.
`not_found`	An absent row, operation or task.	Check the ID.
`invalid_transition`	A state change that is not permitted.	Read the current state.
`read_only`	A write through a read-only store.	Use a command that owns the store.
`closed`	Use after `Close`.	Open the store again.
`corrupt`	Damaged or inconsistent stored data.	Restore from a backup.
`invalid_snapshot`	A snapshot file that is not valid `ka2a.snapshot/1` JSON.	Read the file with the ka2a revision that wrote it, or a newer one. See the output formats.
`invalid_catalog`	The catalog file is malformed or unsafe.	Run `ka2a catalog validate`. See Catalog check fails.
`invalid_key_file`	The signing key file is malformed.	Generate the key again with `ka2a keys generate`.
`unsafe_file`	A file with an unsafe type, owner or permissions.	Make the file a regular file of the service user, not group or other writable.
`unknown_key`	The catalog does not list the key ID.	Add the key to the catalog.
`revoked_key`	The key is revoked.	Use another key.
`key_not_valid`	The key is outside its validity window.	Use a key with a current window. See Signing key expires soon.
`key_mismatch`	The private key does not match the catalog.	Check `--key` and the catalog entry.
`source_mismatch`	The claimed source differs from the source bound to the key.	Check the catalog entry of the key.
`unknown_endpoint`	The catalog does not list the endpoint.	Add the endpoint to the catalog.
`unsupported_operation`	An operation outside the profile.	Use an operation of the profile.
`not_authorized`	The catalog denies the request.	Add the grant.

Broker contact failure classes

The node status (Status.Broker.FailureClass), the log, ka2a doctor and the result of a credential reload name the class of the last failed broker contact. A class never holds an address. See Refused broker connection.

Class	Meaning	Action
`connection_refused`	The broker host refused the TCP connection.	Check that the broker runs and listens on the port of `--brokers`.
`timeout`	The contact attempt timed out.	Check the network path and the broker load.
`address_unresolved`	DNS cannot resolve the broker name.	Check the name in `--brokers` and the resolver.
`network_unreachable`	No route to the broker network.	Check the routing and the firewall.
`tls_failed`	TLS refused the connection. The hint names the cause.	See Connection hints.
`authentication_failed`	SASL authentication failed.	Check the user name, the password file and the mechanism. See Rotated SASL password.
`disconnected`	The broker closed the connection.	Usually transient. If it stays, check the listener security.
`request_failed`	Another request failure.	Read the log line.
`authorization_failed`	The broker answered and denied the request: the principal lacks an ACL. It is not a contact failure; the scope names the ACL.	Add the ACL of the scope. See Missing broker ACL.

Authorization scopes

Scope	Missing ACL
`topic_write`	Write on a peer mailbox topic (produce).
`topic_read`	Read on the own mailbox topic (fetch, offsets).
`topic_describe`	Describe or DescribeConfigs on a topic (metadata, the mailbox check).
`group_read`	Read on the consumer group (join, commit).
`cluster`	A cluster or transactional ID permission, for example IdempotentWrite on a broker before Kafka 3.0.

ka2a topics acl-plan prints the minimal ACLs of an endpoint.

Connection hints

A refused TLS or SASL connection has one of these fixed hints (Status.Broker.FailureHint, the doctor detail and the reload result). A hint never holds a credential, a certificate field or broker text.

Hint	Action
"the broker certificate does not chain to a trusted CA"	Give the CA of the broker in `--tls-ca`.
"the broker certificate is not valid for the server name"	Set `--tls-server-name`, or use a broker name of the certificate.
"the broker certificate is not valid (for example expired or not yet valid)"	Renew the broker certificate. Check the clock of the host.
"the broker listener needs TLS, and the client connected without TLS"	Give `--tls-ca` (TLS on).
"the broker listener does not speak TLS"	Use the TLS listener port, or remove the TLS flags.
"the TLS handshake failed (for example no common TLS version or cipher, or a missing or refused client certificate)"	Check the TLS versions, the ciphers and the client certificate.
"SASL authentication failed (user name, password or mechanism)"	Check `--sasl-username`, `--sasl-password-file` and `--sasl-mechanism`.
"the broker does not enable this SASL mechanism"	Use a mechanism that the listener enables.
"the broker closed the connection at once (SASL is missing, or TLS is used against a listener without TLS)"	Match the client security to the listener.
"the broker refused the client certificate (untrusted CA, expired, revoked or not valid for client authentication)"	See Refused or expiring client certificate.
"the broker listener needs a client certificate (mutual TLS), and the client sent none that a CA trusted by the broker issued"	Give `--tls-cert` and `--tls-key`.
"a certificate of the broker chain is revoked by the revocation list of the client (--tls-crl)"	Replace the broker certificate. See Revocation list due.

Publish reasons

The publisher classifies each publish attempt. A record that the broker refused for sure gets the outbox state error with the reason publish_<reason>, for example publish_record_too_large, and its exchange becomes held. A retried attempt keeps the record queued.

Reason	Outcome	Meaning
`record_too_large`	refused	The record exceeds `max.message.bytes` of the topic or the broker.
`invalid_record`	refused	The broker refused the record or the batch as invalid, or a topic policy refused it.
`invalid_partition`	refused	The partition does not exist.
`topic_not_found`	refused	The topic does not exist. Provision the peer mailbox.
`unauthorized`	refused	The principal lacks Write on the topic (scope `topic_write`) or a cluster permission.
`authentication_failed`	unknown, retried	SASL refused the connection. A credential reload fixes it.
`canceled`	not sent or unknown, retried	A stop or a deadline ended the attempt.
`closed`	unknown, retried	The client was closed during the attempt.
`buffer_full`	not sent, retried	The client buffer was full.
`timeout`	unknown, retried	The broker did not answer in time.
`not_enough_replicas`	unknown, retried	Fewer in-sync replicas than `min.insync.replicas`.
`disconnected`	unknown, retried	The connection broke during the attempt.
`broker_error`	unknown, retried	Another broker error.
`not_sent`	retried	Outbox reason: the record was not sent. The publisher tries again.
`publish_outcome_unknown`	retried	Outbox reason: the outcome is unknown. The publisher sends the same bytes again.
`publish_rejected`	refused	Outbox reason: the broker refused the record without a specific reason.

Rejection reasons

The node records a bounded rejection disposition for each received record that it does not accept (ka2a_input_rejected_total, ka2a_responses_rejected_total, the rejections list of ka2a observe). A rejected record never reaches a handler. The disposition holds only syntactically valid claims of the headers; they are not authenticated.

Reason	Direction	Meaning	Action
`malformed`	request, response	Framing, header, JSON or canonical form that does not agree with KA2A-WIRE-1.	Check the sender version.
`too_large`	request, response	A value, a header, the depth or the node count exceeds a bound.	Check the sender.
`signature_invalid`	request, response	The signature is missing, malformed or incorrect.	Check the sender key and the catalog. A forged record has this reason.
`unknown_key`	request, response	The catalog does not know the signing key.	Add the key of the sender to the catalog and restart.
`revoked_key`	request, response	The signing key is revoked.	The sender must use another key.
`key_not_valid`	request, response	The signing key is outside its validity window.	Check the key window and the clocks.
`unauthorized`	request, response	The key is not bound to the claimed source, or the catalog has no grant.	Fix the catalog grant.
`route_mismatch`	request, response	The record names another destination or came on another topic.	Check the routing of the sender.
`expired`	request, response	The record is at or after its signed expiry.	Check the clocks and the delivery delay.
`future_time`	request, response	The creation time is later than the local clock plus the allowed skew.	Synchronize the clocks (NTP).
`horizon_exceeded`	request, response	The expiry is farther from creation than the horizon of the record kind.	Check the sender.
`unknown_schema`	request, response	The schema digest is not in the embedded catalog.	Check the sender version.
`schema_invalid`	request, response	The document does not agree with its schema.	Check the sender.
`semantic_invalid`	request, response	The document breaks a semantic rule of A2A 1.0 or of the profile.	Check the sender.
`unsupported_operation`	request, response	The operation is outside the profile.	Check the sender.
`unsupported_feature`	request, response	An optional feature is outside the profile.	Check the sender.
`session_mismatch`	request, response	The session header does not mirror the A2A contextId.	Check the sender.
`identity_conflict`	request	The operation identity is reused for another request.	See identity_conflict.
`replay_after_purge`	request	A request whose identity retention purged arrived again.	None. The node does not run it again.
`late_response`	response	A response for an operation that retention purged.	None.
`orphan_response`	response	A verified response for an operation that the store does not know. The log warns that local sender state can be lost.	Check for a restore of an old backup.
`correlation_mismatch`	response	The response comes from another peer or names another parent.	Check the peer.
`response_conflict`	response	A second, different final response. The first response stays.	Check the peer.

Hold reasons

A held dispatch row or a held exchange has a reason. See Held work.

Reason	Where	Meaning
`ambiguous_after_restart`	dispatch	The handler had started before a crash or a stop, so its effect is unknown.
`restart_retry`	dispatch	An `idempotent_with_key` handler is queued again after a restart with the same key.
`store_quarantined`	dispatch	A restore quarantine held the row.
`handler_error`	dispatch	The handler returned an error whose effect is unknown.
`handler_panic`	dispatch	The handler panicked.
`handler_timeout`	dispatch	The handler did not return before its deadline.
`invalid_result`	dispatch	The handler returned a result that the profile refuses.
`task_conflict`	dispatch	The result conflicts with the stored task.
`response_build_failed`	dispatch	The node could not build the response record.
`send_failed`	exchange	The broker refused the request record. An earlier attempt can have reached the receiver.
`send_expired`	exchange	The request record expired before a publication.
`operator_abandoned`	outbox	An operator abandoned the exchange of the record.

ka2a held release and ka2a held abandon record the reason token that the operator gives.

Quarantine reasons

A restore quarantine has one of these reasons. Each one is a recovery incident. See Quarantine after a restore.

Reason	Meaning
`restore_marker`	The state directory has a recovery marker.
`restored_backup`	The database is a backup copy.
`epoch_witness_missing`	The database exists, but the epoch witness does not.
`epoch_witness_node_mismatch`	The witness names another node.
`generation_mismatch`	The witness names another store generation.
`epoch_behind`	The database epoch is behind the witness.
`epoch_witness_behind`	The witness is more than one epoch behind.
`broker_ahead`	The broker committed input that the store never accounted for.

Protocol codes

pkg/protocol classifies a failure of the wire codec with these codes. The receiver maps them to the rejection reasons.

Code	Meaning
`malformed`	Framing, header syntax, JSON syntax, JSON encoding or canonical form that does not agree with KA2A-WIRE-1.
`too_large`	A value, header, depth or node bound that the input exceeds.
`unknown_header`	A header key that is not in the profile table.
`duplicate_header`	A header key that occurs more than once.
`bad_signature`	A missing, malformed or incorrect signature.
`unknown_key`	A signing key that the trusted catalog does not know or no longer accepts.
`unauthorized_source`	A verified key that is not bound to the asserted source endpoint.
`unauthorized`	A verified principal that may not send this record kind and operation to the destination.
`wrong_destination`	A record that names another destination or arrived on another topic.
`expired`	A record at or after its signed expiry.
`future_time`	A creation time later than the verifier clock plus the allowed skew.
`horizon_exceeded`	An expiry farther from creation than the horizon of the record kind.
`schema_unknown`	A schema digest that is not in the embedded catalog.
`schema_invalid`	A document that does not agree with its structural schema or the pinned A2A types.
`semantic_invalid`	A structurally valid document that breaks a semantic rule.
`unsupported_operation`	An operation that is not in the declared profile.
`unsupported_feature`	An optional A2A feature that the profile does not support.
`session_mismatch`	A session header that does not mirror the A2A contextId.
`parent_mismatch`	A response parent that is not the expected request event ID.
`peer_mismatch`	A response from another verified endpoint than the expected peer.
`operation_mismatch`	A record that does not belong to the expected operation exchange.
`identity_conflict`	One operation identity with two different logical requests or frozen records.
`invalid_argument`	Builder input that does not agree with the profile.
`invalid_config`	A configuration that fails validation.
`signing_failed`	A signer that failed or returned a signature that does not verify.

Topic check results

ka2a topics verify and ka2a topics provision print a result.

Result	Meaning	Exit
`verified`	Every check passed.	0
`unverified`	The broker did not let the command read a setting. `--strict` makes it a problem.	0
`mismatch`	A setting differs from the profile.	3
`authorization_failed`	The broker denied a request of the check.	3
`connection_failed`	TLS, SASL or the network refused the connection.	3

All ka2a documents