Tarn
Reference

Failure Categories

Stable failure taxonomy used in JSON reports, MCP tools, and fix plans. Each category has a specific diagnosis and fix path.

Overview

Every failed step includes a failure_category field in the JSON report. Agents branch on this field instead of parsing assertion messages. This gives a stable taxonomy for routing failures to the right fix logic.

Category Tag Meaning
assertion_failed assertion Request succeeded, response received, but assertion(s) didn't match
connection_error connection DNS / connect / TLS failure before a usable response
timeout timeout Request exceeded allowed time
parse_error parse Invalid YAML syntax or schema validation failure
capture_error capture Assertions passed, but capture extraction failed

assertion assertion_failed

What it means: The HTTP request was sent successfully, a response was received, but one or more assertions didn't match the actual response.

Typical example: Expected status: 201 but got 400. Expected $.name to equal "Jane" but got "John".

How to diagnose:

  • Inspect response.status and response.body in the report
  • Compare assertions.failures[].expected vs assertions.failures[].actual
  • Check if the API changed its response shape

How to fix:

  • Adjust assertion values to match the real response
  • Use tarn run --verbose to see full responses
  • Use tarn-lsp's "scaffold assert.body from last response" code action

connection connection_error

What it means: The HTTP request could not be completed. No usable HTTP response was received.

Typical example: DNS resolution failure, connection refused, TLS certificate error, or network unreachable.

How to diagnose:

  • Check the error message in the step output
  • Verify the URL is correct and the server is running
  • Check DNS resolution: nslookup api.example.com
  • Verify TLS certificates if using HTTPS

How to fix:

  • Ensure the target server is running and reachable
  • Check base_url in env files
  • Use --insecure flag (temporarily) if TLS is the issue
  • Use --cacert for custom CA bundles
Common Mistake

A common cause is forgetting to start the local demo server (cargo run -p demo-server) before running tests against http://localhost:3000.

timeout timeout

What it means: The request exceeded the allowed time and was aborted before a response was received.

Typical example: A slow endpoint, network latency, or too low a timeout value.

How to diagnose:

  • Check the configured timeout for the step or globally
  • Test the endpoint manually with curl to measure response time

How to fix:

  • Increase step-level timeout in the test file
  • Set a higher global timeout in tarn.config.yaml
  • Use polling for endpoints that need time to process

parse parse_error

What it means: The YAML file has syntax errors, or fails schema validation. No tests were executed.

Typical example: Invalid indentation, missing comma in YAML, unknown field name, missing required field.

How to diagnose:

  • Run tarn validate to get precise error location
  • Run tarn validate --format json for machine-readable diagnostics
  • Check for {{ ... }} interpolation syntax errors

How to fix:

  • Fix YAML indentation (use spaces, not tabs)
  • Ensure all required fields are present
  • Use tarn fmt to normalize formatting
  • Add the schema comment for IDE autocompletion

capture capture_error

What it means: Assertions passed, but the response didn't contain the field or header needed for a capture extraction.

Typical example: JSONPath $.id points to a field that doesn't exist in the response body, or a header set-cookie isn't present.

How to diagnose:

  • Check the response body/headers in the report
  • Verify the JSONPath expression matches the actual response structure
  • Check if the API changed response format

How to fix:

  • Update the JSONPath to match the actual response
  • Make captures optional: true if the field isn't always present
  • Add default: <value> for fallback values