# OpenTelemetry

Hatchet supports exporting traces from your tasks to an [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/) to improve visibility into your Hatchet tasks.

## Setup

#### Python

Install the `otel` extra:

```bash
pip install hatchet-sdk[otel]
```

Then create the instrumentor and call `instrument()`:

```python
from hatchet_sdk.opentelemetry.instrumentor import HatchetInstrumentor

HatchetInstrumentor().instrument()
```

By default, `HatchetInstrumentor` creates a `TracerProvider` that sends spans to the Hatchet engine's OTLP collector. You can also pass your own `TracerProvider`:

```python
HatchetInstrumentor(
    tracer_provider=your_tracer_provider,
).instrument()
```

### Options

Option, Type, Default, Description

`tracer_provider`, `TracerProvider`, —, Custom TracerProvider. If not set, one is created automatically.
`enable_hatchet_otel_collector`, `bool`, `True`, Send traces to the Hatchet engine's OTLP collector.
`schedule_delay_millis`, `int`, —, Delay between consecutive exports of the BatchSpanProcessor.
`max_export_batch_size`, `int`, —, Maximum batch size for the BatchSpanProcessor.
`max_queue_size`, `int`, —, Maximum queue size for the BatchSpanProcessor.

#### TypeScript

Install the required OpenTelemetry packages:

```bash
npm install @opentelemetry/api @opentelemetry/instrumentation @opentelemetry/sdk-trace-base @opentelemetry/exporter-trace-otlp-grpc
```

Register the instrumentor before creating any Hatchet clients or workers:

```typescript
const { registerInstrumentations } = require("@opentelemetry/instrumentation");
import { HatchetInstrumentor } from "@hatchet-dev/typescript-sdk/opentelemetry";

registerInstrumentations({
  instrumentations: [new HatchetInstrumentor()],
});
```

By default, `HatchetInstrumentor` sends spans to the Hatchet engine's OTLP collector. You can disable this with `enableHatchetCollector: false`.

### Options

Option, Type, Default, Description

`enableHatchetCollector`, `boolean`, `true`, Send traces to the Hatchet engine's OTLP collector.
`clientConfig`, `object`, —, Override Hatchet client config for the collector connection.
`includeTaskNameInSpanName`, `boolean`, `false`, Append the task action ID to the `hatchet.start_step_run` span name.
`excludedAttributes`, `array`, `[]`, List of `hatchet.*` attribute keys to exclude from spans.

#### Go

Import the `opentelemetry` package (commonly aliased as `hatchetotel`):

```go
import hatchetotel "github.com/hatchet-dev/hatchet/sdks/go/opentelemetry"
```

Create the instrumentor, then register its middleware on the worker:

```go
instrumentor, err := hatchetotel.NewInstrumentor()
if err != nil {
    log.Fatalf("failed to create instrumentor: %v", err)
}

worker.Use(instrumentor.Middleware())
```

By default, `NewInstrumentor` creates a `TracerProvider` that sends spans to the Hatchet engine's OTLP collector.

Remember to shut down the instrumentor on exit to flush remaining spans:

```go
defer instrumentor.Shutdown(context.Background())
```

### Options

Option, Description

`hatchetotel.WithTracerProvider(tp)`, Use a custom `*sdktrace.TracerProvider` instead of creating a new one.
`hatchetotel.DisableHatchetCollector()`, Disable sending traces to the Hatchet engine's OTLP collector.
`hatchetotel.WithBatchSpanProcessorOptions(...)`, Configure the `BatchSpanProcessor` for the Hatchet collector.

#### Ruby

> **Info:** OpenTelemetry support for the Ruby SDK is coming soon.

## Spans

By default, Hatchet creates spans at the following points in the lifecycle of a task run:

1. **Producer spans** — when a trigger is called on the client side (e.g. `run()`, `runNoWait()`, `push()`, `schedule()`).
2. **Consumer spans** — when a worker handles a task run, such as starting to run the task (`hatchet.start_step_run`) or cancelling a task (`hatchet.cancel_step_run`).

### Span Names

Span Name, Kind, Description

`hatchet.start_step_run`, `CONSUMER`, Worker begins executing a task.
`hatchet.cancel_step_run`, `CONSUMER`, Worker cancels a running task.
`hatchet.run_workflow`, `PRODUCER`, Client triggers a workflow run.
`hatchet.run_workflows`, `PRODUCER`, Client triggers multiple workflow runs.
`hatchet.schedule_workflow`, `PRODUCER`, Client creates a scheduled workflow run.
`hatchet.push_event`, `PRODUCER`, Client pushes an event.
`hatchet.bulk_push_event`, `PRODUCER`, Client pushes events in bulk.
`hatchet.durable.wait_for`, `INTERNAL`, Durable task waits for a signal/condition.

### Span Attributes

All spans include an `instrumentor` attribute set to `"hatchet"`. Consumer spans (`hatchet.start_step_run`) include the following `hatchet.*` attributes:

Attribute, Type, Description

`hatchet.tenant_id`, `string`, The tenant ID for the task run.
`hatchet.worker_id`, `string`, The worker handling the task.
`hatchet.workflow_run_id`, `string`, The workflow run ID.
`hatchet.step_run_id`, `string`, The task run ID.
`hatchet.step_id`, `string`, The task ID.
`hatchet.workflow_name`, `string`, The workflow name.
`hatchet.action_name`, `string`, The action ID (format: `workflowName:taskName`).
`hatchet.step_name`, `string`, The human-readable task name.
`hatchet.retry_count`, `int`, Current retry attempt (0-indexed).
`hatchet.workflow_id`, `string`, The workflow definition ID (if available).
`hatchet.workflow_version_id`, `string`, The workflow version ID (if available).
`hatchet.parent_workflow_run_id`, `string`, Parent workflow run ID (for child workflows).
`hatchet.child_workflow_index`, `int`, Child workflow index (for child workflows).
`hatchet.child_workflow_key`, `string`, Child workflow key (for child workflows).

Producer spans (`hatchet.run_workflow`) include `hatchet.step_name` (the workflow being triggered) and, on success, `hatchet.child_workflow_run_id`.

### Context Propagation

All SDKs:

1. Automatically inject W3C `traceparent` into `additionalMetadata` on producer spans, so consumer spans on the worker become children of the trigger span.
2. Provide an `HatchetAttributeSpanProcessor` that propagates `hatchet.*` attributes from the parent task run span to all child spans created within the task. This means any custom spans you create inside a task function will automatically carry the same `hatchet.*` attributes.