Splunk distribution of OpenTelemetry Go

The Splunk distribution of OpenTelemetry Go provides multiple packages that automatically instruments your Go application to capture and report distributed traces to Splunk APM.

🚧 This project is currently in BETA. It is officially supported by Splunk. However, breaking changes MAY be introduced.

Table of Contents:

• Getting Started
  • Basic Configuration
• Library Instrumentation
• Manual Instrumentation
• Advanced Configuration
  • Splunk Distribution Configuration
  • Trace Configuration
  • Trace Exporter Configuration
  • Trace Propagation Configuration
• License

Getting Started

This Splunk distribution comes with the following defaults:

• B3 context propagation.
• Jaeger Thrift over HTTP exporter configured to send spans to a locally running Splunk OpenTelemetry Connector (http://localhost:14268/api/traces).
• Unlimited default limits for configuration options to support full-fidelity traces.

Install the distribution:

go get github.com/signalfx/splunk-otel-go/distro

Configure OpenTelemetry using the distro package:

package main

import (
	"context"

	"github.com/signalfx/splunk-otel-go/distro"
)

func main() {
	sdk, err := distro.Run()
	if err != nil {
		panic(err)
	}
	// Ensure all spans are flushed before the application exits.
	defer func() {
		if err := sdk.Shutdown(context.Background()); err != nil {
			panic(err)
		}
	}()

	// ...

Basic Configuration

The service.name resource attribute is the only configuration option that needs to be specified using the OTEL_RESOURCE_ATTRIBUTES environment variable.

The deployment.environment and service.version resource attributes are not strictly required, but recommended to be set if they are available.

It can be done in the shell:

OTEL_RESOURCE_ATTRIBUTES="service.name=my-app,service.version=1.2.3,deployment.environment=production"

As well as in Go code before executing distro.Run():

os.Setenv("OTEL_RESOURCE_ATTRIBUTES", "service.name=my-app,service.version=1.2.3,deployment.environment=development")

Library Instrumentation

Supported libraries are listed here.

Additional recommended Splunk specific instrumentations:

• splunkbuntdb
• splunkchi
• splunkclient-go
• splunkdns
• splunkgorm
• splunkgraphql
• splunkhttp
• splunkhttprouter
• splunkkafka
• splunkleveldb
• splunkmysql
• splunkpgx
• splunkpq
• splunksql
• splunksqlx

Manual Instrumentation

Documentation on how to manually instrument a Go application is available here.

Advanced Configuration

Below you will find all the configuration options supported by this distribution.

Splunk Distribution Configuration

Environment variable	Option	Default value	Description
SPLUNK_ACCESS_TOKEN	WithAccessToken		The Splunk's organization access token. [1]
OTEL_RESOURCE_ATTRIBUTES			Comma-separated list of resource attributes added to every reported span.

[1]: The Splunk's organization access token allows exporters sending data directly to the Splunk Observability Cloud. To do so, the OTEL_EXPORTER_JAEGER_ENDPOINT must be set or distro.WithEndpoint must be passed to distro.Run with Splunk back-end ingest endpoint URL: https://ingest.<REALM>.signalfx.com/v2/trace.

Trace Configuration

Environment variable	Option	Default value	Description
OTEL_RESOURCE_ATTRIBUTES			Comma-separated list of resource attributes added to every reported span.

Trace Exporter Configuration

Environment variable	Option	Default value	Description
OTEL_EXPORTER_JAEGER_ENDPOINT	WithEndpoint	http://localhost:14268/api/traces	Jaeger Thrift HTTP endpoint for sending spans.
OTEL_EXPORTER_JAEGER_USER			Username to be used for HTTP basic authentication.
OTEL_EXPORTER_JAEGER_PASSWORD			Password to be used for HTTP basic authentication.

Trace Propagation Configuration

The trace propagtor can be changed by using otel.SetTextMapPropagator after distro.Run() is invoked e.g.:

distro.Run()
otel.SetTextMapPropagator(propagation.TraceContext{})

License

The Splunk distribution of OpenTelemetry Go is a distribution of the OpenTelemetry Go project. It is released under the terms of the Apache Software License version 2.0. See the license file for more details.