Sources Contact Advanced Search Tutorials

An Interest In:

Web News this Week

Search Archive

Some of Our Sources

View All Sources

Help Webnuz

Referal links:

January 19, 2023 06:41 pm GMT

Observability-driven development with Go and Tracetest

Were entering a new era of observability-driven development. ODD uses OpenTelemetry instrumentation as assertions in tests!

Here's an awesome explanation on Twitter!

This is driving a new culture of trace-based testing. With trace-based testing, you can generate integration tests from OpenTelemetry-based traces, enforce quality, encourage velocity, and increase test coverage in microservices and distributed apps.

Today, youll learn to build a distributed system with Go and Docker. Youll instrument it with OpenTelemetry traces and use Tracetest to run trace-based tests on top of OpenTelemetry infrastructure.

Well follow observability-driven development principles and showcase why its powerful in todays world of developing distributed systems in the cloud.

By the end of this tutorial youll learn observability-driven development, how to develop microservices with Go, and how to run trace-based tests with Tracetest.

To check out the entire code, jump over to GitHub.

What are we building?

The distributed app were building will have two microservices and a dedicated module for OpenTelemetry trace instrumentation.

Well use Go to write the code and Docker Compose to deploy the microservices. The app itself is a bookstore that shows books with their price and availability.

We will follow observability-driven development best practices of first writing a test, then writing the code and adding OpenTelemetry instrumentation to validate the test spec, and finally run trace-based tests with Tracetest and make sure they pass.

The tutorial consists of 3 parts.

The first part will be dedicated to configuring the basic bookstore infrastructure and books service, setting up OpenTelemetry instrumentation, and installing Tracetest.

In the second part we will focus on running ODD tests hands-on and creating trace-based tests for the books service.

The third, and final, part will focus on creating the availability service and covering it with tests.

Before we begin, lets quickly explain what OpenTelemetry and Tracetest are.

What is OpenTelemetry?

OpenTelemetry is an observability framework that assist in generating and capturing telemetry data from cloud-native software.

OpenTelemetry gathers observability data includingtraces, metrics, and logs.

OpenTelemetry is a community-driven open-source project and as of August 2021 is a CNCF incubating project. OpenTelemetry is the second most active CNCF project behind Kubernetes.

These are the three components well use in this guide:

OpenTelemetry Go APIs and SDKsfor generating and emitting telemetry
OpenTelemetry Collectorto receive, process and export telemetry data
OTLPprotocol for transmitting telemetry data

Because OpenTelemetry is a framework, you need a data store to persist traces. Well demo how to use Jaeger as the trace data store and the OpenTelemetry Collector as the gateway to funnel traces to Jaeger for storage.

What is Tracetest?

Tracetest uses your existingOpenTelemetrytraces to power trace-based testing with assertions against your trace data at every point of the request transaction.

Youll point Tracetest to the existing Jaeger trace data source. Tracetest will then pull traces from Jaeger while running integration tests to run assertions against the trace data itself.

There is an option to bypass using a trace data store entirely, and send traces to Tracetest right away by configuring the OpenTelemetry collector as explained in our docs.

With that, were ready to start coding!

Setting up the infrastructure for observability-driven development

This section will explain the initial configuration of both the bookstore app and Tracetest. Well use Docker Compose for the infrastructure. Once done, youll have a running app with Tracetest configured for running tests.

Installing Tracetest for local development

You can follow the sample code weve prepared for part 1 of this tutorial. Follow the instructions below:

git clone [email protected]:kubeshop/tracetest.gitcd examples/observability-driven-development-go-tracetest/bookstore/part1docker compose -f docker-compose.yaml -f tracetest/docker-compose.yaml up

Note: View the 1st part of our observability-driven development video tutorial, here!

Lets walk through the installation step-by-step.

Start by installing the Tracetest CLI. Its the easiest way of getting started with Tracetest.

brew install kubeshop/tracetest/tracetest

Note: Follow this guide to install for your specific operating system.

Once the CLI is installed, create a directory called bookstore, and install a Tracetest server.

tracetest server install

Follow the prompts and install a bare-bones setup with just Tracetest. This will generate an emptydocker-compose.yamlfile and a./tracetest/directory that contains anotherdocker-compose.yaml.

Configuring OpenTelemetry Collector and Jaeger with Tracetest

Lets edit the docker-compose.yaml to add OpenTelemetry Collector and Jaeger.

# ./tracetest/docker-compose.yamlservices:    jaeger:        healthcheck:            test:                - CMD                - wget                - --spider                - localhost:16686            timeout: 3s            interval: 1s            retries: 60        image: jaegertracing/all-in-one:latest        networks:            default: null        restart: unless-stopped    otel-collector:        command:            - --config            - /otel-local-config.yaml        depends_on:            jaeger:                condition: service_started        image: otel/opentelemetry-collector:0.54.0        networks:            default: null        volumes:            - type: bind              source: tracetest/otel-collector.yaml              target: /otel-local-config.yaml              bind:                create_host_path: true    postgres:        environment:            POSTGRES_PASSWORD: postgres            POSTGRES_USER: postgres        healthcheck:            test:                - CMD-SHELL                - pg_isready -U "$$POSTGRES_USER" -d "$$POSTGRES_DB"            timeout: 5s            interval: 1s            retries: 60        image: postgres:14        networks:            default: null    tracetest:        depends_on:            otel-collector:                condition: service_started            postgres:                condition: service_healthy        extra_hosts:            host.docker.internal: host-gateway        healthcheck:            test:                - CMD                - wget                - --spider                - localhost:11633            timeout: 3s            interval: 1s            retries: 60        image: kubeshop/tracetest:v0.9.3        networks:            default: null        ports:            - mode: ingress              target: 11633              published: 11633              protocol: tcp        volumes:            - type: bind              source: tracetest/tracetest.yaml              target: /app/config.yamlnetworks:    default:        name: _default

Let me explain whats going on in the docker-compose.yaml file:

We hook up the OpenTelemetry Collector to act as a gateway for all the traces our app will generate and Jaeger as a trace data store.
The OpenTelemetry Collector will receive all traces from our Go microservices and send them to Jaeger.
We will then configure Tracetest to fetch trace data from Jaeger when running trace-based tests.

Make sure your config files for Tracetest and the OpenTelemetry Collector match the sample code. First copy-paste this into your otel-collector.yaml.

# ./tracetest/otel-collector.yamlexporters:    jaeger:        endpoint: jaeger:14250        tls:            insecure: trueprocessors:    batch:        timeout: 100msreceivers:    otlp:        protocols:            grpc: null            http: nullservice:    pipelines:        traces:            exporters:                - jaeger            processors:                - batch            receivers:                - otlp

Now, from the bookstore directory, start Docker Compose to test the Tracetest installation.

docker compose -f docker-compose.yaml -f tracetest/docker-compose.yaml up

This command will spin up the infrastructure, and expose Tracetest on port 11633. Open up http://localhost:11633/ in your browser.

Configure your trace data store to point to Jaeger in the Web UI.

You can also configure Jaeger via the CLI.

Adding the `books` microservice

In the bookstore directory, create a books directory, and initialize a Go module.

cd ./booksgo mod init github.com/your-username/bookstore/books

In the books directory create a main.go file. Paste this code into the main.go:

// ./books/main.gopackage mainimport (    "context"    "fmt"    "io"    "log"    "net/http"    "time"    "github.com/gorilla/mux"    "go.opentelemetry.io/contrib/instrumentation/github.com/gorilla/mux/otelmux"    "go.opentelemetry.io/otel"    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"    "go.opentelemetry.io/otel/propagation"    "go.opentelemetry.io/otel/sdk/resource"    sdktrace "go.opentelemetry.io/otel/sdk/trace"    semconv "go.opentelemetry.io/otel/semconv/v1.12.0"    "go.opentelemetry.io/otel/trace"    "google.golang.org/grpc"    "google.golang.org/grpc/credentials/insecure")const svcName = "books"var tracer trace.Tracerfunc newExporter(ctx context.Context) (sdktrace.SpanExporter, error) {    ctx, cancel := context.WithTimeout(ctx, time.Second)    defer cancel()    conn, err := grpc.DialContext(ctx, "otel-collector:4317", grpc.WithTransportCredentials(insecure.NewCredentials()), grpc.WithBlock())    if err != nil {        return nil, fmt.Errorf("failed to create gRPC connection to collector: %w", err)    }    traceExporter, err := otlptracegrpc.New(ctx, otlptracegrpc.WithGRPCConn(conn))    if err != nil {        return nil, fmt.Errorf("failed to create trace exporter: %w", err)    }    return traceExporter, nil}func newTraceProvider(exp sdktrace.SpanExporter) *sdktrace.TracerProvider {    // Ensure default SDK resources and the required service name are set.    r, err := resource.Merge(        resource.Default(),        resource.NewWithAttributes(            semconv.SchemaURL,            semconv.ServiceNameKey.String(svcName),        ),    )    if err != nil {        panic(err)    }    tp := sdktrace.NewTracerProvider(        sdktrace.WithBatcher(exp),        sdktrace.WithResource(r),    )    otel.SetTextMapPropagator(        propagation.NewCompositeTextMapPropagator(            propagation.TraceContext{},            propagation.Baggage{},        ),    )    return tp}func main() {    ctx := context.Background()    exp, err := newExporter(ctx)    if err != nil {        log.Fatalf("failed to initialize exporter: %v", err)    }    // Create a new tracer provider with a batch span processor and the given exporter.    tp := newTraceProvider(exp)    // Handle shutdown properly so nothing leaks.    defer func() { _ = tp.Shutdown(ctx) }()    otel.SetTracerProvider(tp)    // Finally, set the tracer that can be used for this package.    tracer = tp.Tracer(svcName)    r := mux.NewRouter()    r.Use(otelmux.Middleware(svcName))    r.HandleFunc("/books", booksListHandler)    http.Handle("/", r)    log.Fatal(http.ListenAndServe(":8001", nil))}func booksListHandler(w http.ResponseWriter, r *http.Request) {    _, span := tracer.Start(r.Context(), "Books List")    defer span.End()    io.WriteString(w, "Hello!
")}

Lets walk through whats happening in the ./books/main.go file:

The newExporter function is defining how to export trace data and to forward it to the OpenTelemetry Collector we have running on otel-collector:4317.
The newTraceProvider function is initializing the tracer that we use to instrument the code.
The main function is initializing everything and defines an HTTP route called /books to trigger a booksListHandler.
The booksListHandler function will return a simple "Hello!" string. It also starts the OpenTelemetry tracer and defines a span called "Books List" .

With all this added, fetch Go dependencies by running this command in the terminal from the books directory:

go mod tidy

This will generate a go.sum file.

Lastly, add a books service to the docker-compose.yaml file in the bookstore directory. This is the root docker-compose.yaml file, not the one inside the ./tracetest/ directory.

# ./docker-compose.yamlservices:  books:    image: your_username/books    build:      args:        SERVICE: books    ports:      - 8001:8001    depends_on:      otel-collector:        condition: service_started

Next, create a Dockerfile and paste this code into it:

# ./DockerfileFROM golang:1.19ARG SERVICEWORKDIR /app/${SERVICE}COPY ./${SERVICE}/go.* /app/${SERVICE}RUN go mod downloadCOPY ./${SERVICE}/* /app/${SERVICE}RUN go build -o /app/server .ENTRYPOINT [ "/app/server" ]

Finally, restart Docker Compose to try out the books service.

docker compose -f docker-compose.yaml -f tracetest/docker-compose.yaml up

Running a trace-based test in the Tracetest web UI

With the Tracetest service running on port 11633, open it up on http://localhost:11633/ in your browser.

Create a new HTTP test. Give it a name and make sure to set the URL to http://books:8001/books.

Click create. This will trigger the test to run right away.

The test will return a 200 status code. Next, we need to add assertions against the trace data to make sure our OpenTelemetry tracing instrumentation works in our Go code.

Open the Trace tab and lets start by adding a status code assertion. Click on the Tracetest trigger span. In the left navigation, select tracetest.response.status and click Create test spec. If youre writing the assertion by hand, make sure to preface the attribute with a attr: to enable autocompletion when selecting what attribute to assert on.

Save the test spec and add another assertion to the Books list span. This time add the attribute called attr:tracetest.selected_spans.count = 1.

Save and publish the test specs. Re-run the test.

You now have passing tests that ensure the service responds with a 200 status code and verifies the OpenTelemetry manual code instrumentation works!

Running a trace-based test with the Tracetest CLI

Lets re-trace, no pun intended, our steps with the Tracetest CLI.

Create an e2e directory in the bookstore directory. Create a file called books-list.yaml. This will contain a test definition we will trigger with the CLI.

Paste this code into the books-list.yaml:

# ./e2e/books-list.yamltype: Testspec:  id: k6hEWU54R  name: Books Listing  description: Try books service  trigger:    type: http    httpRequest:      url: http://books:8001/books      method: GET      headers:      - key: Content-Type        value: application/json  specs:    - selector: span[name="Tracetest trigger"]      assertions:        - attr:tracetest.response.status = 200    - selector: span[name="Books List"]      assertions:        - attr:tracetest.selected_spans.count = 1

Take a moment to read the code. You see the assertions match what we just added in the Tracetest Web UI.

To trigger the test from the command line, first configure the Tracetest CLI. Make sure to point the CLI to the URL where the Tracetest service is running. In this sample its http://localhost:11633/.

tracetest configure[Output]Enter your Tracetest server URL [http://localhost:11633]: http://localhost:11633[Output]Enable analytics? [Y/n]: Yes

Now, we can run the test. From the bookstore dir, run:

tracetest test run -d ./e2e/books-list.yaml -w[Output] Books Listing (http://localhost:11633/test/k6hEWU54R/run/2/test)

Clicking the link will open the test run in the Web UI.

With the initial setup done, were ready to move on and tackle getting hands-on with observability-driven development!

Hands-on observability-driven development

To follow along, you can check out the sample code weve prepared for part 2.1. Follow the instructions below:

git clone [email protected]:kubeshop/tracetest.gitcd examples/observability-driven-development-go-tracetest/bookstore/part2.1docker compose -f docker-compose.yaml -f tracetest/docker-compose.yaml up###tracetest test run -d ./e2e/books-list.yaml -w

Note: View the 2nd part of our observability-driven development video tutorial, here!

To get started lets first add more detailed assertions to the books-list.yaml, and make our test fail.

Open the books-list.yaml file and add a custom attribute called attr:books.list.count. This means we are expecting the Books List API test to return 3 books.

# ./e2e/books-list.yaml# ...  specs:    - selector: span[name="Tracetest trigger"]      assertions:        - attr:tracetest.response.status = 200    - selector: span[name="Books List"]      assertions:        - attr:tracetest.selected_spans.count = 1        - attr:books.list.count = 3

Jump back into the terminal, and run the test again.

tracetest test run -d ./e2e/books-list.yaml -w[Output] Books Listing (http://localhost:11633/test/k6hEWU54R/run/1/test)     span[name="Tracetest trigger"]         #ebae1f382ecb81f6             attr:tracetest.response.status = 200 (200)     span[name="Books List"]         #f6c5fa3aa5527a7a             attr:tracetest.selected_spans.count = 1 (1)             attr:books.list.count = 3 (http://localhost:11633/test/k6hEWU54R/run/1/test?selectedAssertion=1&selectedSpan=f6c5fa3aa5527a7a)

The "Books List" span now fails the test.

In true ODD fashion, lets add the code to satisfy the test spec.

We need to add a getBooks function to retrieve the books and make sure to add OpenTelemetry instrumentation to validate that it is indeed returning the books in the form of an array.

Open up the ./books/main.go file. We will edit the booksListHandler function, and add a getBooks function that simulates getting books from a database.

// ./books/main.go// ...func booksListHandler(w http.ResponseWriter, r *http.Request) {    ctx, span := tracer.Start(r.Context(), "Books List")    defer span.End()    books, err := getBooks(ctx)    if err != nil {        w.WriteHeader(http.StatusInternalServerError)        io.WriteString(w, "cannot read books DB")        return    }  // This is how we instrument the code with OpenTelemetry  // This is the attribute we run the assertion against    span.SetAttributes(        attribute.Int("books.list.count", len(books)),    )    jsonBooks, err := json.Marshal(books)    if err != nil {        w.WriteHeader(http.StatusInternalServerError)        io.WriteString(w, "cannot json encode books DB")        return    }    w.Write(jsonBooks)}type book struct {    ID    string `json:"id"`    Name  string `json:"name"`    Price int    `json:"price"`}// Mocking a database requestfunc getBooks(ctx context.Context) ([]book, error) {    return []book{        {"1", "Harry Potter", 0},        {"2", "Foundation", 0},        {"3", "Moby Dick", 0},    }, nil}

Save the changes, and restart Docker Compose. Now, run the same test.

tracetest test run -d ./e2e/books-list.yaml -w[Output] Books Listing (http://localhost:11633/test/k6hEWU54R/run/1/test)

The test passes. Clicking the link in the test will open up the Tracetest Web UI.

Were starting to look like ODD pros right now! But, were not done yet. We want to add an availability check to our bookstore. What if a book is not in stock? We need to be able to check that.

Setting up observability-driven tests for multiple microservices

To follow along, you can check out the sample code weve prepared for part 2.2. Follow the instructions below:

git clone [email protected]:kubeshop/tracetest.gitcd examples/observability-driven-development-go-tracetest/bookstore/part2.2docker compose -f docker-compose.yaml -f tracetest/docker-compose.yaml up###tracetest test run -d ./e2e/books-list.yaml -w

First add another book entry to the getBooks function. Well add an availability check that will confirm it is out of stock.

// ./books/main.go// ...func getBooks(ctx context.Context) ([]book, error) {    return []book{        {"1", "Harry Potter", 0},        {"2", "Foundation", 0},        {"3", "Moby Dick", 0},        {"4", "The art of war", 0}, // Add this book    }, nil}

Once again, open up the books-list.yaml. Lets add assertions for the availability.

# books-list.yaml# ...  specs:    - selector: span[name="Tracetest trigger"]      assertions:        - attr:tracetest.response.status = 200    - selector: span[name="Books List"]      assertions:        - attr:tracetest.selected_spans.count = 1        - attr:books.list.count = 3    # This selector will look for a descendant of the     # "Books List" span called "Availability Check"    - selector: span[name = "Books List"] span[name = "Availability Check"]      assertions:        - attr:tracetest.selected_spans.count = 4

We want to make sure that an availability check is performed for every single book from the getBooks function.

Re-running the test will cause it to fail because of the availability check, as expected.

tracetest test run -d ./e2e/books-list.yaml -w[Output] Books Listing (http://localhost:11633/test/k6hEWU54R/run/2/test)     span[name="Tracetest trigger"]         #b81c6b68711908e1             attr:tracetest.response.status = 200 (200)     span[name="Books List"]         #392fcfe7690310d8             attr:tracetest.selected_spans.count = 1 (1)             attr:books.list.count = 3 (3)     span[name = "Books List"] span[name = "Availability Check"]         #meta             attr:tracetest.selected_spans.count = 4 (0) (http://localhost:11633/test/k6hEWU54R/run/2/test?selectedAssertion=2)

Next, lets write the code to send an HTTP request to an availability microservice.

// ./books/main.go// ...func httpError(span trace.Span, w http.ResponseWriter, msg string, err error) {    w.WriteHeader(http.StatusInternalServerError)    io.WriteString(w, msg)    span.RecordError(err)    span.SetStatus(codes.Error, msg)}func booksListHandler(w http.ResponseWriter, r *http.Request) {    ctx, span := tracer.Start(r.Context(), "Books List")    defer span.End()    books, err := getAvailableBooks(ctx)    if err != nil {        httpError(span, w, "cannot read books DB", err)        return    }    span.SetAttributes(        attribute.Int("books.list.count", len(books)),    )    jsonBooks, err := json.Marshal(books)    if err != nil {        httpError(span, w, "cannot json encode books", err)        return    }    w.Write(jsonBooks)}func getAvailableBooks(ctx context.Context) ([]book, error) {    books, err := getBooks(ctx)    if err != nil {        return nil, err    }    availableBook := make([]book, 0, len(books))    for _, book := range books {        available, err := isBookAvailable(ctx, book.ID)        if err != nil {            return nil, err        }        if !available {            continue        }        availableBook = append(availableBook, book)    }    return availableBook, nil}var httpClient = &http.Client{    Transport: otelhttp.NewTransport(http.DefaultTransport),}func isBookAvailable(ctx context.Context, bookID string) (bool, error) {    ctx, span := tracer.Start(ctx, "Availability Request", trace.WithAttributes(        attribute.String("bookID", bookID),    ))    defer span.End()    url := "http://availability:8000/" + bookID    req, _ := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)    resp, err := httpClient.Do(req)    if err != nil {        span.RecordError(err)        span.SetStatus(codes.Error, "cannot do request")        return false, err    }    if resp.StatusCode == http.StatusNotFound {        span.SetStatus(codes.Error, "not found")        return false, nil    }    stockBytes, err := io.ReadAll(resp.Body)    if err != nil {        span.RecordError(err)        span.SetStatus(codes.Error, "cannot read response body")        return false, err    }    stock, err := strconv.Atoi(string(stockBytes))    if err != nil {        span.RecordError(err)        span.SetStatus(codes.Error, "cannot parse stock value")        return false, err    }    return stock > 0, nil}

Let me explain the code in detail.

Were adding a isBookAvailable function that checks if a book is available based on a provided bookID. It calls the "http://availability:8000/" endpoint and appends a bookID value.
The isBookAvailable function is then used in the getAvailableBooks function that iterates through the books from the getBooks function.
The booksListHandler function now calls the getAvailableBooks function instead of calling getBooks.
The httpError is just a helper function.

Note: Do not forget to re run go mod tidy if you change code that requires modules to be downloaded. Make sure to also restart Docker Compose after editing code!

Lets re-run the test.

tracetest test run -d ./e2e/books-list.yaml -w[Output] Books Listing (http://localhost:11633/test/qasYcU54R/run/1/test)     span[name="Tracetest trigger"]         #2f9bc366597fb472             attr:tracetest.response.status = 200 (500) (http://localhost:11633/test/qasYcU54R/run/1/test?selectedAssertion=0&selectedSpan=2f9bc366597fb472)     span[name="Books List"]         #1f0e9347869fd8c2             attr:tracetest.selected_spans.count = 1 (1)             attr:books.list.count = 3 (http://localhost:11633/test/qasYcU54R/run/1/test?selectedAssertion=1&selectedSpan=1f0e9347869fd8c2)     span[name = "Books List"] span[name = "Availability Check"]         #meta             attr:tracetest.selected_spans.count = 4 (0) (http://localhost:11633/test/qasYcU54R/run/1/test?selectedAssertion=2)

Were getting a different error now! The response status of the "Tracetest trigger" span equals 500. Hmm, not great, right?

Wrong! Were on the correct path! The test is failing because we added code that sends an HTTP request to an availability service that does not exist. Lets fix that.

Next up, creating an availability service.

Trace-based testing across multiple services

To follow along, you can check out the sample code weve prepared for part 3.1. Follow the instructions below:

git clone [email protected]:kubeshop/tracetest.gitcd examples/observability-driven-development-go-tracetest/bookstore/part3.1docker compose -f docker-compose.yaml -f tracetest/docker-compose.yaml up###tracetest test run -d ./e2e/books-list.yaml -w

Note: View the 3rd part of our observability-driven development video tutorial, here!

When developing distributed apps and microservices its best practice to extract the OpenTelemetry instrumentation to a dedicated module.

This will let you import the OpenTelemetry configuration into all of your microservices without duplicating code.

Lets start with pulling out the OpenTelemetry SDKs from the ./books/main.go and putting them in a dedicated file called instrumentation.go.

Create a lib directory in the root of the bookstore directory.

Initialize a module with:

cd ./libgo mod init github.com/your-username/bookstore/lib

Note: File paths can be tricky in Go. Makes sure the name of the file path matches the location on GitHub when you import the module in your microservices.

Once you create the Go module, create another directory called instrumentation. Add a single file called instrumentation.go.

Remove the OpenTelemetry instrumentation code from ./books/main.go, and add i to the ./lib/instrumentation/instrumentation.go.

// ./lib/instrumentation/instrumentation.go package instrumentationimport (    "context"    "fmt"    "time"    "go.opentelemetry.io/otel"    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"    "go.opentelemetry.io/otel/propagation"    "go.opentelemetry.io/otel/sdk/resource"    sdktrace "go.opentelemetry.io/otel/sdk/trace"    semconv "go.opentelemetry.io/otel/semconv/v1.12.0"    "google.golang.org/grpc"    "google.golang.org/grpc/credentials/insecure")func NewExporter(ctx context.Context) (sdktrace.SpanExporter, error) {    ctx, cancel := context.WithTimeout(ctx, time.Second)    defer cancel()    conn, err := grpc.DialContext(ctx, "otel-collector:4317", grpc.WithTransportCredentials(insecure.NewCredentials()), grpc.WithBlock())    if err != nil {        return nil, fmt.Errorf("failed to create gRPC connection to collector: %w", err)    }    traceExporter, err := otlptracegrpc.New(ctx, otlptracegrpc.WithGRPCConn(conn))    if err != nil {        return nil, fmt.Errorf("failed to create trace exporter: %w", err)    }    return traceExporter, nil}func NewTraceProvider(svcName string, exp sdktrace.SpanExporter) *sdktrace.TracerProvider {    // Ensure default SDK resources and the required service name are set.    r, err := resource.Merge(        resource.Default(),        resource.NewWithAttributes(            semconv.SchemaURL,            semconv.ServiceNameKey.String(svcName),        ),    )    if err != nil {        panic(err)    }    tp := sdktrace.NewTracerProvider(        sdktrace.WithBatcher(exp),        sdktrace.WithResource(r),    )    otel.SetTextMapPropagator(        propagation.NewCompositeTextMapPropagator(            propagation.TraceContext{},            propagation.Baggage{},        ),    )    return tp}

Dont forget to run go mod tidy in the terminal from the ./lib folder to make sure the dependencies are downloaded and saved. You can now safely commit and push this code to GitHub. This will let you download it and use it in both the books and availability microservices. Lets move on to updating the books service first.

// ./books/main.gopackage mainimport (    "context"    "encoding/json"    "io"    "log"    "net/http"    "strconv"    "github.com/gorilla/mux"    // Add the instrumentation module from lib    // Make sure to first push the module to GitHub    // Watch out to get the directory tree and name to match    "github.com/your-username/bookstore/lib/instrumentation"    "go.opentelemetry.io/contrib/instrumentation/github.com/gorilla/mux/otelmux"    "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"    "go.opentelemetry.io/otel"    "go.opentelemetry.io/otel/attribute"    "go.opentelemetry.io/otel/codes"    "go.opentelemetry.io/otel/trace")const svcName = "books"var tracer trace.Tracerfunc main() {    ctx := context.Background()    // Calling the "instrumentation" module    exp, err := instrumentation.NewExporter(ctx)    if err != nil {        log.Fatalf("failed to initialize exporter: %v", err)    }    // Calling the "instrumentation" module    // Create a new tracer provider with a batch span processor and the given exporter.    tp := instrumentation.NewTraceProvider(svcName, exp)    // Handle shutdown properly so nothing leaks.    defer func() { _ = tp.Shutdown(ctx) }()    otel.SetTracerProvider(tp)    // Finally, set the tracer that can be used for this package.    tracer = tp.Tracer(svcName)    r := mux.NewRouter()    r.Use(otelmux.Middleware(svcName))    r.HandleFunc("/books", booksListHandler)    http.Handle("/", r)    log.Fatal(http.ListenAndServe(":8001", nil))}// ...

The ./books/main.go looks exactly the same except for removing the OpenTelemetry instrumentation code and importing the ./lib/instrumentation module instead. Make sure to edit the import to use the instrumentation module you pushed to GitHub.

Then, we use the instrumentation object to call the NewExporter and NewTraceProvider methods.

To make sure the behavior is the same after this change lets restart Docker Compose and re-run the same test as above.

tracetest test run -d ./e2e/books-list.yaml -w[Output] Books Listing (http://localhost:11633/test/qasYcU54R/run/1/test)     span[name="Tracetest trigger"]         #831e781a89050f81             attr:tracetest.response.status = 200 (500) (http://localhost:11633/test/qasYcU54R/run/1/test?selectedAssertion=0&selectedSpan=831e781a89050f81)     span[name="Books List"]         #9f05d0fe6d4966e6             attr:tracetest.selected_spans.count = 1 (1)             attr:books.list.count = 3 (http://localhost:11633/test/qasYcU54R/run/1/test?selectedAssertion=1&selectedSpan=9f05d0fe6d4966e6)     span[name = "Books List"] span[name = "Availability Check"]         #meta             attr:tracetest.selected_spans.count = 4 (0) (http://localhost:11633/test/qasYcU54R/run/1/test?selectedAssertion=2)

Awesome! Were getting the same issue as before. Wild, isnt it? Im cheering because were getting the same issue as before!

With that out of the way, time to build our availability service.

Hands-on observability-driven tests across multiple microservices

Check out the sample code weve prepared for part 3.2 to follow along. Follow the instructions below:

git clone [email protected]:kubeshop/tracetest.gitcd examples/observability-driven-development-go-tracetest/bookstore/part3.2docker compose -f docker-compose.yaml -f tracetest/docker-compose.yaml up###tracetest test run -d ./e2e/books-list.yaml -w

To begin, we need a new directory for our additional microservice. Create an availability directory in the bookstore directory. Initialize a Go module.

cd ./availabilitygo mod init github.com/your-username/bookstore/availability

Create a file called ./availability/main.go. Paste this code into it.

package mainimport (    "context"    "io"    "log"    "net/http"    // Make sure this module matches the lib/instrumentation    // module from the previous section!    "github.com/your-username/bookstore/lib/instrumentation"    "github.com/gorilla/mux"    "go.opentelemetry.io/contrib/instrumentation/github.com/gorilla/mux/otelmux"    "go.opentelemetry.io/otel"    "go.opentelemetry.io/otel/attribute"    "go.opentelemetry.io/otel/codes"    "go.opentelemetry.io/otel/trace")const svcName = "availability"var tracer trace.Tracerfunc main() {    ctx := context.Background()    exp, err := instrumentation.NewExporter(ctx)    if err != nil {        log.Fatalf("failed to initialize exporter: %v", err)    }    // Create a new tracer provider with a batch span processor and the given exporter.    tp := instrumentation.NewTraceProvider(svcName, exp)    // Handle shutdown properly so nothing leaks.    defer func() { _ = tp.Shutdown(ctx) }()    otel.SetTracerProvider(tp)    // Finally, set the tracer that can be used for this package.    tracer = tp.Tracer(svcName)    r := mux.NewRouter()    r.Use(otelmux.Middleware(svcName))    r.HandleFunc("/{bookID}", stockHandler)    http.Handle("/", r)    log.Fatal(http.ListenAndServe(":8000", nil))}var books = map[string]string{    "1": "10",    "2": "1",    "3": "5",    "4": "0",}func stockHandler(w http.ResponseWriter, r *http.Request) {    _, span := tracer.Start(r.Context(), "Availability Check")    defer span.End()    vars := mux.Vars(r)    bookID, ok := vars["bookID"]    if !ok {        span.SetStatus(codes.Error, "no bookID in URL")        w.WriteHeader(http.StatusBadRequest)        io.WriteString(w, "missing bookID in URL")        return    }    // The span we will run an assertion against    span.SetAttributes(        attribute.String("bookID", bookID),    )    stock, ok := books[bookID]    if !ok {        span.SetStatus(codes.Error, "book not found")        w.WriteHeader(http.StatusNotFound)        io.WriteString(w, "book not found")        return    }    w.WriteHeader(http.StatusOK)    io.WriteString(w, stock)}

As always, run go mod tidy to generate a go.sum file and download modules.

Let me explain the code:

We are using the NewExporter and NewTraceProvider in the main function just as we did in the ./books/main.go.
We are running an HTTP server on port 8000 that expects a bookID as a parameter.
The HTTP route /{bookID} will trigger a stockHandler function. This function checks if the book is in stock or not.

Wonderful! With the availability service added, we need to add it to the docker-compose.yaml as well.

services:  books:    image: your_username/books    build:      args:        SERVICE: books    ports:      - 8001:8001    depends_on:      - otel-collector  availability:    image: your_username/availability    build:      args:        SERVICE: availability    depends_on:      - otel-collector

The availability service will use the same Dockerfile as the books service.

Thats all! Were done! Lets restart Docker Compose and see if the "Books Listing" test passes.

For reference, heres the full ./e2e/books-list.yaml test file were running:

# ./e2e/books-list.yamltype: Testspec:  id: qasYcU54R  name: Books Listing  description: Try books service  trigger:    type: http    httpRequest:      url: http://books:8001/books      method: GET      headers:      - key: Content-Type        value: application/json  specs:    - selector: span[name="Tracetest trigger"]      assertions:        - attr:tracetest.response.status = 200    - selector: span[name="Books List"]      assertions:        - attr:tracetest.selected_spans.count = 1        - attr:books.list.count = 3    - selector: span[name = "Books List"] span[name = "Availability Check"]      assertions:        - attr:tracetest.selected_spans.count = 4

In your terminal, run:

tracetest test run -d ./e2e/books-list.yaml -w[Output] Books Listing (http://localhost:11633/test/qasYcU54R/run/1/test)

Clicking the link will open the Tracetest Web UI and show the assertions in detail.

We can clearly see how an availability check was triggered four times. One time for every book in the list.

Conclusion

Youve learned how to practice observability-driven development with Go and Tracetest across multiple microservices by using a dedicated OpenTelemetry instrumentation module. Well done!

To continue learning about observability-driven development, check out our 3-part video tutorial about observability-driven development with Go and Tracetest!

Give Tracetest a try in your own applications and tracing infrastructure by following either our quick start guide, which sets you up with the CLI tooling and the Tracetest server in a few steps, or our detailed guide, for more details. Feel free to give us a star on GitHub as well!

By practicing observability-driven development and trace-based testing best practices we want you to have a more developer friendly experience by increasing your test coverage, freeing yourself from manual testing procedures, and identifying bottlenecks you didnt even know existed.

Wed love to hear about your ODD success stories in Discord! We truly value your feedback, so don't be shy!

Original Link: https://dev.to/kubeshop/observability-driven-development-with-go-and-tracetest-88d

Share this article:

View Full Article

Dev To

An online community for sharing and discovering great ideas, having debates, and making friends

More About this Source Visit Dev To