Application Real-Time Monitoring Service:Use Managed Service for OpenTelemetry to filter out specific spans

Java application

You can use Managed Service for OpenTelemetry samplers to filter out spans. If you do not want to modify the application code, or you are using Managed Service for OpenTelemetry Java Agent to automatically report application data, you can create an extension for the agent. You can write a custom OpenTelemetry sampler and define filter rules. This way, you can filter out specific spans without the need to modify the application code. For more information, see Method 1. If you are using Managed Service for OpenTelemetry SDK for Java to manually report application data, you can write a custom sampler in the application code. For more information, see Method 2.

Method 1: Create an extension for Managed Service for OpenTelemetry Java Agent

Prerequisites

The application is automatically instrumented by using OpenTelemetry Java Agent. For more information, see Use OpenTelemetry to submit the trace data of Java applications.

1. Create a project.

   Create a Maven project. This project is used to create and build the extension.

2. Add dependencies to the pom.xml file.

   Important

   To ensure that the dependencies are compatible, make sure that all dependencies related to Managed Service for OpenTelemetry are of the same version as the version of OpenTelemetry Java Agent that you use.

    <dependency>
    <groupId>com.google.auto.service</groupId>
    <artifactId>auto-service</artifactId>
    <version>1.1.1</version>
    </dependency>
    
    <dependency>
    <groupId>io.opentelemetry.javaagent</groupId>
    <artifactId>opentelemetry-javaagent</artifactId>
    <version>1.28.0</version>
    <!--Set the value of the scope field to compile.-->
    <scope>compile</scope>
    </dependency>
    
    <dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-sdk-trace</artifactId>
    <version>1.28.0</version>
    </dependency>
    
    <dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-sdk-extension-autoconfigure</artifactId>
    <version>1.28.0</version>
    </dependency>
    
    <dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-semconv</artifactId>
    <version>1.28.0-alpha</version>
    </dependency>

3. Create the SpanFilterSampler class. You can also use a custom class name.

   The class must implement the io.opentelemetry.sdk.trace.samplers.Sampler interface and the shouldSample and getDescription methods.

   • shouldSample

     You can define custom filtering rules in the method. For spans that you want to filter out, SamplingResult.create(SamplingDecision.DROP) is returned. For spans that you want to retain and report, SamplingResult.create(SamplingDecision.RECORD_AND_SAMPLE) is returned.

   • getDescription

     This method returns the name of the custom sampler.

   In the following example, the SpanFilterSampler sampler filters out spans whose name is spanName1 or spanName2 and spans whose attributes.http.target is /api/checkHealth or /health/checks.

    package org.example;
    
    import io.opentelemetry.api.common.Attributes;
    import io.opentelemetry.api.trace.SpanKind;
    import io.opentelemetry.context.Context;
    import io.opentelemetry.sdk.trace.data.LinkData;
    import io.opentelemetry.sdk.trace.samplers.Sampler;
    import io.opentelemetry.sdk.trace.samplers.SamplingDecision;
    import io.opentelemetry.sdk.trace.samplers.SamplingResult;
    import io.opentelemetry.semconv.trace.attributes.SemanticAttributes;
    
    import java.util.*;
    
    public class SpanFilterSampler implements Sampler {
    /*
    * Filter out spans whose name is in EXCLUDED_SPAN_NAMES.
    */
    private static List<String> EXCLUDED_SPAN_NAMES = Collections.unmodifiableList(
    Arrays.asList("spanName1", "spanName2")
    );
    
    /*
    * Filter out spans whose attributes.http.target is in EXCLUDED_HTTP_REQUEST_TARGETS.
    */
    private static List<String> EXCLUDED_HTTP_REQUEST_TARGETS = Collections.unmodifiableList(
    Arrays.asList("/api/checkHealth", "/health/checks")
    );
    
    @Override
    public SamplingResult shouldSample(Context parentContext, String traceId, String name, SpanKind spanKind, Attributes attributes, List<LinkData> parentLinks) {
    String httpTarget = attributes.get(SemanticAttributes.HTTP_TARGET) != null ? attributes.get(SemanticAttributes.HTTP_TARGET) : "";
    
    if (EXCLUDED_SPAN_NAMES.contains(name) || EXCLUDED_HTTP_REQUEST_TARGETS.contains(httpTarget)) { // Filter out spans based on the rules.
    return SamplingResult.create(SamplingDecision.DROP);
    } else {
    return SamplingResult.create(SamplingDecision.RECORD_AND_SAMPLE);
    }
    }
    
    @Override
    public String getDescription() {
    return "SpanFilterSampler"; // You can replace SpanFilterSampler with a custom name.
    }
    }

4. Create the SpanFilterSamplerProvider class. You can also use a custom class name.

   The class must implement the io.opentelemetry.sdk.autoconfigure.spi.traces.ConfigurableSamplerProvider interface and the createSampler and getName methods.

   • createSampler

     This method creates and returns the custom sampler instance.

   • getName

     This method returns the name of the custom sampler. OpenTelemetry Java Agent finds this sampler by name.

    package org.example;
    
    import com.google.auto.service.AutoService;
    import io.opentelemetry.sdk.autoconfigure.spi.ConfigProperties;
    import io.opentelemetry.sdk.autoconfigure.spi.traces.ConfigurableSamplerProvider;
    import io.opentelemetry.sdk.trace.samplers.Sampler;
    
    @AutoService(ConfigurableSamplerProvider.class)
    public class SpanFilterSamplerProvider implements ConfigurableSamplerProvider {
    @Override
    public Sampler createSampler(ConfigProperties configProperties) {
    return new SpanFilterSampler();
    }
    
    @Override
    public String getName() {
    return "SpanFilterSampler"; // You can replace SpanFilterSampler with a custom name.
    }
    }

5. Build the sampler.

   Package the project into a Java archive (JAR) file and store the JAR file in the target directory.

   mvn clean package

6. Load the extension of OpenTelemetry Java Agent when the application is started.

   • Method 1: Add the otel.traces.sampler parameter to the original VM parameters.

     -Dotel.traces.sampler=<your-sampler-name> // Replace <your-sampler-name> with the custom sampler name, which is the return value of the getName method.

     Show the complete startup command example

      java -javaagent:path/to/opentelemetry-javaagent.jar \
      -Dotel.javaagent.extensions=path/to/opentelemetry-java-agent-extension.jar \
      -Dotel.traces.sampler=<your-sampler-name> \ 
      -Dotel.exporter.otlp.headers=Authentication=<token> \
      -Dotel.exporter.otlp.endpoint=<endpoint> \
      -Dotel.metrics.exporter=none 
      -jar yourapp.jar

   • Method 2: Configure the OTEL_TRACES_SAMPLER environment variable.

     export OTEL_TRACES_SAMPLER="<your-sampler-name>" // Replace <your-sampler-name> with the custom sampler name, which is the return value of the getName method.

     Show the complete startup command example

      export OTEL_JAVAAGENT_EXTENSIONS="path/to/opentelemetry-java-agent-extension.jar"
      export OTEL_TRACES_SAMPLER="<your-sampler-name>"
      export OTEL_EXPORTER_OTLP_HEADERS="Authentication=<token>"
      export OTEL_EXPORTER_OTLP_ENDPOINT="<endpoint>"
      export OTEL_METRICS_EXPORTER="none"
      
      java -javaagent:path/to/opentelemetry-javaagent.jar \
      -jar yourapp.jar

Method 2: Use Managed Service for OpenTelemetry SDK for Java

Prerequisites

The application is manually instrumented by using Managed Service for OpenTelemetry SDK for Java. For more information, see Use OpenTelemetry to submit the trace data of Java applications.

1. Create a custom sampler class in your application code.

   The class must implement the io.opentelemetry.sdk.trace.samplers.Sampler interface and the shouldSample and getDescription methods.

   • shouldSample

     You can define custom filtering rules in the method. For spans that you want to filter out, SamplingResult.create(SamplingDecision.DROP) is returned. For spans that you want to retain and report, SamplingResult.create(SamplingDecision.RECORD_AND_SAMPLE) is returned.

   • getDescription

     This method returns the name of the custom sampler.

   In the following example, the SpanFilterSampler sampler filters out spans whose name is spanName1 or spanName2 and spans whose attributes.http.target is /api/checkHealth or /health/check.

    package org.example;
    
    import io.opentelemetry.api.common.Attributes;
    import io.opentelemetry.api.trace.SpanKind;
    import io.opentelemetry.context.Context;
    import io.opentelemetry.sdk.trace.data.LinkData;
    import io.opentelemetry.sdk.trace.samplers.Sampler;
    import io.opentelemetry.sdk.trace.samplers.SamplingDecision;
    import io.opentelemetry.sdk.trace.samplers.SamplingResult;
    import io.opentelemetry.semconv.trace.attributes.SemanticAttributes;
    
    import java.util.*;
    
    public class SpanFilterSampler implements Sampler {
    /*
    * Filter out spans whose name is in EXCLUDED_SPAN_NAMES.
    */
    private static List<String> EXCLUDED_SPAN_NAMES = Collections.unmodifiableList(
    Arrays.asList("spanName1", "spanName2")
    );
    
    /*
    * Filter out spans whose attributes.http.target is in EXCLUDED_HTTP_REQUEST_TARGETS.
    */
    private static List<String> EXCLUDED_HTTP_REQUEST_TARGETS = Collections.unmodifiableList(
    Arrays.asList("/api/checkHealth", "/health/checks")
    );
    
    @Override
    public SamplingResult shouldSample(Context parentContext, String traceId, String name, SpanKind spanKind, Attributes attributes, List<LinkData> parentLinks) {
    String httpTarget = attributes.get(SemanticAttributes.HTTP_TARGET) != null ? attributes.get(SemanticAttributes.HTTP_TARGET) : "";
    
    if (EXCLUDED_SPAN_NAMES.contains(name) || EXCLUDED_HTTP_REQUEST_TARGETS.contains(httpTarget)) { // Filter out spans based on the rules.
    return SamplingResult.create(SamplingDecision.DROP);
    } else {
    return SamplingResult.create(SamplingDecision.RECORD_AND_SAMPLE);
    }
    }
    
    @Override
    public String getDescription() {
    return "SpanFilterSampler"; // You can replace SpanFilterSampler with a custom name.
    }
    }

2. Set the custom sampler when you create a SdkTracerProvider instance.

   When you create a SdkTracerProvider instance, call the setSampler(new SpanFilterSampler()) method to set the custom sampler.

    ...
    
    
    SdkTracerProvider sdkTracerProvider = SdkTracerProvider.builder()
    . setSampler(new MySampler()) // Add this line.
    .addSpanProcessor(BatchSpanProcessor.builder(OtlpGrpcSpanExporter.builder()
    .setEndpoint("<endpoint>")
    .addHeader("Authentication", "<token>")
    .build()).build())
    .setResource(resource)
    .build();
    
    
    ...

3. Start the application.

Node.js application

Demo: opentelemetry-nodejs-demo.

Prerequisites

The application is instrumented by using the Managed Service for OpenTelemetry API for JavaScript. For more information, see Use OpenTelemetry to submit the trace data of a Node.js application.

Method 1: Filter out spans when the spans are created

1. Configure the ignoreIncomingRequestHook parameter when you construct HttpInstrumentation.

   When you construct HttpInstrumentation, you can configure the HttpInstrumentationConfig parameter, which includes the ignoreIncomingRequestHook parameter. The ignoreIncomingRequestHook parameter allows you to specify a custom method to determine whether to instrument the application before the request is processed. The custom method only determines that a span is not created. The request is processed as expected.

   In the following example, requests whose request.url is /api/checkHealth are ignored for instrumentation.

    ...
    
    // The content to be replaced.
    // registerInstrumentations({
    // tracerProvider: provider,
    // instrumentations: [new HttpInstrumentation(), ExpressInstrumentation],
    // });
    
    const httpInstrumentation = new HttpInstrumentation({
    
    // Specify a custom method in the ignoreIncomingRequestHook parameter.
    ignoreIncomingRequestHook: (request) => {
    // Ignore requests whose request.url is /api/checkHealth.
    if (request.url === '/api/checkHealth') {
    return true;
    }
    return false;
    },
    
    });
    
    registerInstrumentations({
    tracerProvider: provider,
    instrumentations: [httpInstrumentation, ExpressInstrumentation],
    });
    
    ...

2. Start the application.

Method 2: Filter out spans when the spans are reported

1. Create a custom sampler class.

   Create a custom sampler class that implements the Sampler interface. This interface defines the sampling rules. Example:

    const opentelemetry = require('@opentelemetry/api');
    
    class SpanFilterSampler {
    shouldSample(spanContext, parentContext) {
    // Implement your custom sampling logic here.
    }
    }

2. Set the custom sampler when you create a NodeTracerProvider instance.

    ...
    const provider = new NodeTracerProvider({
    sampler: new SpanFilterSampler(), // Add this line of code to set the custom sampler.
    resource: new Resource({
    [SemanticResourceAttributes.HOST_NAME]: require("os").hostname(), // your host name
    [SemanticResourceAttributes.SERVICE_NAME]: "<your-service-name>",
    }),
    });
    ...