All Products
Search

This Product

    :Trace fields for LLM applications

    Document Center

    :Trace fields for LLM applications

    Last Updated:Nov 18, 2024

    Trace fields for Large Language Model (LLM) applications are developed by Alibaba Cloud based on the OpenTelemetry standard and characteristics of LLM applications. These fields are used to describe trace data of LLM applications by expanding the attributes, resources, and events in OpenTelemetry. These fields can be used to describe key spans of LLM applications, including input, output, and token usage. Trace fields for LLM applications provide a variety of context-related semantic data for multiple scenarios such as completion, chat, retrieval-augmented generation (RAG), agent, and tool. This facilitates data tracking and reporting. These trace fields will be constantly updated and optimized as the Managed Service for OpenTelemetry community grows.

    Span fields

    For more information about the level-1 span fields and the underlying level-1 trace fields stored in Managed Service for OpenTelemetry, see Trace Explorer parameters.

    Attributes

    The fields described in this section are reserved fields used in LLM-related scenarios. If you use Managed Service for OpenTelemetry SDKs to report span data to servers, you must add relevant information as attributes in the form of key-value pairs to spans.

    Common attributes

    Note

    A common attribute can be passed through or added to multiple spans in a trace for an LLM-based conversational search regardless of the span kind.

    Attribute

    Type

    Description

    Required

    Example

    gen_ai.session.id

    String

    The session ID.

    No

    "ddde34343-f93a-4477-33333-sdfsdaf"

    gen_ai.user.id

    String

    The user ID on the application client.

    No

    "u-lK8JddD"

    gen_ai.span.kind

    String

    The span kind.

    For more information, see the Span kinds section of this topic.

    Yes

    "CHAIN"

    gen_ai.framework

    String

    The framework type.

    No

    "langchain" or "llama_index"

    Span kinds

    Multiple spans are often involved in actual LLM-based interaction scenarios. The following span kinds are defined by Managed Service for OpenTelemetry based on the LLM application paradigm and research and development (R&D) framework. You can expand these span kinds. The attributes vary based on the span kind.

    • CHAIN: connects LLM and other components to implement complex tasks. A CHAIN span may contain RETRIEVER, EMBEDDING, LLM, and other CHAIN spans.

    • EMBEDDING: performs embedding operations for LLM applications such as word embedding models to implement similarity-based queries and question optimization.

    • RETRIEVER: retrieves data from vector storage or databases to supplement the context. This allows LLMs to provide answers in a more accurate and efficient way.

    • RERANKER: sorts multiple input documents based on the relevance of the question. This span may return top K documents for LLMs.

    • LLM: invokes LLMs. For example, you can invoke different LLMs to perform model inference or text generation by using Managed Service for OpenTelemetry SDKs or API operations.

    • TOOL: uses external tools. For example, you can use a calculator or call a weather operation to obtain the latest weather conditions.

    • AGENT: a complex CHAIN span used in intelligence scenarios. An AGENT span represents the consecutive steps performed based on the inference results of LLMs. For example, multiple LLM spans and TOOL spans may be contained in an AGENT span in sequence to produce final answers.

    • TASK: specifies internal custom methods, such as calling a local function to apply custom logic.

    Important

    The LLM-based span kinds defined by Managed Service for OpenTelemetry are different from the span kinds defined by OpenTelemetry.

    CHAIN

    Attribute

    Type

    Description

    Required

    Example

    gen_ai.span.kind

    String

    The span kind.

    Yes

    "CHAIN"

    gen_ai.span.sub_kind

    String

    The level-2 span kind.

    No

    "WORKFLOW" or "TASK"

    input.value

    String

    The input data.

    Yes

    "Who Are You!"

    output.value

    String

    The output data.

    Yes

    "I am ChatBot"

    EMBEDDING

    Attribute

    Type

    Description

    Required

    Example

    gen_ai.span.kind

    String

    The span kind.

    Yes

    "EMBEDDING"

    embedding.model_name

    String

    The name of the model for embedding.

    No

    "text-embedding-v1"

    gen_ai.usage.prompt_tokens

    Integer

    The number of tokens used for embedding.

    No

    10

    gen_ai.usage.total_tokens

    Integer

    The total number of tokens used for embedding.

    No

    10

    embedding.embeddings.0.embedding.text

    String

    The text data that is converted into embeddings.

    No

    "hello world"

    embedding.embeddings.0.embedding.vector

    Array

    The embedding vector that consists of floating-point numbers.

    No

    [0.123, 0.456, ...]

    embedding.embeddings.0.embedding.vector_size

    Integer

    The vector length.

    No

    1536

    RETRIEVER

    Attribute

    Type

    Description

    Required

    Example

    gen_ai.span.kind

    String

    The span kind.

    Yes

    "RETRIEVER"

    retrieval.documents.0.document.id

    String

    The unique ID of the document.

    Yes

    "2aeab544-f93a-4477-b51d-bec27351325b"

    retrieval.documents.0.document.score

    Float

    The relevance score of the document.

    Yes

    0.98

    retrieval.documents.0.document.content

    String

    The content of the retrieved document.

    Yes

    "This is a sample document content."

    retrieval.documents.0.document.metadata

    String

    The metadata related to the document.

    No

    {"file_path": "/dev/EasyRAG/data/laws/laws.txt",

    "file_name": "/dev/EasyRAG/data/laws/laws.txt", "file_type": "text/plain",

    "file_size": 15618, "creation_date": "2024-03-20", "last_modified_date": "2024-03-20",

    "last_accessed_date": null}

    RERANKER

    Attribute

    Type

    Description

    Required

    Example

    gen_ai.span.kind

    String

    The span kind.

    Yes

    "RERANKER"

    reranker.query

    String

    The query of the reranker.

    No

    "How to format timestamp?"

    reranker.model_name

    String

    The name of the model that the reranker uses.

    No

    "cross-encoder/ms-marco-MiniLM-L-12-v2"

    reranker.top_k

    Integer

    The number of results that the reranker returns.

    No

    3

    reranker.input_documents.0.document.id

    String

    The unique ID of the document.

    Yes

    "2aeab544-f93a-4477-b51d-bec27351325b"

    reranker.input_documents.0.document.score

    Float

    The relevance score of the document.

    Yes

    0.98

    reranker.input_documents.0.document.content

    String

    The content of the retrieved document.

    Yes

    "This is a sample document content."

    reranker.input_documents.0.document.metadata

    String

    The metadata related to the document.

    No

    {"file_path": "/dev/EasyRAG/data/laws/laws.txt",

    "file_name": "/dev/EasyRAG/data/laws/laws.txt", "file_type": "text/plain",

    "file_size": 15618, "creation_date": "2024-03-20", "last_modified_date": "2024-03-20","last_accessed_date": null}

    reranker.output_documents.0.document.id

    String

    The unique ID of the document.

    Yes

    "2aeab544-f93a-4477-b51d-bec27351325b"

    reranker.output_documents.0.document.score

    Float

    The relevance score of the document.

    Yes

    0.98

    reranker.output_documents.0.document.content

    String

    The content of the retrieved document.

    Yes

    "This is a sample document content."

    reranker.output_documents.0.document.metadata

    String

    The metadata related to the document.

    No

    {"file_path": "/dev/EasyRAG/data/laws/laws.txt",

    "file_name": "/dev/EasyRAG/data/laws/laws.txt", "file_type": "text/plain",

    "file_size": 15618, "creation_date": "2024-03-20", "last_modified_date": "2024-03-20","last_accessed_date": null}

    LLM

    Attribute

    Type

    Description

    Required

    Example

    gen_ai.span.kind

    String

    The span kind.

    Yes

    "LLM"

    gen_ai.span.sub_kind

    String

    The level-2 span kind.

    No

    "CHAT" or "COMPLETION"

    gen_ai.prompt_template.template

    String

    The template that is used to generate prompts as Python f-strings.

    No

    "Weather forecast for {city} on {date}"

    gen_ai.prompt_template.variables

    String

    The key-value pairs in the JSON format that apply to the specified prompt template.

    No

    { context: "<context from retrieval>", subject: "math" }

    gen_ai.prompt_template.version

    String

    The version number of the prompt template.

    No

    "1.0"

    gen_ai.system

    String

    The LLM provider.

    Yes

    "OPENAI"

    gen_ai.request.parameters

    String

    The parameters that are used to invoke the LLM or call the API operation.

    Yes

    {"temperature": 0.7}"

    gen_ai.model_name

    String

    The name of the LLM.

    Yes

    "gpt-4"

    gen_ai.request.model

    String

    The name of the LLM to which the request is sent.

    Yes

    "gpt-4"

    gen_ai.request.max_tokens

    Integer

    The maximum number of tokens that the LLM generates for a request.

    No

    100

    gen_ai.request.temperature

    Float

    The temperature setting for the LLM request.

    No

    0.1

    gen_ai.request.top_p

    Float

    The top-p sampling setting for the LLM request.

    No

    1

    gen_ai.request.is_stream

    Boolean

    Specifies whether the LLM generates a response as a stream.

    No

    false

    gen_ai.request.stop_sequences

    Array

    The string array that the LLM uses as the stop sequence.

    No

    "stop"

    gen_ai.request.tool_calls

    Array

    The list of tool calls, such as function invocations, generated by the LLM.

    No

    [{"tool_call.function.name": "get_current_weather"}]

    gen_ai.response.model

    String

    The name of the LLM that generates the response.

    No

    "gpt-4-0613"

    gen_ai.response.finish_reason

    String

    The reason why the model stops generating tokens. Each reason corresponds to a generation process.

    No

    "stop"

    gen_ai.prompts.0.content

    String

    The prompt content.

    Yes

    "What is the capital city of China?"

    gen_ai.prompts.0.message.role

    String

    The role of the entity in the message, such as user or system.

    Yes

    "user" or "system"

    gen_ai.prompts.0.message.content

    String

    The message content in a chat.

    Yes

    "What's the weather today?"

    input.value

    String

    The parameters of the API request. Generally, the value is in the JSON format.

    Yes

    "Who Are You!"

    gen_ai.completions.0.content

    String

    The full response returned by the LLM.

    Yes

    "The capital city of China is Beijing."

    gen_ai.completions.0.message.role

    String

    The role of the entity in the message during a chat, such as assistant.

    Yes

    "assistant"

    gen_ai.completions.0.message.content

    String

    The content of the message in a chat.

    Yes

    "Chat content 1"

    gen_ai.completions.0.message.tool_calls

    Array

    The list of tool calls, such as function invocations, generated by the LLM in a chat.

    No

    [{"tool_call.function.name": "get_current_weather"}]

    output.value

    String

    The complete content returned for the API request. Generally, the value is in the JSON format. You can extract key fields, such as token_usage, from the returned content.

    Yes

    "I am ChatBot"

    gen_ai.usage.prompt_tokens

    Integer

    The number of tokens in the prompt.

    Yes

    100

    gen_ai.usage.completion_tokens

    Integer

    The number of tokens in the completion.

    Yes

    200

    gen_ai.usage.total_tokens

    Integer

    The total number of tokens, including those in the prompt and completion.

    Yes

    300

    TOOL

    Attribute

    Type

    Description

    Required

    Example

    gen_ai.span.kind

    String

    The span kind.

    Yes

    "TOOL"

    tool.name

    String

    The tool name.

    Yes

    "WeatherAPI"

    tool.description

    String

    The tool description.

    Yes

    "An API to get weather data."

    tool.parameters

    String

    The tool parameters.

    Yes

    "{'a': 'int' }"

    AGENT

    Attribute

    Type

    Description

    Required

    Example

    gen_ai.span.kind

    String

    The span kind.

    Yes

    "AGENT"

    input.value

    String

    The input data.

    Yes

    "Please help me plan xxxx!"

    input.mime_type

    String

    The Multipurpose Internet Mail Extensions (MIME) type of the input data.

    No

    "text/plain" or "application/json"

    output.value

    String

    The output data.

    Yes

    "The planning is complete. Please check the result xxx."

    output.mime_type

    String

    The MIME type of the output data.

    No

    "text/plain" or "application/json"

    TASK

    Attribute

    Type

    Description

    Required

    Example

    gen_ai.span.kind

    String

    The span kind.

    Yes

    "TASK"

    input.value

    String

    The input data.

    No

    "Input parameter in the JSON format"

    input.mime_type

    String

    The MIME type of the input data.

    No

    "text/plain" or "application/json"

    output.value

    String

    The output data.

    No

    "Output parameter in the JSON format"

    output.mime_type

    String

    The MIME type of the output data.

    No

    "text/plain" or "application/json"