Advanced Production Observability and Evaluation

Relax. We'll talk through this in plain words — no textbook voice.

What you'll walk away with

Understand why tracing matters
Build evaluation cases
Check for regressions

Let's break this down simply

You can't trust an LLM app just by looking at one output. Trace each prompt, model, and tool call step by step, and check things like expected tool sequence, grounded answers, and latency against a dataset. Strip out sensitive data before it ever goes into metadata.

python

test_cases = [
    {
        "input": "ရန်ကုန်အတွက် မိုးလေဝသကိုရှာပါ",
        "expected_tool": "get_weather",
        "must_include": "ရန်ကုန်",
    }
]

for case in test_cases:
    result = agent.invoke({
        "messages": [{"role": "user", "content": case["input"]}]
    })
    assert result["messages"], "Agent returned no messages"
    print("Evaluation case completed")

You should see

Evaluation case completed

Try it yourself

Write at least 10 evaluation cases for your agent, covering normal, ambiguous, and unsafe inputs.

LangSmith Observability — LangChain

Easy traps

Testing only the happy path
Putting sensitive user data into trace metadata
Changing a prompt without running a regression evaluation

Exercise

Write at least 10 evaluation cases for your agent, covering normal, ambiguous, and unsafe inputs.

You'll know it worked when: Evaluation case completed