LangSmith Essentials
> Once an LLM application has more than one step — retrieval, tool calls, chained prompts — debugging it from logs alone stops working. LangSmith is an observability and evaluation platform (from the LangChain team) built specifically for tracing, testing, and monitoring LLM applications. This notebook covers what it's for and how it fits into a real workflow.
What is LangSmith?
LangSmith gives you visibility into what actually happens inside an LLM application: every prompt sent, every model response, every tool call, and every intermediate step in a chain or agent — captured as a structured trace you can inspect after the fact.
It's framework-agnostic in principle (you can instrument raw API calls), but it integrates most seamlessly with LangChain and LangGraph, auto-capturing traces for any chain or graph built with them.
Why plain logging isn't enough:
Core Capabilities
1. Tracing Every run of a chain, agent, or raw LLM call is captured as a trace — a tree of spans showing inputs, outputs, latency, and token usage at each step. For a multi-step agent, this means you can see exactly which tool was called, with what arguments, and what it returned, at every hop.
2. Evaluation Define test datasets (input/expected-output pairs) and run them against your application to measure quality — using exact match, an LLM-as-judge, or custom scoring functions. This turns "does this prompt change make things better or worse?" into a measurable question instead of a vibe check.
3. Monitoring In production, track latency, error rate, token cost, and feedback scores over time, and set up alerts when a metric drifts — catching regressions from a model version change or a prompt edit before users report them.
4. Prompt/Dataset Management Version prompts, collect examples where the model got something wrong, and turn them into regression test cases — closing the loop between "we found a bug" and "we have a test that prevents it recurring."
Practical Example: Tracing + Evaluation
Enable tracing (works automatically for LangChain/LangGraph code once configured):
import os
os.environ["LANGCHAIN_TRACING_V2"] = "true"
os.environ["LANGCHAIN_API_KEY"] = "..."
os.environ["LANGCHAIN_PROJECT"] = "support-bot"
# any LangChain/LangGraph call below is now automatically traced
response = chain.invoke({"question": "How do I reset my password?"})
Run an evaluation against a labeled dataset:
from langsmith import Client
from langsmith.evaluation import evaluate
client = Client()
def correctness(run, example) -> dict:
predicted = run.outputs["answer"]
expected = example.outputs["answer"]
score = llm_judge(predicted, expected) # e.g. an LLM-as-judge call
return {"key": "correctness", "score": score}
results = evaluate(
lambda inputs: chain.invoke(inputs),
data="support-bot-eval-set", # a dataset created in LangSmith
evaluators=[correctness],
)
This produces a report comparing outputs against expected answers, which is what makes it possible to say "this prompt change improved accuracy from 82% to 89%" instead of "it feels better."