Fine-Tuning LLMs for Custom Agent Behaviors - Part 5: Integration & Troubleshooting

A fine-tuned model can look excellent in an evaluation notebook and disappoint the moment it joins a real agent.

Production requests are messier. Tools fail. Context is longer. Output schemas are strict. Users ask questions outside the training distribution. The model is only one component in a loop, so a regression may come from retrieval, prompting, tool design, or orchestration rather than the fine-tune itself.

This final part focuses on integration boundaries and a disciplined troubleshooting process.

TL;DR

Put the fine-tuned model behind a provider-neutral adapter, keep prompts and tool contracts versioned, and validate every structured output. Compare failures against the base model before blaming training. Release through shadow traffic and canaries, preserve an immediate rollback path, and retrain only from reviewed examples with clear provenance.

1. Hide the provider behind an adapter

Your workflow should depend on a stable application interface, not a vendor-specific response object.

type AgentRequest = {
  messages: Array<{ role: "system" | "user" | "assistant"; content: string }>;
  tools?: ToolDefinition[];
  outputSchema?: object;
};

type AgentResult = {
  text?: string;
  toolCalls: ToolCall[];
  usage: { inputTokens: number; outputTokens: number };
  modelVersion: string;
};

interface ModelAdapter {
  run(request: AgentRequest): Promise<AgentResult>;
}

The adapter normalizes authentication, retries, tool-call formats, structured outputs, and usage metadata. Moving from a base model to a fine-tuned model should be a routing change, not a rewrite of Slack, web, or workflow integrations.

2. Keep the surrounding contract stable

A fine-tune does not eliminate the need for clear instructions. Keep a concise system contract that defines:

• task and audience
• allowed tools
• output schema
• refusal and escalation behavior
• facts that must come from retrieval or live tools

Do not bury changing business knowledge in model weights. Fine-tune durable behavior such as classification style, domain terminology, or response structure. Retrieve volatile facts such as prices, policies, inventory, and account state.

Version the model, system instructions, tool schemas, retrieval configuration, and output schema independently. Without those versions in traces, you cannot reproduce a failure.

3. Validate outputs before the workflow trusts them

Fine-tuned models can still omit fields, produce invalid enum values, or call the wrong tool.

Validate at the boundary:

const result = await model.run(request);
const parsed = TicketDecisionSchema.safeParse(result.text);

if (!parsed.success) {
  return {
    status: "needs_review",
    reason: "invalid_output_schema",
    validationErrors: parsed.error.issues,
  };
}

For tool calls, validate both syntax and authorization. A structurally valid refund request may still exceed the user's approval limit.

Choose a safe fallback: retry once with validation feedback, route to the base model, return a partial draft, or escalate to a human. Do not create an unlimited repair loop.

4. Route only the tasks the model was trained for

A fine-tune is usually a specialist. Treat it like one.

Use an explicit router based on task type, language, risk, and input shape. Requests outside the evaluated distribution should go to a general model or human review.

Record the routing reason. If the fine-tuned model performs poorly, you need to know whether the model failed its intended job or the router sent it the wrong job.

5. Diagnose regressions in the right order

When quality drops, resist the urge to retrain immediately.

Check in this order:

1. Input integrity: Was the user request complete and correctly normalized?
2. Retrieval: Were the right documents found, fresh, and in scope?
3. Prompt and context: Did conflicting or irrelevant instructions enter the window?
4. Tools: Did schemas or downstream behavior change?
5. Parsing: Was a correct response rejected or transformed incorrectly?
6. Routing: Was the task appropriate for this model?
7. Model behavior: Does the failure reproduce with the same input and configuration?

Then run the same case against the base model and previous fine-tuned version. This comparison distinguishes a training regression from a system regression.

Common failure patterns

The model follows style but loses facts

The training set may overemphasize polished answers and underrepresent evidence use. Add evaluated examples where the model cites retrieved material, asks for missing facts, or refuses to invent them.

Tool calling gets worse

Training examples may contain outdated tool schemas or too few tool-use trajectories. Keep current schemas in context and evaluate tool selection separately from final prose.

The model is overconfident out of domain

Fine-tuning can make a behavior more consistent without making knowledge broader. Improve routing and include boundary examples that demonstrate escalation.

Outputs become repetitive

The dataset may contain near-duplicates or one dominant template. Audit cluster diversity and remove repeated synthetic examples.

Latency or cost rises unexpectedly

Check whether outputs became longer, retries increased, or schema failures trigger repair calls. Track cost per accepted outcome, not only cost per model request.

6. Release through shadow traffic and canaries

Do not replace the production model in one step.

Use a progression:

1. Replay a fixed evaluation set.
2. Run production traffic in shadow mode without using the outputs.
3. Send a small percentage of low-risk requests to the new model.
4. Compare acceptance, correction, escalation, latency, and cost.
5. Expand only when quality gates hold.

Keep the prior model version available for immediate rollback. A deployment is not safe if rollback requires retraining or rebuilding an image.

7. Build a reviewed improvement loop

Production failures are valuable training candidates, not automatic training data.

For each candidate:

• preserve the original input and context
• record the expected behavior
• identify the actual root cause
• remove or protect sensitive data
• have a domain owner approve the example
• add it to a held-out evaluation before adding it to training

If you train on every user correction, you will absorb inconsistent preferences, malicious inputs, and fixes for problems that were never model problems.

Maintain a dataset changelog explaining why examples were added or removed. This makes later regressions easier to investigate.

8. Monitor model-specific health

Track:

• task success and human acceptance
• schema and tool-call validity
• escalation and refusal quality
• base-model versus fine-tune win rate
• latency and tokens by task type
• performance on critical subgroups
• distance between production inputs and training data

Input drift is a prompt to investigate, not proof that retraining is required. Sometimes the product has attracted a new use case that deserves a separate workflow.

Production-readiness checklist

• A stable adapter isolates provider-specific APIs.
• Every run records model, prompt, tool, and retrieval versions.
• Structured outputs and tool calls are validated.
• Routing keeps the model inside its evaluated scope.
• Base and previous model versions are available for comparison.
• Shadow and canary stages precede broad release.
• Rollback is fast and tested.
• Training candidates receive human review and provenance.
• Critical evaluation cases remain held out from training.

Summary

Fine-tuning changes model behavior; it does not remove the engineering around the model.

The reliable pattern is a stable adapter, strict contracts, scoped routing, layered diagnostics, and evidence-based release gates. When something breaks, reproduce it across model versions before changing the dataset. When you do retrain, use reviewed examples that fix a demonstrated behavior rather than whatever happened to appear in the logs.

That closes the lifecycle: prepare, train, deploy, integrate, observe, and improve without losing control of what changed.