Build a Personal AI Assistant – Part 4: Tooling and API Integrations

Your assistant now remembers past conversations, but it still cannot act. This tutorial wires external services (calendar, docs, email) through a structured tool system so the model can call APIs safely.

---

Tool Registry Design

Store tool definitions in tools/registry.yaml:

- name: list_calendar_events
  description: "List next meetings for a user on Google Calendar"
  request:
    method: GET
    url: https://www.googleapis.com/calendar/v3/calendars/primary/events
    query:
      key: env:GOOGLE_API_KEY
      maxResults: number
  response:
    format: json
  auth: oauth
- name: send_email
  description: "Send an email via Gmail"
  request:
    method: POST
    url: https://gmail.googleapis.com/gmail/v1/users/me/messages/send
    body:
      subject: string
      to: string
      body: string
  auth: oauth

Parser:

// src/tools/registry.ts
import fs from "node:fs";
import yaml from "js-yaml";

export interface ToolDefinition {
  name: string;
  description: string;
  request: Record<string, unknown>;
  response: Record<string, unknown>;
  auth: "none" | "bearer" | "oauth";
}

export class ToolRegistry {
  private tools: Map<string, ToolDefinition>;
  constructor(path = "tools/registry.yaml") {
    const parsed = yaml.load(fs.readFileSync(path, "utf-8")) as ToolDefinition[];
    this.tools = new Map(parsed.map((tool) => [tool.name, tool]));
  }
  list() {
    return [...this.tools.values()];
  }
  get(name: string) {
    const tool = this.tools.get(name);
    if (!tool) throw new Error(`Unknown tool: ${name}`);
    return tool;
  }
}

Takeaway: Config files make tools auditable by security/legal.

---

Planner Middleware

We’ll use OpenAI’s function calling again. Extend createCompletion to include tool specs:

import { ToolDefinition } from "../tools/registry";

export async function createCompletion(messages: Message[], tools: ToolDefinition[]) {
  const toolSpecs = tools.map((tool) => ({
    type: "function",
    function: {
      name: tool.name,
      description: tool.description,
      parameters: tool.request.body ?? { type: "object", properties: {} },
    },
  }));
  const response = await client.responses.create({
    model: "gpt-4o-mini",
    input: messages,
    tools: toolSpecs,
  });
  return response;
}

When response.output[0].content[0].tool_calls exists, handle them before returning to the user.

---

Tool Executor with Policy Enforcement

Install axios or got. We'll use got.

npm install got

src/tools/executor.ts:

import got from "got";
import { ToolDefinition, ToolRegistry } from "./registry";
import { logger } from "../utils/logger";

export class ToolExecutor {
  constructor(private registry = new ToolRegistry()) {}

  async run(call: { name: string; arguments: string }) {
    const def = this.registry.get(call.name);
    const args = JSON.parse(call.arguments || "{}");
    this.validate(def, args);
    const response = await this.dispatch(def, args);
    logger.info({ tool: def.name, status: response.statusCode }, "Tool call");
    return response.body;
  }

  private validate(def: ToolDefinition, args: Record<string, unknown>) {
    const schema = def.request.body ?? {};
    const missing = Object.keys(schema).filter((key) => !(key in args));
    if (missing.length) {
      throw new Error(`Tool ${def.name} missing params: ${missing.join(",")}`);
    }
  }

  private async dispatch(def: ToolDefinition, args: Record<string, unknown>) {
    const headers: Record<string, string> = {};
    if (def.auth === "bearer") headers.Authorization = `Bearer ${process.env.API_TOKEN}`;
    const method = (def.request.method as string).toLowerCase();
    if (method === "get") {
      return got(def.request.url as string, { searchParams: args, headers });
    }
    return got(def.request.url as string, { method: def.request.method, json: args, headers });
  }
}

Add circuit breakers, retries, and rate limiting as needed.

---

Integrate Tools into the Assistant

Update Assistant.send:

const result = await createCompletion(combinedHistory, this.registry.list());
const toolCalls = result.output?.[0]?.content?.[0]?.tool_calls ?? [];
if (toolCalls.length) {
  for (const call of toolCalls) {
    try {
      const outcome = await this.executor.run(call);
      this.workingMemory.append({
        role: "system",
        content: `Tool ${call.name} result: ${outcome}`,
      });
    } catch (err) {
      this.workingMemory.append({
        role: "system",
        content: `Tool ${call.name} failed: ${String(err)}`,
      });
    }
  }
  return this.send(message); // re-run LLM with tool results
}

Takeaway: Always append tool responses to context so the LLM reasons about them.

---

OAuth and Secrets

Sensitive tools require OAuth tokens. Create services/oauth.ts to refresh tokens and store them in keytar or a secrets manager.

import { google } from "googleapis";

export async function getGoogleClient() {
  const oauth2Client = new google.auth.OAuth2(
    process.env.GOOGLE_CLIENT_ID,
    process.env.GOOGLE_CLIENT_SECRET,
    process.env.GOOGLE_REDIRECT_URI,
  );
  oauth2Client.setCredentials({
    refresh_token: process.env.GOOGLE_REFRESH_TOKEN,
  });
  const { credentials } = await oauth2Client.refreshAccessToken();
  return { client: oauth2Client, accessToken: credentials.access_token };
}

Reference env:GOOGLE_API_KEY or env:ACCESS_TOKEN placeholders in the registry and substitute them before requests.

---

CLI Enhancements

Add commands to inspect tools:

program
  .command("tools:list")
  .action(() => {
    new ToolRegistry().list().forEach((tool) =>
      console.log(`${tool.name} - ${tool.description}`),
    );
  });

program
  .command("tools:describe")
  .argument("<name>")
  .action((name) => {
    const tool = new ToolRegistry().get(name);
    console.log(JSON.stringify(tool, null, 2));
  });

Takeaway: Developers need visibility into available actions.

---

Testing

Mock tool execution:

import { ToolExecutor } from "../src/tools/executor";
import { describe, it, expect, vi } from "vitest";
import got from "got";

vi.mock("got");

describe("ToolExecutor", () => {
  it("validates missing args", async () => {
    const executor = new ToolExecutor(/* inject mock registry */);
    await expect(
      executor.run({ name: "send_email", arguments: "{}" }),
    ).rejects.toThrow("missing params");
  });
});

Integration tests: spin up mock servers with msw or nock to emulate external APIs.

---

Verification Checklist

• npm run cli tools:list shows defined tools.
• npm run cli chat --system "You plan meetings" triggers list_calendar_events when appropriate.
• Logs capture tool.call metrics with status codes.
• OAuth tokens refresh automatically and never print to console.
• Tests cover executor validation and planner branching.

With tools connected, the assistant can act. The final part (5) will harden testing, simulations, and deployment.

---