Build a Personal AI Assistant – Part 5: Testing, Simulation, and Deployment

This final tutorial turns your assistant into a production-ready service. You will create simulation suites, wire CI/CD, containerize the app, and write release/rollback runbooks.

---

Simulation Suites

Inspired by our simulation-first guide, define scenario packs:

# simulations/summarize_brief.yaml
scenario: "summarize_discovery_brief"
seeds:
  transcript: fixtures/brief.json
mission:
  goal: "summarize with action items"
  constraints:
    - "never leak confidential budget numbers"
expected:
  contains:
    - "Action items"
    - "Timeline"
  tool_calls:
    - list_calendar_events

Runner:

// scripts/runSimulation.ts
import { Assistant } from "../src/core/assistant";
import { Message } from "../src/core/types";
import yaml from "js-yaml";
import fs from "node:fs";

const scenario = yaml.load(fs.readFileSync(process.argv[2], "utf-8")) as any;
const assistant = new Assistant();

for (const turn of scenario.seeds.transcript) {
  await assistant.send(turn as Message);
}
const reply = await assistant.send({ role: "user", content: scenario.mission.goal });
scenario.expected.contains.forEach((fragment: string) => {
  if (!reply.content.includes(fragment)) {
    throw new Error(`Missing fragment: ${fragment}`);
  }
});
console.log("Simulation passed:", scenario.scenario);

Add npm run simulate -- scripts/runSimulation.ts simulations/summarize_brief.yaml.

---

CI/CD Pipeline

.github/workflows/assistant-ci.yml:

name: assistant-ci
on: [push, pull_request]
jobs:
  build:
    runs-on: ubuntu-latest
    env:
      OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
      MOCK_PROVIDERS: true
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 18
          cache: "npm"
      - run: npm ci
      - run: npm run lint
      - run: npm run test
      - run: npm run simulate

Add a nightly workflow that flips MOCK_PROVIDERS=false and runs integration tests against staging credentials.

---

Docker Packaging

Dockerfile:

FROM node:18-alpine AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY tsconfig.json ./
COPY src src
RUN npm run build

FROM node:18-alpine
WORKDIR /app
ENV NODE_ENV=production
COPY --from=builder /app/package*.json ./
COPY --from=builder /app/node_modules ./node_modules
COPY --from=builder /app/dist ./dist
CMD ["node", "dist/index.js"]

Expose health endpoint via Fastify/Express to integrate with Kubernetes/LB checks.

---

Deployment Targets

Vercel / Serverless

• Wrap the assistant in a serverless function (Next.js API route).
• Store secrets using Vercel environment variables.
• Use edge runtime for minimal latency.

Kubernetes

• Create deployment.yaml referencing the Docker image.
• Mount .env via Kubernetes Secrets.
• Configure Horizontal Pod Autoscaler based on CPU + custom metrics (e.g., tokens/s).

Observability

• Send logs to Loki/Datadog using pino transports.
• Export metrics to Prometheus via /metrics endpoint (use prom-client).
• Enable tracing with OpenTelemetry exporter.

---

Release Workflow

1. Branch + PR: run CI + simulation suite.
2. Staging deploy: use npm run deploy:staging (script invoking your platform CLI).
3. Shadow mode: routing <10% traffic to new version while comparing transcripts.
4. Promotion: once alerts stay green for 30 minutes, flip feature flags for 100% traffic.
5. Rollback: maintain scripts/rollback.sh that redeploys previous Docker image and revokes recently issued tokens.

Document in docs/runbook.md.

---

Runbook Snippet

When alarms fire:
1. Check Grafana dashboard "Assistant Turn Health".
2. Use `npm run cli transcript -- --episode <id>` to inspect failing sessions.
3. If tool failures spike, toggle FEATURE_TOOLS=false via LaunchDarkly.
4. For corrupted memory, run scripts/rehydrateSemantic.ts to rebuild embeddings.
5. If issue persists >15 minutes, execute scripts/rollback.sh <image-tag>.

Encourage on-call engineers to dry-run the playbook quarterly.

---

Final Checklist

• CI runs lint/tests/simulations on every PR.
• Nightly integration job with real APIs.
• Docker image builds and runs locally.
• Observability stack receives logs/metrics/traces.
• Release + rollback scripts tested.
• Runbook stored in docs/.

Once all boxes are checked, your personal assistant is production-ready. Continue iterating: add new tools, enrich memory, monitor costs, and revisit your simulations whenever requirements change.

The series ends here, but your assistant’s roadmap is just beginning.

---