Foyer

Operations runbook

The handful of things that go wrong in a booking agent, and what to do about each. Kill switch, calendar drift, cost ceiling, backup and restore. Most pages here translate to a Filament admin toggle or a single shell command.

Kill switch

Every business has a per-tenant kill switch. Toggling it via POST /v1/businesses/:id/kill-switch or the Filament admin puts the tenant into read-only mode:

  • The agent stops replying.
  • Inbound texts get a static message — “we are temporarily handling this manually; someone will reach out shortly” — configurable per business.
  • STOP/START/HELP still work; they are caught before the kill-switch check.
  • Pending slot holds and bookings are preserved; nothing is canceled.

Use it when: a misconfigured persona is producing bad replies, the owner is unreachable for the day, or you need to drain the queue without committing anything new.

Calendar drift

The push channel (events.watch) notifies Foyer when a managed event is moved or deleted out-of-band. If the channel expires or a push is missed, the five-minute fallback poll catches the gap. The owner dashboard surfaces a sync-health indicator if drift exceeds threshold.

Manual reconciliation:

php artisan foyer:calendar-reconcile --business=01J4YQX9MA0RBPV6N7K8WJ6XYZ

The reconcile job is idempotent — running it twice in a row produces the same result as running it once.

Cost ceiling

Each business has a per-day LLM spend budget. Today’s spend is rolled into llm_cost_daily on every turn-result write. When the budget hits 80%, an alert fires; at 100%, Foyer downshifts to the cheap-mode model (Haiku / GPT-4o-mini). If the cheap-mode spend also exceeds the budget, the kill switch flips for that business and the owner is notified.

# Inspect today's spend
php artisan foyer:cost-report --business=... --date=today

# Reset a stuck budget
php artisan foyer:cost-reset --business=...

Twilio outbound failures

The message_deliveries table records every status callback. The two codes that matter:

  • 30007 — carrier filtered. The destination network rejected the message; usually a content / pattern flag.
  • 30008 — unknown error. Often a transient issue but treated as a sign to pause that destination.

Both raise an alert and pause that destination number for 24 hours — no further outbound. If the pattern repeats across destinations, check the campaign for content drift since registration.

Backup and restore

Postgres nightly dumps to the S3 backup bucket (same bucket the other portfolio projects use). Retention is 30 days. The cron is in scripts/backup.sh and runs at 03:00 server-local under cron.

# List backups
aws s3 ls s3://philiprehberger-backups/foyer/

# Restore the most recent
./scripts/restore.sh --from-latest

# Restore a specific date
./scripts/restore.sh --from=2026-06-12

The restore script restores into a separate database name, runs the sanity checks (row counts per table, last booking date, last conversation date), and swaps the connection only after the operator confirms. It is not silently destructive.

Slot-hold backlog

The slot-cleanupworker runs every 60 seconds and expires stale holds. If it falls behind — Horizon dashboard shows queue depth climbing — the customer-side effect is that the “hold expired” text is delayed. The fix is normally Horizon worker capacity, not a Foyer code change. The Horizon slot-cleanup pool has processes set to 1 by default; bump to 2 if depth stays high.

Health endpoints

curl https://api.foyer.example.com/v1/healthz
# {"healthy":true,"db":"ok","redis":"ok","agent_worker":"ok"}

BetterStack polls this every 30 seconds; a sustained 5xx pages the on-call.

When in doubt

Toggle the kill switch first, then diagnose. A wrong reply that goes out is a customer-trust failure; a temporary silence with a static fallback is recoverable.