KIRA — AI Executive Secretary
A complete reference for KIRA's web dashboard and mobile application: what it does, how it's built, and — the throughline of this document — how AI models are integrated as production infrastructure rather than a single API call bolted onto a feature.
Introduction
KIRA is a personal executive-secretary product: it records meetings, transcribes them — including natural Hindi–English code-switching ("Hinglish") common in Indian business conversation — and turns the transcript into structured intelligence: a written summary, every decision logged, every action item assigned, every commitment tracked with a due date, and a drafted follow-up email. Recording happens on a phone or in a browser; review happens anywhere, on either surface.
The system is deliberately split across two client surfaces and one backend:
- Mobile app (Flutter, Android + iOS) — the primary recording surface, built for capturing a meeting on the go, including offline-tolerant chunked upload.
- Web dashboard — a lightweight browser interface for reviewing meeting intelligence, managing tasks, chatting with the assistant over your own knowledge base, and — as of the latest iteration — recording directly from a desktop browser.
- Backend — a single FastAPI service that owns the recording pipeline, the AI orchestration layer, and the data model both clients read from.
Core Capabilities
Live microphone capture (mobile & browser) or existing file upload, chunked and resumable.
Multilingual speech-to-text with automatic Hindi / Hinglish detection and translation.
AI-generated summary, sentiment, key topics, decisions, commitments, and a drafted follow-up email.
Entities, relationships, timeline events, decisions and promises are extracted and cross-linked, not just stored as flat text.
Action items surface automatically from meetings, alongside manual tasks, prioritized and due-dated.
A daily-priority view — overdue, due-today, and high-priority items — assembled automatically each day.
A conversational assistant that answers questions against your own ingested documents and meetings via semantic search.
In-app and webhook-driven alerts when a meeting finishes processing.
System Architecture
Both clients speak to the same versioned REST API. Nothing about the AI pipeline is visible to either client — they upload audio and poll a job status; every decision about which model, which provider, and which fallback tier handles that audio lives entirely in the backend, so the orchestration layer can evolve without a client release.
This separation is deliberate: AI providers are treated as a swappable infrastructure tier, not a hardcoded dependency threaded through the app. That design choice is what made the resilience work described later in this document possible without touching either client.
Overview & Stack
The mobile app is built in Flutter, targeting Android and iOS from a single codebase. State is managed with Riverpod; secure token storage uses platform keychain/keystore integration rather than plain preferences.
| Layer | Choice | Notes |
|---|---|---|
| Framework | Flutter | Single codebase, Android + iOS |
| State management | Riverpod | Provider-based, testable state notifiers per feature |
| Networking | Dio | Interceptor-based JWT attach & auto-refresh on 401 |
| Secure storage | flutter_secure_storage | Android Keystore / iOS Keychain-backed |
| Recording | record package | AAC-LC encoding, foreground service for background capture |
Recording & Upload
A recording moves through four stages on-device before the backend ever sees it as a job:
- Capture — microphone audio is recorded locally to disk; a foreground service keeps recording alive if the app is backgrounded.
- Session init — the app requests an upload session, receiving a token, a server-confirmed chunk size, and a job ID it can start polling immediately.
- Chunked transfer — the file is sliced and uploaded chunk-by-chunk against that token, so a dropped connection resumes rather than restarts.
- Finalize & poll — the server stitches chunks, verifies integrity, and starts the AI pipeline in the background; the client polls job status until the meeting is ready.
The same four-stage flow also accepts an existing audio file picked from the device, not just a live recording — useful for meetings recorded on separate hardware.
Auth & Sessions
Authentication uses short-lived JWT access tokens paired with longer-lived refresh tokens. An HTTP interceptor attaches the access token to every request and, on a 401, transparently exchanges the refresh token for a new access token and retries the original request — a session stays alive across normal usage without the user ever seeing a login prompt mid-task.
Screens
Daily priorities at a glance on open.
Live capture with timer, or file upload with a topic label.
List and detail view with full intelligence breakdown.
Filterable list, swipe-to-complete with resolution notes.
Entities recognized across meetings, with interaction history.
Conversational chat plus voice-command wake-word support.
Overview & Stack
The web dashboard is a single-page application served directly by the backend — no separate frontend build or deploy pipeline. It's deliberately dependency-light: vanilla JavaScript against the same REST API the mobile app uses, so every capability exposed to the browser is guaranteed to already be battle-tested by mobile traffic.
Panes & Features
Overdue, due-today, and high-priority items in one view.
Ask questions against your own ingested knowledge base.
Full task list with status filters and completion notes.
Every ingested document, searchable and browsable.
List, intelligence detail, and now — recording — in one pane.
Live token-consumption monitor across every AI provider in use.
Browser Recording
The Meetings pane can record directly through the browser's own microphone API, or accept a file upload — using the identical chunked-upload protocol the mobile app uses, down to the same endpoints and job-polling logic. No new backend surface was needed to add this; only the client-side capture code changed, which is the architecture in System Architecture paying off directly.
Design Philosophy
This is the section that matters most for how KIRA is actually engineered. AI models here are not called once and trusted — they're treated the way any other unreliable external dependency would be treated in serious infrastructure: with verified operating limits, layered fallbacks, and defensive handling of what they hand back.
Verify against the live provider, not the documentation. Every quota, rate limit, and model behavior referenced in this pipeline was confirmed by inspecting real API responses directly — not assumed from a spec sheet or a blog post — before any code was written against it.
Pipeline Flow
Every recording — regardless of which client it came from — passes through the same eight-stage pipeline:
if Hinglish
embed + index
entities, decisions
Meeting
Each stage is independently retried and logged; a failure in one stage doesn't silently corrupt the ones after it, and the whole run is fully observable after the fact — which model handled it, how long each stage took, and what it produced.
Transcription Layer
Speech-to-text is the highest-volume, most quota-sensitive stage, so it's the most heavily layered. Rather than depend on a single model, transcription falls through three independently-verified tiers:
- Primary model — the main speech-to-text model on the primary provider.
- Secondary model, same provider — confirmed via direct API inspection to track a completely separate daily allowance on the same account, not a shared pool.
- Independent provider — an entirely separate infrastructure vendor, with its own quota, wired in and confirmed end-to-end with real audio before being trusted as a fallback.
Each tier is only ever engaged if the one before it reports it's out of capacity — the pipeline doesn't guess which tier to use; it tries the cheapest/fastest option first and only escalates on an explicit rejection from the provider.
Language Handling
Meetings aren't assumed to be monolingual English. Detection of Hindi and "Hinglish" (code-switched Hindi-English, the norm in a lot of Indian business conversation) doesn't depend solely on what the transcription model reports — it's backed by a text-level heuristic layer that checks independently for Devanagari script and a curated set of commonly romanized Hindi markers. This matters because it means translation still triggers correctly even on a fallback transcription tier that doesn't report a language code at all — the system doesn't lose language awareness just because it had to reroute.
Extraction Layer
A second, separate LLM call reads the (translated, if needed) transcript and pulls out structured memory: named entities, relationships between them, timeline events, decisions, and promises with owners and due dates — each cross-linked back to the source meeting. This extraction layer has its own Groq-then-Gemini fallback chain, independent from the transcription chain, so the two highest-value stages of the pipeline don't share a single point of failure.
Hardening AI Output
A model instructed to "return an ISO date" will still sometimes return
"this Friday". Treating that as an edge case to shrug off would mean losing
real commitments silently. Instead, every date-shaped field an LLM produces passes
through a dedicated normalization layer before it reaches the database — one that
resolves relative phrases ("next Friday", "tomorrow", "in three days") against the
meeting's own timeline before they're ever written to a strict date column.
due_date: "this Friday" → rejected by the database, item lost due_date: "2026-07-24" → resolved, saved, correctly sorted in Tasks
This is a small piece of engineering with an outsized effect: it's the difference between a pipeline that occasionally, silently drops a commitment someone made out loud in a meeting, and one that doesn't.
Usage Observability
Every AI call — transcription or text generation, whichever provider actually served it — logs its own token or duration cost. That feeds a live usage panel (visible in the web dashboard) showing consumption per provider, per model, and over time, so capacity planning is based on real, observed usage rather than an assumed ceiling.
Data Model
The schema separates raw content from derived intelligence, so re-processing or re-extracting never requires re-uploading audio:
| Entity | Purpose |
|---|---|
| meetings | One row per processed recording — transcript, summary, sentiment, language, linked job |
| documents / chunks | Ingested text, chunked and embedded for semantic search |
| entities / people | Named things and individuals recognized across meetings, deduplicated and merged over time |
| decisions / promises / timeline_events | Structured memory extracted from meetings, cross-linked to the entities involved |
| tasks | Action items, whether extracted from a meeting or entered manually |
| jobs / job_logs | Background pipeline execution state and full per-stage audit trail |
| api_usage_log | Per-call token/duration cost by provider and model, feeding the usage panel |
Security & Auth
- Passwords are hashed with bcrypt; never stored or logged in plain text.
- Sessions use short-lived JWT access tokens plus a longer-lived refresh token, rotated on each refresh.
- Upload sessions use a separate, single-purpose token scoped only to that one upload — a large chunked transfer doesn't need to carry a full user credential on every chunk request.
- Ownership checks are enforced server-side on every meeting, task, and document read — a valid token for one account cannot read another account's data by guessing an ID.
Deployment
The backend runs as a single FastAPI/Uvicorn service under a process supervisor,
backed by PostgreSQL with the pgvector extension for semantic search. The
web dashboard is served as a static asset directly from the same service — there is no
separate frontend deployment to keep in sync. The mobile app builds independently for
the Play Store and App Store against the same versioned API.
Roadmap
- Re-enable automatic task extraction from meeting text once the secondary extraction provider's billing is fully provisioned.
- Extend the relative-date hardening layer to timeline events surfaced in the chat assistant, not only meeting-derived tasks.
- Add a manual per-job retry action to the dashboard, so a stalled recording can be recovered without developer intervention.