Chat Transcript Analysis Tool
- internal users
- 1,000+
- internal users
- daily active users
- ~100
- daily active users
- conversations analyzed a day
- 200K+
- conversations analyzed a day
- LLM-generated metrics
- 15+
- LLM-generated metrics
Overview
An internal tool that uses Gemini to analyze transcripts from an enterprise chatbot with 300K+ sessions a day. It scores each conversation for sentiment, writes a summary and reports agent performance metrics. Product, design, operations and analytics teams use it to see how the assistant behaves in real conversations, not just in dashboards. As a PM, I built much of it myself with AI coding tools and took it from first prototype to a tool with 1,000+ users.
Problem
At hundreds of thousands of sessions a day, nobody can read enough transcripts to know why a conversation succeeded, failed or escalated. Session counts and containment rates show what happened but not why. Teams needed a way to see sentiment, summaries and recurring failure patterns across all conversations, not a hand-picked sample.
My Role
Product manager and hands-on builder. I co-built the first version with our product director and an engineer. I defined the problem, designed the prompts and metrics, and wrote much of the tool myself with Claude, Antigravity and other AI coding tools. The engineer later made the production analysis pipeline and relay code more robust. I led adoption across internal teams.
What it looks like
Conversations analyzed
1.42M
last 7 days
Containment
78%
▲ 0.6 pts
Negative sentiment
14%
▼ 1.2 pts
Repeat contact · 48h
9.3%
▲ 0.4 pts
Sentiment mix by day
Top escalation drivers
- Asked for a person on the first turn 29%
- Billing dispute (needs a person) 21%
- Same intent missed 2+ times 17%
- Identity check failed 12%
- Outage status unclear 8%
Transcripts · 5 of 2,906 flagged
- C-20931 Wi-Fi drops Negative
- C-20928 Bill explanation Neutral
- C-20925 Plan upgrade Positive
- C-20919 Equipment return Neutral
- C-20914 Appointment Positive
C-20931 · Wi-Fi drops
App · 9 turns · 6m 12sLLM summary
Customer reported evening Wi-Fi drops for three nights. The assistant ran a connection check and suggested a channel change. The customer had already tried that and asked for a technician. The assistant offered the same fix again before escalating.
Flagged moment · turn 6
“I already did that yesterday. Can someone just come look at it?”
Approach & Key Decisions
- ▹ Used Gemini to turn raw chat transcripts into structured output: a sentiment score, a short summary and a set of agent performance metrics for each conversation.
- ▹ Designed the metrics with the people who use them, so each one answers a question a product or operations team actually asks.
- ▹ Started with a working prototype instead of a spec. I built it with Claude, Antigravity and other AI coding tools so ideas went from question to working feature in days, not sprints.
- ▹ Co-built the first version with the product director and an engineer. Once usage grew, the engineer made the production analysis pipeline and relay code more robust, so the prototype could become a tool teams depend on.
- ▹ Treated LLM output as something to check: compared the generated scores and summaries against human-reviewed transcripts before teams relied on them.
- ▹ Grew it from a small prototype to 1,000+ internal users, with one engineer on production hardening and no dedicated engineering team for the rest.
Outcomes
- ★ 1,000+ internal users, with about 100 using it each day.
- ★ Used across product, design and operations to find what drives sentiment and escalation in the chatbot.
- ★ Shows that a PM working with AI coding tools can find a gap, build the fix and get it adopted at enterprise scale.
More work