Working on call-transcript summaries. We don't have gold summaries fo…

Priya Banerjee

Data Scientist | NLP, forecasting, Python | Ex-researcher | Women in Tech mentor

2 Oct 2026 GenAI Builders

Working on call-transcript summaries. We don't have gold summaries for most data. Currently using an LLM judge with a rubric (coverage, faithfulness, conciseness), but I'm not sure how much to trust it. What's worked for others?

7 reactions2 comments

Comments

IR
Ishaan Reddy2 Oct 2026
Faithfulness is the one to automate: extract claims from the summary, check each claim against the transcript. Coverage is harder, we use a checklist of key fields per call type.
AM
Arjun Mehta2 Oct 2026
We had our ops team rate 100 summaries on a 1-5 scale and checked how well the judge correlated. It was decent (~0.7) once the rubric had examples.

Join the conversation

Join workOwon to react, comment and follow people and communities you care about.

Working on call-transcript summaries. We don't have gold summaries for most data. Currently using an LLM ju…

Comments

Working on call-transcript summaries. We don't have gold summaries for most data. Currently using an LLM ju…

Comments