←Back

Customer Service Conversation Wrap-Up Automation Pipeline

2025 to 2026 · AWS Bedrock · SageMaker Pipelines · DynamoDB · Snowflake · MLflow · Python

Impact

  • Over a million conversations per year processed automatically across inbound customer service
  • 10% reduction in agent call handling time, with wrap-up notes no longer written manually after each call
  • Consistent summary insight on contact reason, sentiment, churn risk, and agent-customer dynamics on every call, replacing inconsistent manual notes

Business Problem

After every customer service call, agents manually wrote a wrap-up note and picked a topic category. Across thousands of interactions per day, this created two compounding problems: significant unproductive post-call time per agent, and inconsistent output that made downstream analytics unreliable. Two agents handling identical calls would classify them differently, corrupting any reporting built on CRM data.

The raw transcripts already contained everything needed to fill the wrap-up form. The challenge was extracting it reliably at scale with structured output that matched CRM field requirements, without sending unmasked personal data outside the customer service platform, and without disrupting the existing agent workflow.

Solution

Three pipelines, split along the line that matters: getting conversations safely stored, enriching them, and checking the enrichment.

Ingestion runs every five minutes. A scheduled job reads ended Genesys conversations from a cursor, fetches each transcript, masks the PII before anything is written, and merges the result into an ingestion table in Snowflake on conversation id. A row with no batch id assigned is waiting work, so the table is its own backlog. The split exists for durability: a conversation is stored within five minutes of the call, and a failed enrichment run costs compute rather than data.

Processing claims a batch. The ingestion job keeps a running count and starts a pipeline execution once 100 or more conversations are waiting, but that trigger carries no authority: the execution counts for itself, then claims the batch through a conditional write on a single DynamoDB state row. The claim seals the waiting rows under one batch id, and that seal is the authoritative record count. Enrichment then runs summarisation and classification as Bedrock batch jobs. Classification never reads the transcript, only the summary, so summarisation is upstream of everything else. Ten labels come out of it: eight depend only on the summary and run in one wave, two depend on another label's output and run after it. A validation step normalises the raw text and checks each label against the agreed taxonomy, retrying once and flagging a mismatch rather than guessing. Summaries and labels land in contract tables in the warehouse and go back to the CRM as the automated wrap-up.

The evaluation pipeline runs weekly as its own execution on the same pattern. It draws a stratified sample of finished output and scores it with an LLM judge on completeness, factfulness and usefulness for summaries, and accuracy for labels. Judge scores go back to the warehouse alongside the output they scored, and every run logs metrics and prompt versions to MLflow, so a quality change can be traced to the prompt version that caused it.

Architecture

Ingestion, processing and evaluation pipeline architecture
architecture_v8.svg

Technical Challenges

The state row is the lock, not the application logic. Two executions can race for the same batch, and an execution can die mid-run without writing anything at all. Both cases are handled by one DynamoDB row per batch: a conditional write on status = waiting means exactly one execution wins the claim, and a crash simply leaves the row in progress with a stale lease. Every later write is pinned to the lease it claimed, so a run whose row was taken gets a refused write instead of a silent overwrite. Each stage write refreshes the lease as a heartbeat, because time since the claim is the wrong clock in both directions: a long batch would be reclaimed while still working, and a run that died ten minutes in would hold the row for six hours.

Clearing the batch minimum, not the batch cap. Bedrock Batch refuses a job under 100 records, and that floor is not adjustable. The per-job cap never binds at this volume, so the real constraint sits at the quiet end: a thin batch fails at summarisation before any classification runs. The floor is therefore a trigger condition rather than an abort: unassigned rows keep accumulating in the ingestion table and no batch is claimed until the count clears it. A batch that stays open past twelve hours alerts a human rather than changing state on its own.

Counting before claiming, not after. The count the ingestion job keeps is advisory and drifts, because conversations arrive between the count and the claim. An execution that claimed first and then found too few records would need to hand the row back, which means a state transition the rest of the design assumes cannot happen, and it would livelock against the drifted count. A read-only count before the claim removes the case instead of compensating for it: rows leave the waiting pool exactly once, by a seal, and only the claim holder seals, so a fresh count above the floor guarantees a batch above the floor.

Batch for generation, sync for the judge. Summarisation and classification take the flat 50% batch discount and let AWS own throughput: no 429s, no worker tuning, no contention with production traffic. The weekly judge went the other way, because at sample size that discount buys ten process steps instead of three, a second failure surface and no fast test path. Sync catches a bad response and retries it inside the same run, and its traces carry score plus human label, which is what the calibration loop reads. The price of splitting is two mechanisms in one repo.

Three batch waves, not one fan-out. Every label reads summary text rather than the transcript, and two of the ten read another label's output, so the call graph collapses to three sequential stages at best. Each stage waits out its own turnaround, which is the main risk in the design, and it is why the eight independent labels are packaged as one job rather than eight parallel ones. A run can outlive the batch that started it, so the stage is recorded before each submit, Bedrock job names are derived from the batch and stage so an orphaned job can be found again, and the warehouse write is idempotent.

Status

  • Summarisation is running at scale: over a million conversations per year across inbound customer service
  • Five summary texts produced and written back to the CRM for every conversation
  • Masking runs before any transcript is stored, in every environment
  • Output stored in the warehouse for downstream analytics
  • Batch inference and the weekly evaluation run in production, both on the same pipeline pattern
  • The ingestion and processing split is designed and accepted; the scheduled gatherer and the rewritten claim module are the current build

Next Steps

  • Ship the scheduled gatherer, including a deduplicating merge into the ingestion table, since the Genesys analytics query can return the same record on a later page
  • Alarm both loops separately: a dead gatherer is silent unless it is alarmed, and batch staleness is a different signal from pipeline failure
  • Close the reclaim gap for a run that is alive but silent past its heartbeat, by recording the execution on the row at claim time and checking it before any reclaim
  • Calibrate the judge against the golden dataset and establish the human agreement ceiling it is scored against
  • Wire in the promotion gate, so a prompt version that scores below threshold does not reach the CRM
  • Scale to other countries and outbound conversations, which involve different CRM integrations and conversation structures

Code is proprietary; happy to walk through the architecture in detail.