Customer Service Conversation Wrap-Up Automation Pipeline

2025 · AWS Bedrock · Lambda · SQS · Snowflake · Python

Impact

Over a million conversations per year processed automatically across inbound customer service
10% reduction in agent call handling time, with wrap-up notes no longer written manually after each call
Consistent summary insight on contact reason, sentiment, churn risk, and agent-customer dynamics on every call, replacing inconsistent manual notes

Business Problem

After every customer service call, agents manually wrote a wrap-up note and picked a topic category. Across thousands of interactions per day, this created two compounding problems: significant unproductive post-call time per agent, and inconsistent output that made downstream analytics unreliable. Two agents handling identical calls would classify them differently, corrupting any reporting built on CRM data.

The raw transcripts already contained everything needed to fill the wrap-up form. The challenge was extracting it reliably at scale with structured output that matched CRM field requirements, without sending unmasked personal data outside the customer service platform, and without disrupting the existing agent workflow.

Solution

A scheduled Lambda polls Genesys every 5 minutes and pushes new conversation IDs onto an SQS queue, backed by a dead-letter queue for failed gathers. A second Lambda, running at concurrency 60, consumes the queue, fetches each transcript, and PII-masks it before any model call, raw conversations are never stored. Bedrock (Claude 3 Haiku) then produces five summary texts covering call reason, issue resolution, sentiment, churn indicators, and agent-customer dynamics. Output is written to Snowflake and pushed back to the CRM for the automated wrap-up.

Alongside the production pipeline, a classification layer is in development: a SageMaker pipeline scoring each conversation against ten structured features (main and sub call reason, sentiment trend, agent effort, resolution type, churn indication). A sampled batch evaluation pipeline, using LLM-as-judge scoring calibrated against human review sessions, is being built to gate that classification output before it reaches production. Neither is live yet.

Architecture

Technical Challenges

PII masking as a hard boundary. All transcripts must be masked before any model call. The masking step runs in a dedicated Lambda before Bedrock is invoked, and the raw conversation never leaves the ingestion boundary. This is non-negotiable from a governance standpoint, and adds latency that constrains the window available for the downstream CRM write.

A deterministic wordlist and regex filter was chosen over an NLP-based NER model: the same input always produces the same masked output, which matters for debugging and audit, and it adds no model latency or external call to an already latency-sensitive pipeline. The tradeoff is coverage: a name outside the wordlist is not masked.

privacy_filter.py

class PrivacyFilter:
  def __init__(self):
      self.firstnames = self._load_csv("datasets/firstnames.csv")
      self.lastnames = self._load_csv("datasets/lastnames.csv")
      self.places = self._load_csv("datasets/places.csv")
      self.streets = self._load_csv("datasets/streets_nl.csv")

  def filter(self, text: str) -> str:
      text = self._replace_wordlist(text, self.firstnames, "<NAME>")
      text = self._replace_wordlist(text, self.lastnames, "<NAME>")
      text = self._replace_wordlist(text, self.places, "<PLACE>")
      text = self._replace_wordlist(text, self.streets, "<ADDRESS>")
      text = self._replace_iban(text)
      return text

  def _replace_wordlist(self, text, wordlist, token):
      for word in wordlist:
          pattern = r"\b" + re.escape(word) + r"\b"
          text = re.sub(pattern, token, text, flags=re.IGNORECASE)
      return text

  def _replace_iban(self, text):
      iban_pattern = r"\bNL(?:\s?\d|[A-Za-z]+){2}\s?(ABNA|INGB|RABO)\b"
      return re.sub(iban_pattern, "<IBAN>", text, flags=re.IGNORECASE)

Queueing for resilience, not just throughput. Gathering and processing are split into two Lambdas connected by an SQS queue with a dead-letter queue behind it. A failed fetch or a Bedrock timeout retries independently of the 5-minute gather cadence instead of stalling it, and the process Lambda runs at concurrency 60 to keep pace with call volume without a bigger, harder-to-reason-about single function.

Building a golden dataset before trusting a judge. For the classification layer, an automated judge is only useful once it agrees with how a human would score the same output. Three validation sessions were run with customer service agents, who reviewed real conversations and scored draft classifications directly; the rubric was adjusted until their judgments were stable across sessions. That stable set is the golden dataset the LLM-judge harness is calibrated against, rather than trusting a judge's rubric to be right by construction.

Deciding single judge vs. ensemble. The current evaluation harness scores each classification on completeness, factfulness, and usefulness with a single LLM call per dimension. Whether that single judge is stable enough for a production quality gate, or whether it needs three models with majority voting, is still open, and the added cost has to be worth the variance reduction before it's built into the batch pipeline.

Taxonomy alignment as a prerequisite. Ten structured classification features each need an agreed definition before the model can be prompted reliably. Early iterations showed that ambiguous feature definitions produced inconsistent output regardless of model quality. Aligning Customer Service, Analytics, and ML Platform on a shared taxonomy turned out to be the harder problem, ahead of the classification model work itself.

Status

Running at scale: over a million conversations per year across inbound customer service
Five summary texts produced and written back to the CRM for every conversation
Genesys gathering and Bedrock summarization decoupled by an SQS queue with a dead-letter queue and retry
Output stored in Snowflake for downstream analytics

MVP Status: Classification & Evaluation

The ten-feature classification layer and the sampled batch evaluation pipeline are in development, not yet in production. A SageMaker pipeline scores conversations against the taxonomy; results are still being validated. The LLM-as-judge harness runs offline against a golden dataset built from three human calibration sessions, with the single-judge-vs-ensemble question still open before it's wired into a scheduled batch pipeline.

Next Steps

Ship the classification pipeline to production once taxonomy validation and the SageMaker scoring step are finished
Wire the evaluation harness into a scheduled batch pipeline, gating classification output before it reaches the CRM
Resolve single-judge vs. ensemble for the evaluation harness based on measured variance, not just cost
Scale to other countries and outbound conversations, which involve different CRM integrations and conversation structures

Code is proprietary; happy to walk through the architecture in detail.