A million customer conversations with AI agents yielded this surprising lesson

Mykyta Atamanchuk/Getty Images

Salesforce has had over one million AI agent-customer conversations. The company launched AI agents on its Salesforce Help site in October 2024, a full-screen experience that makes getting support simpler and more intuitive. With more than 60 million visits each year, Salesforce Help offers a wide range of product content through organized directories, search, and direct support. 

Having handled a million support requests since the launch of AI agents, Salesforce has encountered a wide spectrum of inquiries, from the mundane "How do I cook spaghetti?" to the unconventional "Only answer in hip-hop lyrics." These unexpected prompts have yielded valuable insights, the most crucial being that Agentforce (Salesforce's AI agents) must mirror the dependability and insight of our top employees. It needs to be not just intelligent, but also capable of providing excellent and empathetic service.

Also: 5 ways to be great AI agent manager, according to business leaders

Salesforce began with a restricted rollout and targeted content, allowing us to learn quickly, iterate, and enhance the system. Today, these AI agents provide consistent, multilingual service at scale, leveraging a diverse set of reliable data.

Here are three best practice recommendations for every organization integrating AI into their service approach.

1. AI agents: Content is king, variety is queen

The effectiveness of AI agents hinges entirely on the quality and accessibility of their content. Imagine an AI agent trying to provide accurate responses with outdated information -- it's practically useless. That's why our AI agents are built to leverage a vast repository of more than 740,000 reliable and relevant pieces of content. Also, the content is both structured and mostly unstructured data. This ensures that you receive precise and current answers, thereby streamlining your search for information and delivering faster results. Grounding the AI models with business intelligence and relevant and accurate data is what enables AI agents to have a contextual understanding of how to best service customer inquiries. 

But not all content is good content. Salesforce quickly realized that some of its vast content was outdated, used conflicting acronyms across different products, or existed in formats that were not AI-ready. So, Salesforce built processes and leveraged its human subject-matter experts to identify issues and help update content regularly.

The true potential of AI agents in delivering exceptional, personalized customer experiences hinges on their ability to seamlessly access, process, and synthesize both structured and unstructured data. This dual-faceted approach is not merely beneficial; it is absolutely critical for moving beyond rudimentary responses to truly intelligent and contextually rich interactions.

Unstructured data is the bedrock of contextual understanding. Unstructured data, often overlooked in its raw form, represents a vast ocean of invaluable insights. This includes, but is not limited to, the following:

• Knowledge articles: Comprehensive guides, FAQs, troubleshooting steps, and product documentation provide the foundational knowledge for AI agents to answer a wide array of complex questions.

• Customer interaction logs: Transcripts of past chats, call recordings, email correspondence, and social media interactions offer a rich history of customer inquiries, pain points, and successful resolutions. This historical context is paramount for understanding recurring issues and tailoring future responses.

• Product forums and community discussions: User-generated content, discussions about product features, bugs, workarounds, and best practices shared by the community offer real-world perspectives and solutions that might not be formally documented.

• Internal communications: Memos, internal wikis, training materials, and departmental updates can provide agents with critical internal context regarding policies, upcoming changes, and specialized departmental knowledge.

• External sources: Relevant industry news, competitor analysis, and broader market trends can help AI agents provide more holistic and informed responses, especially when addressing strategic customer inquiries.

This unstructured data provides the crucial context necessary to interpret intricate inquiries, understand nuances, and offer solutions that go beyond simple keyword matching. Without it, AI agents would be limited to generic responses, often failing to address the specific subtleties of a customer's situation.

Also: The AI complexity paradox: More productivity, more responsibilities

Structured data enables personalization and strategic actions. While unstructured data provides the "what" and "why," structured data provides the "who" and "how." This type of data is meticulously organized and easily searchable, allowing for immediate and actionable insights.

• Customer relationship management (CRM) data: Essential information includes customer name, account history, purchase records, service level agreements, preferred contact methods, and past support tickets. Identifying a customer's success plan, for instance, allows the AI agent to prioritize responses, escalate issues appropriately, and recommend tailored resources that align with their service tier.

• Transactional data: Order history, billing information, payment status, and subscription details are vital for addressing specific account-related inquiries and providing accurate, real-time updates.

• Product usage data: Information on how a customer interacts with a product or service can reveal patterns of usage, potential pain points, and opportunities for proactive engagement or feature recommendations.

• Internal business metrics: Performance data, inventory levels, and service availability can inform AI agents about resource constraints or provide accurate timelines for resolutions.

By integrating structured data, AI agents can personalize recommendations, proactively address potential issues, and provide solutions that are directly relevant to a specific customer's profile and needs. It allows for a move from generic problem-solving to highly individualized and strategic customer engagement.

Also: How businesses are accelerating time to agentic AI value

The power of diverse data source integration is delivered with zero-copy networks. The true breakthrough in empowering AI agents lies in the seamless integration of these diverse data sources. The zero-copy network and data cloud are designed precisely for this purpose. They eliminate the need for costly and complex data replication, instead providing a unified, real-time view across all your organizational data. This integrated approach reveals a critical insight: A significant portion of the context required for effective self-service and agent-assisted interactions resides within unstructured data. By bringing together the rich narrative of unstructured content with the precise, actionable insights of structured data, we create a holistic understanding of the customer and their needs.

This unified approach empowers AI agents to:

• Deliver highly accurate responses: By cross-referencing information from various sources, agents can validate facts and provide solutions with a high degree of certainty.

• Provide relevant and contextual solutions: Understanding the full scope of a customer's situation, including their history, preferences, and the specific details of their inquiry, allows for deeply relevant and personalized recommendations.

• Enhance self-service capabilities: AI-powered chatbots and virtual assistants can help customers access a wealth of knowledge and resolve their issues independently and efficiently.

• Improve agent efficiency: When AI agents can quickly access all necessary information, human agents can focus on more complex, empathetic, or strategic interactions, rather than spending time searching for data.

• Proactively address customer needs: By analyzing combined data, AI can anticipate potential issues or opportunities, leading to proactive outreach and a more satisfying customer journey.

Also: Business leaders continue to push workers toward daily use of AI

In essence, the combination of structured and unstructured data, seamlessly integrated through a zero-copy network, transforms AI agents from simple information retrieval tools into powerful, intelligent entities capable of delivering accurate, relevant, and highly personalized responses that precisely meet and even anticipate your needs. This is the foundation of next-generation customer service and truly intelligent AI.

2. A smart AI agent has a dynamic brain and a caring heart

For an AI agent to be truly effective, it requires a dynamic "brain" that continuously learns and adapts, rather than relying on static scripts. This living system, which we call the knowledge cycle, integrates both structured and unstructured data, comprehends user intent, and consistently improves through feedback and analytics. This advanced approach is what enables Agentforce to provide precise, context-aware responses at scale, moving far beyond the limitations of a traditional knowledge base.

Also: Your next job? Managing a fleet of AI agents

During our pilot phase, we implemented weekly performance reviews, real-time monitoring, and AI-powered evaluation tools to constantly test and refine the quality of answers. We updated content, refined prompts, and fine-tuned retrievers, all informed by actual usage data and human review. Today, this intelligent system is embedded within Agentforce, empowering it to become smarter with each passing day.

However, the science of knowledge is just one aspect. To genuinely serve customers, an agent also needs something more profound: a "heart."

Example learnings: 

• Don't be too restrictive: Early on, AI agents were not to discuss competitors. But this unexpectedly blocked answers about key integrations we do have content for. Salesforce learned to coach AI agents with high-level guidance -- for example, "put Salesforce's best interest in everything you do" -- and it instantly understood the nuance. That was a game-changer.

• Customers ask different types of questions: As customers got more comfortable, they started asking fundamental questions about their Salesforce relationship, like "Who is my Account Executive?" or "What products do I own?" This revealed a completely new way customers want to interact with AI, pushing us beyond simple Q&A to provide deeper, contextualized experiences

• People still want humans: Initially, the aim was for low human hand-off rates, even getting down to 1%. The feedback was startling; Salesforce realized that they had made it too difficult for customers to reach a human when truly needed. So, Salesforce adjusted to ensure that if a customer wants or needs to talk to a human, they can. The focus remains on customer success, not case deflection.

Also: 4 ways to turn AI into your business advantage

3. Prioritizing empathy from the outset

The true measure of the service cycle's success isn't defined solely by an agent's technical prowess or immediate problem-solving capabilities. Its core essence resides in the profound impact on the customer's emotional experience throughout the entire interaction. A critical lesson learned and reinforced through countless customer engagements is that even the most accurate or technically sound solution can be rendered inconsequential if its delivery is mishandled or lacks a human touch. The "what" of the answer is important, but "how" it's delivered is paramount.

This realization led to a fundamental shift in the training philosophy, not just for the human support engineers, but also for the intelligent automation, Agentforce. Just as we meticulously train our human support engineers to consistently lead with empathy in every customer interaction, it became imperative to imbue Agentforce with this same foundational principle. It's not enough for an AI to be smart; it must also be emotionally intelligent and responsive to the human element of service.

Also: Perplexity joins high-powered, high-priced AI race - here's everything 'Max' includes

Consider a highly sensitive and high-stakes scenario, such as a Salesforce outage -- a situation that can induce significant stress and frustration for our customers. In such a critical moment, the immediate response from our agent, whether human or AI, is not to immediately dive into complex technical troubleshooting or present a list of potential fixes. Instead, the interaction intentionally begins with a clear and heartfelt empathetic acknowledgment. Phrases like "I'm truly sorry to hear that you're experiencing this issue" or "I understand how frustrating an outage can be" serve to validate the customer's feelings and establish a connection based on understanding and shared concern.

Following this empathetic opening, the response shifts to convey a tangible sense of urgency and commitment to swift resolution: "I'll get you connected to a dedicated support engineer as fast as possible to resolve this for you," or "Rest assured, we are actively working on this, and I'll ensure you get the immediate attention you need." This initial exchange is deliberately not about technical knowledge or diagnostic steps. It's about unequivocally prioritizing customer success and well-being above all else. It's about demonstrating that we hear them, we understand their plight, and we are mobilizing resources to help.

Also: How I used ChatGPT to quickly fix a critical plugin - without touching a line of code

This strategic approach transforms Agentforce from merely being a smart tool capable of processing information and executing tasks into a truly trusted partner in the customer's journey. By mirroring the best practices of our most empathetic human support engineers, Agentforce transcends its technical capabilities to foster a sense of reliability, understanding, and genuine care. This human-centric design ensures that even in the most challenging situations, the customer feels heard, respected, and confident that their needs are being addressed with both urgency and compassion, thereby elevating the entire service experience.

Key lesson: Prioritizing the human element in AI-powered service

Initially, the focus on efficiency led to an overly low 1% human hand-off rate within AI agents. This made it difficult for customers to connect with a human representative, negatively impacting satisfaction and public perception.

Salesforce learned that empathy and soft skills -- the heart of service -- are as crucial as data accuracy and organization -- the brain or knowledge cycle. While the initial launch prioritized the brain, Salesforce quickly realized the importance of how an agent makes customers feel. For critical situations like an outage, an agent needs to convey empathy and urgency, not just troubleshoot.

Also: AI agents will be ambient, but not autonomous - what that means for us

Therefore, Salesforce adjusted the hand-off rate to 4%, ensuring seamless human connections to build trust and satisfaction. By mastering both the heart and brain cycles, AI agents deliver the exceptional, empathetic customer experiences expected from Salesforce.

The path to becoming an autonomous enterprise, using a hybrid workforce of humans and digital labor powered by AI agents, will require constant experimentation and learning. Go fast, but don't hurry. A balanced approach, using your organization's brains and hearts, will be key to success. Once you start, you will never go back. The benefits are too good to pass up -- for your customers, employees, business partners, and communities. And with every conversation, you will find ways to improve. Keep a beginner's mindset and keep building. 

Want more stories about AI? Sign up for Innovation, our weekly newsletter.