How We Developed an AI Voice Agent for PC*Miler Routing

PC*Miler is Trimble’s industry-leading application designed to generate routes that are safe, efficient, and legal for trucks and other commercial vehicles. This article describes how we developed a working model for an AI voice agent capable of processing a user’s trip details (such as trip origin, destination, waypoint, etc.) and returning a route displayed on a map.

While this agent is not commercially available, we hope to apply what we learned to future applications.

Watch Video

Artificial Intelligence (AI) Agent

An AI agent is a system designed to intelligently interact with its environment, which embodies the principles of perceiving, reasoning, and acting in a rational manner to achieve its designated goals

Semantic Kernel

Semantic Kernel is a lightweight, open-source Software Development Kit (SDK) created by Microsoft that enables developers to easily integrate Large Language Models (LLMs) like OpenAI, Azure OpenAI, and Hugging Face into their applications using common programming languages such as C#, Python, and Java.

As illustrated above, Semantic Kernel:

• Selects the best AI service to run the prompt.
• Builds the prompt using the provided prompt template.
• Sends the prompt to the AI service.
• Receives and parses the response.
• And finally returns the response from the LLM to your application.

It acts as an orchestration layer that simplifies the process of building intelligent applications by combining the power of LLMs with traditional programming logic and external resources. It provides a structured and extensible way to create AI-powered features and autonomous agents.

AI Voice Route Design

A voice-enabled map route system using Semantic Kernel is an intelligent application that allows users to request and interact with map routing functionalities through voice commands, leveraging the Semantic Kernel SDK to orchestrate Large Language Models (LLMs) and other components to understand natural language, retrieve map data, calculate routes, and provide voice-based directions.

Here’s a breakdown of the key elements and how Semantic Kernel facilitates this:

• Voice Input: The system begins with the user providing a voice command, such as “Take me from Boston to Miami via Raleigh.”

• Speech-to-Text (STT): This voice input is converted into text using a Speech-to-Text service. This is typically an external component integrated with the Semantic Kernel application.

• Natural Language Understanding (NLU) with LLMs (via Semantic Kernel): This is where Semantic Kernel plays a crucial role. The text transcribed from the user’s voice command is processed by an LLM (e.g., OpenAI, Azure OpenAI) through Semantic Kernel. The LLM performs several key tasks:

  • Intent Recognition: Identifying the user’s goal (e.g., “take me from … to … ,” “get directions”).

  • Entity Extraction: Identifying key pieces of information within the command, such as the destination (“1 Independence Way, Princeton”), origin (which might be the user’s current location or specified verbally), and any waypoints in route path.

• Semantic Kernel Plugins: Semantic Kernel enables the integration of various plugins to fulfill the user’s request:

• Location Services Plugin: This plugin interacts with APIs to determine the user’s current location (if needed).

• Map Data Plugin: This plugin communicates with map service providers (e.g., Trimble Maps) to retrieve map information, search for points of interest, and calculate routes. This plugin might have native functions for geocoding (converting addresses to coordinates) and generating routes.

• Routing Logic Plugin: This plugin might contain specific logic for route optimization (PC*Miler Web Services) based on criteria like distance, time, and traffic conditions. Semantic Kernel can orchestrate the LLM to use this plugin with the extracted parameters.

Our Working Model

The Trimble voice-base AI routing model is built with 3-Tier architecture, namely:

1. Frontend (Maps User Interface)
2. Middleware (Service Interface)
3. Backend (AI Core engines)

Each tier has a specific responsibility, making the application easier to understand, develop, and maintain. It allows for the use of different technologies for Frontend based on specific requirements.

This execution flow outlines our system where a user’s voice query is cleaned, understood, semantically analyzed, processed using cloud AI services, used to generate a route via the PC*Miler Route Reports API, and finally displayed on Trimble Maps. The feedback loop indicates a potential for iterative refinement of the natural language understanding based on semantic analysis.

Our system process begins with a User’s Query (1), cleaned audio is then processed by Noise Cancellation (2) and Natural Language (3).

Semantic Kernel (4) orchestrates Azure AI (5) services to execute advanced natural language understanding, entity recognition, or preparing the query for routing APIs.

PC*Miler Route Reports API (6), depicted by a map pin icon with a route line. This API is likely used to calculate and determine the optimal route based on the user’s query and potentially other factors.

Finally, the planned route is visually displayed in Trimble Maps (7).