Author: Pat Blachly, Data Science Consultant
Introduction
RunBuggy faces significant challenges in managing its extensive documentation, with both external customers and internal teams often struggling to locate necessary information quickly. This cumbersome process not only slows down customer service but also hampers internal operations. To address these inefficiencies and enhance the accessibility of information, RunBuggy has developed two tools:
- Document Management System (DMS): a tool to organize and govern the data quality of documentation libraries and their associated metadata (tags) stored in RunBuggy’s file system. Newly created tags include flagging which RunBuggy roles (i.e., Internal Users, Shippers, Transporters) should have access to a given document and surfacing a link to the public-facing documentation (on https://support.runbuggy.com/) associated with it. An AI model governs the integrity of the DMS by comparing the contents of each document to identify intra- and inter-document conflicts so we can surface potential errors to document owners. This system helps to ensure the DMS is the best (and cleanest) source of truth for company documentation.
- RunBot: RunBuggy’s virtual assistant leveraging LLM technologies to receive User questions in Slack, discern their intents, apply data restrictions based on the User’s role, and synthesize answers through consultation and evaluation of our system/Data Science data sets and documentation that lives in the DMS subject to those restrictions.
How RunBot answers User questions about RunBuggy documentation
Let’s explore the interplay between the RunBuggy DMS and RunBot (diagrammed above) by seeing what happens when a RunBuggy team member asks RunBot the question:
Team Member #1: How do I add a new transport company in Hitch?
RunBot: Hello I am RunBot, your friendly AI assistant!
Thanks for your request, I’m on it!
RunBot catches the team members’ question in Slack and kicks off a sequence of processes to answer it. Before jumping into this, it’s important to explain how the DMS manages metadata that supports RunBot’s answering capabilities. All internal sites configured so that Subject Matter Experts (SMEs) can tag their data to control RunBot behavior:
- Owner: the SME who owns the content and permissions associated with a document.
- Ingest Into RunBot: a selection of yes/no, indicating whether RunBot should have access to a given document.
- RunBot Permission Group: the RunBuggy personas that are allowed to see the content of a given document (options: RunBuggy Internal Users, Shippers, Transporters).
- Business Line: the RunBuggy platform associated with the document. This may be set to one or more of the following: Winton, Hitch, RunBuggyOne, RunBot, HR.
- Support Site Link: a URL linking out to content published to the public Internet (https://support.runbuggy.com/) that is tied to the document.
Having explained the DMS, we return to the team members question and how it gets answered by RunBot, following the steps from the diagram above:
Step 0: All documents tagged with “Ingest into RunBot = yes” and their associated metadata are stored in Pinecone, a vector database that RunBot can access and reference when answering user questions.
Step 1: Team Member #1’s question is submitted to RunBot.
Step 2: RunBot looks up who the team member is, determines his data access privileges, considers his question, and determines his intent (e.g., is he asking about sourcing? About a TO’s order status? About company documentation?). RunBot thus extracts the following information that guide how it goes about answering Team Member #1’s question:
- Team Member #1 is an internal RunBuggy employee.
- Team Member #1 is asking about “Hitch”.
- Team Member #1’s intent is to have a question of company documentation answered.
Step 3: RunBot acts upon Team member #1’s intent, using the information extracted in Step 2 to try to answer his question using a technique called retrieval-augmented generation (RAG). The steps involved include:
- AI techniques are applied to determine what pieces of documentation in Pinecone are most relevant to Team Member #1’s question: “How do I add a new transport company in Hitch?”, subject to a set of filters:
- “Business Line” must include “Hitch”, since Team Member #1 intends to know how to perform his action in Hitch.
- The two most relevant pieces of content are surfaced to an LLM model to “synthesize” an answer to Team Member #1’s question from them. If the LLM model is unable to find suitable information, it will return that it cannot determine the answer from the content it has been given.
Step 4: RunBot evaluates whether the answer from Step 3 is sufficiently answering Team Member #1’s question. If it is, it publishes this answer and, importantly, cites the references associated with the answer to Slack.
Back to Team Member #1’s actual query, we see the following response after the steps above were followed:
RunBot: Here you go:
To add a new transport company in Hitch, follow these steps:
1. Click your username.
2. From the dropdown, click Account Management.
3. Select the parent account from the active account dropdown.
4. Click Companies Management.
5. Click Transporter Accounts.
6. Click Add Transporter.
7. Enter in the Transporter’s information.
8. Click Save.
Helpful Links:
#1:Hitch_BasicDealershipTrainingResourceGuide_Transporters_AddIn_01172023.pdf ((Internal Site URL), (Support Site URL))
Page Numbers: 1
Owner: employee #1
Business Line: [‘Hitch’]
#2: Hitch_ManageTransporterAccounts_Video_12052023.pdf ((Internal Site URL), (Support Site URL))
Page Numbers: 2
Owner: Admin #2
Business Line: [‘Hitch’]
So, Team Member #1 gets the procedure for adding a new transport company in Hitch along with two citations indicating the documents (and associated page numbers) used to answer his question. The “Internal Site URLs” point to the specific pages of source material (Job Aids, Presentations, or Video Transcripts) used to answer the question, while the Support Site URL redirects Team Member #1 to where content is shown publicly, often accompanied by a video or PDF:
Leveraging the DMS in RunBot in Slack: an example
Here we show how a User can ask @RunBot what address should be used for RunBuggy on (a new transporter’s) certificate of insurance (COI).
- The User, here a RunBuggy employee, asks @RunBot the question in the #runbot_test channel in Slack.
- @RunBot replies, providing both:
- a. the RunBuggy OMI mailing address as an answer
- b. two documents it consulted in providing the answer. These document citations include tags of who owns the document, what RunBuggy business line the documentation is relevant to, and links to both RunBuggy’s sources (internal-facing) and Support Site (external-facing, so accessible to Shippers and Transporters) documents used by @RunBot in its answer.
- The User can and should tap on the links provided by @RunBot to see the reference documents, confirm the answer is what they need and, if helpful, find additional information on the topic.
What’s Next
We’ve developed RunBot to answer questions like the following examples:
- How do I add a new transport company in my Hitch instance?
- How do I create an order?
- What is required for transporters to use the platform?
- How can you help source an order?
- How do I update an ETA?
- What is RunBuggy’s address for certificate of insurance?
In conclusion, the innovative implementation of RunBuggy’s Document Management System (DMS) alongside the AI-driven capabilities of RunBot significantly enhances the efficiency and accuracy of handling documentation within the company. This integrated system not only streamlines internal processes but also optimizes customer interactions by providing rapid, precise answers to queries through advanced AI technology. As RunBuggy continues to refine these tools, the potential to further improve both internal workflow and customer satisfaction remains vast, ensuring that RunBuggy remains at the forefront of technological advancements in the automotive logistics industry.
Learn more about RunBot at https://runbuggy.com/artificial-intelligence-runbuggy-runbot/.
