We are frequently using the word optimise when discussing AI implementations. How do you optimise the AI to help ensure a good authoritative, reliable and accurate response is being produced every time? AI is not a silver bullet - that is to say, it will not work perfectly every time regardless of the scenario. Each scenario and knowledge collection should to be assessed and then sculpted and its settings edited to ensure success.
*See all Atlas AI & IKS articles here*
Recommended validation flow
- Run the question 10 times on the same settings to test , with a new chat each time
- Have a "best" answer to compare, similar to a test script, so you know what good looks like
- Start with a smaller document sample set
- Start with GPT-model 4o Mini
- Consider the importance of user prompts and how they should be structured (initial round of testing keep 'as is', but consider review depending on outcomes)
- Once a comparison has been completed, look at settings configuration and refinement. At this point we can consider use of taxonomy controls and system and user prompt changes
- Make sure you provide feedback with a Thumbs Up or Down for every response so that a feedback report produced can be easily reviewed.
- Consider wide versus specific question and responses based on the scenario and the content available. The more generic content you have the more chance the question returned needs either a better prompt and although deemed as roughly correct, might not be exactly accurate
In the following examples, we are utilizing a Knowledge Collection which points towards approved internal documents on a company's Processes, Policies and Procedures.
In this scenario, it would be a usual request to ask the AI about a certain topic, and which policies may cover it.
I will ask the AI the same question 3 times, each with a different setting. We will be focusing only on the following 2x parameters.
Number of Search Results - this is the quantity of relevant outputs the models generates against the query. It can be understood in terms of diversity and relevance of outputs. A higher number of search results will provide a wider variety of content to leverage from. The lower the number will lead to a more targeted response with very specific answers, ensuring the responses generated are highly relevant and accurate
-
This explanation is from within the Atlas Tooltop: "Represents the number of chunks/facts that will be retrieved as a result of a search operation from the grounded knowledge to be then filtered by relevancy."
-
This parameter will have more or less impact depending on the amount of content available and the type of questions being asked
-
Relevancy Score - The higher the relevancy score, the smaller amount of context is given to the model and therefore should be very relevant to your question (only very relevant content is included). The lower the relevancy score, the higher amount of context is given to the model and therefore (less relevant content is included)
- Technical explanation from within the Atlas Tooltip: "Represents the minimum relevancy for each of the search results/chunks/facts retrieved in order to consider them relevant enough for the model to use them as source of knowledge to generate a response."
- This parameter will have more or less impact depending on the content and the type of questions being asked
There is a screenshot of the particular settings the AI utilized when responding. The differences will be summarized at the bottom of this page.
Question and Answer
Question: 'Which policy will inform me about effective communication?' (medium settings)
Answer:
To learn about effective communication, you can refer to the Supplier Relationship Policy, which emphasizes the importance of effective communication and error reporting in managing supplier relationships. It provides guidelines on how to report errors and request changes, ensuring clarity in communication processes within the context of supplier interactions [1].
Additionally, the Lone Remote Worker Policy highlights the significance of maintaining communication among remote employees, which is crucial for productivity and collaboration in a remote work environment [2].
Moreover, the Information Exchange Policy provides guidelines for effective communication, stressing the need for clarity and adherence to established protocols when exchanging information [3].
These documents collectively offer a comprehensive view of policies related to effective communication within the organization.
--------------------------------------------
Question: 'Which policy will inform me about effective communication?' (high settings)
Answer:
To learn about effective communication, you can refer to the Supplier Relationship Policy. This policy emphasizes the importance of effective communication and error reporting in managing supplier relationships. It provides guidelines on how to report errors and request changes, ensuring that communication is structured and clear [1].
Additionally, the Lone Remote Worker Policy outlines the significance of maintaining productivity and communication among remote employees. It aims to foster a collaborative environment and provides instructions for effective communication while working remotely [2].
The Backup Policy also touches on the need for effective communication in the context of document handling and data integrity. This policy stresses the importance of adhering to established protocols and provides instructions for communication related to maintaining data integrity [3].
Lastly, the Leaving Policy highlights the necessity of clear communication during the process of leaving the company. It emphasizes adherence to guidelines and proper documentation, which are crucial for effective communication within the organization [4].
--------------------------------------------
Question: 'Which policy will inform me about effective communication?' (low settings)
Answer:
The policy that will inform you about effective communication is the Supplier Relationship Policy. This document emphasizes the importance of effective communication and error reporting in managing supplier relationships. It provides guidelines on how to use the policy, report errors, and request changes, ensuring clarity in these processes [1].
Additionally, the Lone Remote Worker Policy also highlights the significance of maintaining communication among remote employees, which is crucial for productivity and collaboration in a remote work environment [2].
Both documents provide insights into the importance of effective communication within their respective contexts.
--------------------------------------------
First Question - A Summary
As expected, due to the difference in settings, the AI responses have differing levels of granularity and due to this, each response references a different number policy documents.
- Low settings referenced 2x policies
-
- Supplier Relationship Policy
- Lone Remote Worker Policy
-
- Medium settings referenced 3x policies
-
- Supplier Relationship Policy
- Lone Remote Worker Policy
- Information Exchange Policy
-
- High settings referenced 4x policies
-
- Supplier Relationship Policy
- Lone Remote Worker Policy
- Backup Policy
- Leaving Policy
-
Please note, the AI response is not incorrect here. The user has asked which policy, and the AI responses have returned at least 2. Although the answers vary, all information can be deemed as correct based on the documentation available and the question asked.
It's important to note that from this answer we cannot be sure of is if there is a 5th or 6th policy. We have not specifically asked the AI how many policies there are, we have asked a more open ended question.
Second Question and Answers
If the user wanted to find out a total quantity of documents which are related to how many policies talk about effective communication, a different question should be asked.
Question: "How many policies contain information about effective communication?"
Answer:
The retrieved documents indicate that several policies contain information about effective communication. Specifically, the following policies highlight this aspect:
-
Supplier Relationship Policy: This document emphasizes the importance of effective communication and error reporting in managing supplier relationships [1].
-
Lone Remote Worker Procedure: This procedure discusses guidelines for effective communication among remote workers, ensuring a structured approach to remote work [2].
-
Social Networking Process: This document provides detailed instructions for effective communication within the context of social networking activities [3].
-
Backup Policy: It includes guidelines for effective communication in maintaining data integrity and handling errors [4].
-
Leaving Policy: This policy stresses the need for clear communication during the process of leaving the organization [5].
-
Information Security Policy: It highlights the importance of clarity in communication regarding security protocols [6].
-
Information Exchange Policy: This document emphasizes guidelines for effective communication and error reporting [7].
-
Virus Protection Policy: It encourages proactive communication regarding issues related to virus protection [8].
-
Information Exchange Process: This document discusses effective communication as crucial for document management [9].
-
Information Exchange Procedure: It outlines the importance of effective communication in managing documentation [10].
-
Leaving Procedure: This procedure highlights the importance of effective communication during the transition process [11].
-
Supplier Relationship Procedure: It emphasizes the importance of effective communication and error reporting in supplier management [12].
In total, there are 12 policies that contain information about effective communication.
Second Question - A Summary
With this second question we are getting more consistent results.
- Low settings referenced 11x policies
- Medium settings referenced 12x policies
- High settings referenced 12x policies
This could be due to the more closed question we are asking, where the AI is making less objective decisions about what it thinks is a correct answer - it is utilising a simple call and retrieve mechanism against the Search index - where it is retrieving any policy which mentions the words "effective communication".
The lower settings has returned one fewer than the others - this is likely due to it having lower search results to make a selection from.
High number of search results but with low relevancy - both questions
We now have a feeling around how the AI is modelling its response based on the setting parameters we give in the KC. However in the previous examples we have moved both sliders in unison. What happens when we only move one slider?
I will now ask both questions separately, each in a new chat to ensure a clean response, but I have set the Number of results to high, and the relevancy score to low. This should mean that it is looking at a wider variety of content AND it will return more generic or potentially less accurate answers. It is doing this because we are asking generic questions.
Question 1: 'Which policy will inform me about effective communication?'
Answer:
The policy that will inform you about effective communication is the Supplier Relationship Policy. This document outlines compliance documentation and procedures for managing supplier relationships and emphasizes the importance of effective communication and error reporting. It provides guidelines on how to use the policy, report errors, and request changes, ensuring clarity in the communication processes involved with suppliers [1].
Additionally, the Lone Remote Worker Policy also discusses the importance of maintaining communication among remote employees, highlighting the need for structured communication practices to foster collaboration in remote work settings [2].
Question 2 : 'How many policies contain information about effective communication?'
Answer:
Based on the retrieved documents, there are several policies that contain information about effective communication. Specifically, the following policies emphasize the importance of effective communication:
-
Supplier Relationship Policy: This policy outlines procedures for managing supplier relationships and emphasizes effective communication and error reporting [1].
-
Lone Remote Worker Procedure: This document discusses guidelines for effective communication among remote employees, highlighting the need for a structured approach to remote work [2].
-
Social Networking Process: This process outlines guidelines for effective communication and documentation, emphasizing clarity and organization in managing social networking activities [3].
-
Information Exchange Policy: This policy includes guidelines for effective communication and stresses the importance of clarity in documentation [4].
-
Backup Policy: It emphasizes effective communication as part of maintaining data integrity and reporting errors [5].
-
Information Security Policy: This policy highlights the need for clarity in communication, particularly in the context of security protocols [6].
-
Lone Remote Worker Policy: It discusses maintaining communication among remote employees and provides guidelines for effective use of the document [7].
In total, seven policies contain information about effective communication.
--------------------------------------------
Although there's a larger net of content to select from as we've increased the number of search results used, because the relevancy score is low, there's still a limit applied and the AI is only returning the most specific content which it deems has to be relevant based on a more severe setting. It's working on a best-case scenario and returning what it can with the settings applied.
Low number of search results but with high relevancy - both questions
I will now ask both questions separately, each in a new chat to ensure a clean response, but I have set the Number of results to low, and the relevancy score to high. This should mean that it is looking at a smaller array of content AND it will return only return specific content it deems very relevant.
Question 1: 'Which policy will inform me about effective communication?'
Answer:
To learn about effective communication within your organization, you can refer to several policies.
The Supplier Relationship Policy outlines the importance of effective communication specifically in the context of managing supplier relationships. It includes guidelines for error reporting and requesting changes, indicating how to maintain clear communication with suppliers [1].
Additionally, the Lone Remote Worker Policy emphasizes maintaining productivity and communication among remote employees, providing instructions for fostering a collaborative environment. This policy can be particularly relevant for those working in remote settings [2].
Furthermore, the Leaving Policy stresses the need for clear communication and adherence to established protocols during the process of leaving the organization, highlighting the importance of proper documentation and communication in such scenarios [3].
Lastly, the Backup Policy mentions the importance of effective communication as it relates to document handling and error reporting, ensuring that all protocols are adhered to for maintaining data integrity [4].
These policies collectively provide insights into the standards and expectations surrounding effective communication within your organization.
Question 2 : 'How many policies contain information about effective communication?'
Answer:
Two policies contain information about effective communication. The first is the Supplier Relationship Policy, which emphasizes the importance of effective communication and error reporting, providing guidelines on how to report errors and request changes [1].
The second document is the Lone Remote Worker Procedure, which also discusses guidelines for effective communication as part of its procedures for lone remote workers. This policy highlights the importance of maintaining a structured approach to remote work while ensuring adherence to company policies [2].
Mixed settings - A Summary on both questions
We can see that depending on the question and the settings, the AI is giving quite varied responses.
High number of search results but with low relevancy
- Question 1 'what policies' - 2 referenced documents
- Question 2 'how many policies' - 7 referenced documents
Low number of search results but with high relevancy
- Question 1 'what policies' - 4 referenced documents
- Question 2 'how many policies' - 2 referenced documents
The results are inverse.
Questions such as 'what policies' surface more results when the relevancy is higher, even though the number of search results was lower. The relevancy therefore influences the number of search results it is using as it needs to ensure relevancy over number of results. We haven't specified how many results, just that something matches.
Questions such as 'how many policies' surface more results when the number of search results is higher, even with low relevancy. Because the question is asking for a numerical output, the number of search results becomes more important, and the relevancy does not matter as much as it's a simple keyword match.
Conclusion
It is hard to select the optimum settings because the answers depend on the question asked. If there is a greater variety of questions being asked of the AI, it is going to be harder to establish what the optimum settings should actually be because there's less certainty over what questions are going to be asked.
The settings greatly depend on the context the KC is providing - is it a comparison tool? Is it a Q&A chatbot? Specifying what kind of questions are going to be asked prior will help ensure the testing and modelling meet the specific purpose of the KC.
If in doubt, the relevancy score settings are the most important for the response as it will more greatly impact the quality of the output, as the number of search results is just limiting the number of chunks (pieces of data) the AI is filling itself with.
The more content you have in the KC - i.e. the more generic the KC - the more important 'number of search results' will become. If you only have several short documents in the KC, the limit of the number of search results will likely not be hit. However if you have 1000 documents covering a variety of topics and a variety of questions - the model will need more specific settings on both the number of search results and the relevancy score to ensure a beneficial outcome as there is a greater chance that the model will just find the first thing which matches and return it - it does not know if the answer is definitely correct or not, but it is trying it's best based on the question, the settings, and the content it has at it's disposal.
Auditing the responses
Please view this article 'IKS Auditing' for how to audit AI responses. It is very important as part of the Pilot/testing/validation period that feedback is provided from users when KCs are being set-up and tested. This feedback (thumbs up, thumbs down, comments) is captured and can be reviewed, along with other stats and settings captured. Utilizing the auditing capabilities, it is possible to intensively test and review the AI tooling to ensure the best results and their related settings at the time is auditable.
Prompts
What we have not covered here is the KC 'model settings instructions'. This is the setting where you can provide additional instructions which will be engrained into every question as well as the user prompt. So you can help guarantee the optimum response to the kinds of questions which might be posed by ensuring the model settings instructions specify how the model should answer. This could be in format, tone, prose, length of response, etc.
You can write very lengthy model settings instructions which aim to cover all bases, but it is not a silver bullet - a fix all - the settings still need to be tested with a variety of questions and optimised to ensure positive and correct results across the board.
You can read more about Prompting here in our article What is Prompting and Why is it important?
Comments
0 comments
Please sign in to leave a comment.