In Atlas 6.0 we introduce robust feedback auditing capabilities to the Intelligent Knowledge Studio (IKS) which presents the opportunity to review in-depth any AI feedback and related responses to help assist with optimization and reviews, as well as the ability to check Knowledge Collection Health.
Please note, full auditing (auditing all responses regardless of if feedback is left or not) is a separate process with different tooling. Please reach out to your ClearPeople AI representative for more information on this.
Note, this article will assume the reader already has an understanding of Atlas AI, including permissions, abbreviations and terminology.
*See all Atlas AI & IKS articles here*
Benefits
Benefits for regulated environments:
- Provides clear audit capabilities to those AI Assistant interactions where feedback has been left.
- Captures feedback provided into the AI for comprehensive review and optimization.
- Allows comprehensive review of questions and answers and stats related to AI conversations.
- Role based access controlled through Entra ID Security Group.
Permissions required
To perform feedback auditing actions for the IKS you will need to be a member of the "Atlas IKS Auditors" Entra ID group. Without this membership you will not be able to access the IKS panel to action the auditing tasks necessary. This is a dedicated role with permissions to export and review user feedback and interaction reports. For more information on AI & IKS permissions please view the dedicated permission article here.
Auditing data is stored in a separate, secure workspace accessible only to authorized auditors. We'll talk more about this further down this article.
Access the Intelligent Knowledge Studio
Once you have the appropriate permissions, to access the IKS go to the 'My Atlas' menu under Settings:
Depending on the permissions you have been granted, this IKS panel is where you can view those Knowledge Collections (KCs/Collections) you have access to, add new collections, see the status for existing collections and edit them, and get reports on usage and user feedback:
6.1 View
6.0 View
Please note that in 6.0, the Export Feedback is not within IKS but instead is within the 'AI Assistant Settings'
Auditing Actions
Auditing actions may be taken from the first time that a Knowledge Collection is submitted to be indexed for the AI. You can use the "reload" icon to refresh the status if any collections are currently syncing. Status for each collection is shown on the Intelligent Knowledge Studio panel using RAG colours;
- Red = Error
- Amber = Warning, partial index, indexed with errors
- Green = Completed
- Blue = Syncing
You can also see the number of items indexed for the collection to the right. If you click on the ellipse you can see your options for the chosen collection. The full range of actions is shown below, with both Administrator and Auditing permissions applied.
If you have auditing permissions only, your view will look like the below and you will not have the ability to edit or delete a KC.
When an auditor selects 'Export Feedback' as part of a KC, a new window appears and the user can decide which duration of time the report should compile. This is a report for that specific KC only.
Select the preferred timeframe, and then select Generate Report
This is a similar process to selecting the 'Export Feedback' option at the top of the IKS window in 6.1 or within the 'AI Assistant Settings' in 6.0 should you have Auditor permissions to see this.
This will export the entirety of feedback provided into the AI Assistant from across all KCs (AI Assistant wide, sometimes referred to as 'Global Feedback'). Once clicked, you will see the same window where you can specify the timeframe.
Atlas AI Auditing Workspace + Feedback Reports
Caveat: Only those responses where a user has left feedback are captured in the reports.
On deployment of the Atlas AI tooling, a new workspace is provisioned named 'Atlas AI Auditing' and will have the URL Suffix '/sites/atlasaiauditing'. All exported reports are created, stored and accessed here.
Please note that the document library is in Search, so auditors will likely have reports appearing in their Atlas Searches. To avoid this, please 'hide' the document library from your Search Results (within Library settings > Advanced Settings)
Reports can be accessed directly from the Reports button at the top of the IKS panel in 6.1 or from within AI Assistant Settings in 6.0.
Selecting this button will take the auditor directly to the SharePoint Document Library where all the Reports within this Atlas AI Auditing workspace are stored. Any IKS Auditors will have permissions to this workspace automatically from the Entra ID group. Other individuals can be added to the workspace permissions manually if required.
Reports for specific KCs can be found in individual folders where the folder name includes the Title you named the collection with alongside its system GUID. So all KCs are grouped by folder automatically. Please note that the names of the folders will not change even if you alter the name of the KC - we drive the report process through the GUID only.
Each folder may have multiple reports contained within, depending on how often the 'Export Feedback' button on the KC has been selected and a new Report created.
We can see in this folder for the 'Business Development' KC Reports, there are multiple files. We recommend to try and produce fresh reports if true auditing is taking place, however previous reports produced will will continue to be stored here.
The reports will contain a number of columns, including the:
- Record ID for the individual piece of feedback
- Knowledge Collection
- Date of Vote
- Type of vote
- Which feedback selection was chosen
- Comments
- Input Question into the AI
- Output Answer produced by the AI
- System Prompt
- Prompt
- Prompt Tokens
- Prompt Response
- Rewritten question (6.1 only)
- Strict Grounding Setting
- Relevancy Threshold Setting
- Number of Facts Retrieved setting
So you can see that 19 columns makes for deep auditing abilities, with all questions, answers, prompts, KC settings, and tokens, captured.
Please note that the .csv may not format correctly if opened in Microsoft Excel. Some of the columns, including 'prompt response' may not format correctly if opened in Microsoft Excel due to the 32,000 cell character limit. If exceed the .csv will automatically shift to use a number of cells and the formatting will reflect this. For in depth reporting and analysis we recommend downloading the raw .csv file for use in using a proper .csv viewer such as CSView.
Spreadsheets of reported feedback can be found in the root as well as individual collection folders. These reports that are not in folders are the global exported feedback covered in the previous section. This report will not be tied to any specific KC but will look across all KCs.
They are the same format of spreadsheet.
AI Health Reports
As well as being able to export feedback, an auditor or administrator will be able to 'Export sync health report'.
These reports are saved as .json files in the Atlas AI Auditing workspace but within a separate folder to the Feedback Reports
Each KC's sync health report will have its own folder
The report is dated as part of the file name
And the .json text will look like the below, detailing:
- Total KC Files
- Total Files in Search
- Total files with indexing discrepancies
- Files Synced in List (list items indexed)
These health reports aim to provide a high-level overview of the syncing health of the index, and as such will not detail which items are failing.
Within the AI Auditing workspace, on the left-hand side site navigation you will see an option called 'Knowledge Collection Sync Logs'. This opens a list showing all sync activities.
If you click the title of this list item, you will see the full view and will see that within 'Detailed Status' there is information showing the Sync ID, processed, deleted, skipped and failed documents. However, to find out which documents these are, you will need to open the .csv attachment at the bottom of the item.
If you open or download this .csv file you will see the logs from the syn which detail the success and errors. These contain the columns:
- Skipped (True/False)
- Retries
- Site ID
- List ID
- File Name
- File Sizr
- File Ref (URL)
- Item ID
- ItemUniqueID
- Error Message
From this error message you will be able to diagnose why certain files may not have synced properly and been indexed. Very large files, unsupported file types, empty documents, or corrupted files are usually the most common problems.
| Error Message |
| File contains corrupted data. |
| [Empty-Content]: Document not found after indexing. Document must be empty or nothing extracted. |
Summary
As you will find out, functionally it's very easy to produce and access feedback and health reports and review the data collected.
It's very important that AI Users leave feedback on responses to questions when validating AI tooling and KC settings. The more responses which are left, the deeper date collated and better auditing abilities you will have at your disposal.
We always advise that a process for auditing the Atlas AI tooling is established, this includes; who is auditing and who needs permissions, when they are auditing and how often, how they aim to audit and what specific markers they are looking for, and, importantly, to utilize the auditing abilities to ensure that the responses are meeting expectations and that the settings are optimized to ensure best-in-class responses.
There are 2 types of auditing - technical auditing (tokens, number of responses, settings indexing health) and business auditing (number of negative responses, reasons given and comments), and both these auditing types should bridge and work together to ensure that the AI tooling is achieving the best results it can to help boost further the Return On Investment.
If you ever have any questions please get in contact with your designated Atlas representatives today.
Comments
0 comments
Please sign in to leave a comment.