Best practices to trace Azure openAI + Enterprise Data + RAG Applications

Best practices to trace Azure openAI + Enterprise Data + RAG Applications with Alert AI Security Analytics

Tracing RAG (Retrieval Augmented Generation) requests in Azure OpenAI Monitor tables primarily involves leveraging Azure Application Insights and configuring your RAG application to send telemetry data.

1. Enable Application Insights:

- Ensure you have an Azure Application Insights resource associated with your Azure OpenAI deployment or the service hosting your RAG application. This is the central location for collecting and analyzing telemetry.
- If you are using Azure AI Foundry, you can associate an Application Insights resource through the Foundry portal.

2. Instrument Your RAG Application:

- OpenTelemetry and OpenAI SDK: Utilize OpenTelemetry and the OpenAI SDK to instrument your RAG application. This allows you to capture detailed information about each stage of the RAG process.
- Telemetry Packages: Employ OpenAI telemetry packages or integrate OpenTelemetry directly to send traces to your Application Insights instance.
- Custom Telemetry: For more granular control, implement custom telemetry within your RAG application to capture specific events, such as:
  - - User query submission.
    - Retrieval of relevant documents from Azure AI Search.
    - Prompt construction with retrieved context.
    - LLM call for response generation.
    - Final response delivery.

3. Configure Diagnostic Settings:

- In Azure Monitor, configure diagnostic settings for your Azure OpenAI resource and potentially other services involved in your RAG workflow (e.g., Azure AI Search).
- Route resource logs to your Application Insights workspace for comprehensive monitoring.

4. Analyze Traces in Application Insights:

- Transaction Search: Use the Transaction Search feature in Application Insights to find specific RAG requests and examine their end-to-end flow.
- Application Map: Visualize the interactions between different components of your RAG system using the Application Map to identify bottlenecks or issues.
- Custom Events and Metrics: If you implemented custom telemetry, query and analyze these events and metrics to gain insights into specific RAG stages and their performance.
- Live Metrics Stream: Monitor your RAG application’s performance in real-time using the Live Metrics Stream.

Auto configure Alert AI “AI security” for Azure OpenAI

Alert AI “Secure AI Anywhere” AI security Gateway & AI Security Analytics, unified AI security AI risk management, seamlessly integrates and Auto configures with Azure OpenAI and provides 500+ detections with Autonomous AI Security AI governance, AI Resilience, AI Polices and Controls.

By following these steps, you can effectively trace RAG requests in Azure OpenAI Monitor tables, gaining valuable insights into the performance, behavior, and potential issues within your RAG application.

Azure AI Search Debug Logs and Monitoring

1. Security analytics in Azure AI Search:

Azure AI Search offers easy to develop Enterprise RAG Services within the Azure portal. ALert AI security analytics allows for the examination of documents as they are processed by an indexer and skillset.

Enforce Security Policy issues: Prevent loss of information and enforce complaince polices with both built-in and custom keywords within a skillset (Keyword list).
Incrementally enrich documents: and test/validate changes before committing them for any AI policy violations.

2. Monitoring Content Retrieved by Azure AI Search:

To monitor the content retrieved by Azure AI Search, you need to enable and configure diagnostic logging.

Enable Diagnostic Logging:

Configure diagnostic settings for your Azure AI Search service to send logs to a destination like Azure Storage, Log Analytics, or Azure Event Hubs.
Analyze Logs:
Azure Storage: Logs are stored as JSON files, which can be analyzed for raw query details and retrieved content information.

Log Analytics: Utilize Kusto Query Language (KQL) in Azure Monitor Logs to query and analyze the collected diagnostic data, including details about queries, indexing, and retrieved documents.

3. Monitoring Query and Retrieval:

Query Monitoring:
Diagnostic Logs: The diagnostic logs (specifically the “SearchServiceLogs” category) contain details about search queries, including the query text, parameters, and results.

Azure Activity Log: This log tracks subscription-level events, including control plane operations like API key usage, but does not provide details about individual search queries.
Retrieval Monitoring:
Diagnostic Logs: These logs provide insights into the documents retrieved in response to a query, offering details about the documents’ IDs and potentially other relevant metadata.

Debug Sessions: For a more granular view during development, Debug Sessions allow you to step through the skillset execution and observe the content of documents at various stages of enrichment and retrieval.

By automating Azure AI search with ALert AI for Monitoring and Security Analytics comprehensive protection and governance for production AI environments, you can effectively monitor and troubleshoot both the query process and the content retrieval in Azure AI Search.