[OPIK-1834] [P SDK] ConversationalCoherence conversation metric #2579
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…entation. - Introduced `ConversationalCoherenceMetric` for evaluating coherence of conversation exchanges within sliding windows. - Implemented templates and schema for constructing evaluation queries and parsing responses. - Added helper methods to generate sliding windows of conversation turns. - Included Pydantic models for validating and handling evaluation responses. - Integrated scoring calculation and irrelevancy extraction logic for detailed analysis.
SDK Unit Tests Results636 tests 635 ✅ 27s ⏱️ For more details on these errors, see this check. Results for commit bf0da27. ♻️ This comment has been updated with latest results. |
<
6104
a href="/comet-ml/opik/pull/2579/commits/bf0da27f82f0bcd5668a73e48a385258254217af" class="Link--secondary">bf0da27
…ss metrics. - Added `ConversationTurn` class for structured representation of conversation exchanges. - Implemented `build_conversation_turns` and `merge_turns` utilities for processing conversational data. - Refactored window-based scoring logic to use conversation turns, improving clarity and modularity. - Introduced comprehensive tests for conversational coherence and session completeness metrics, covering error scenarios and ensuring robustness. - Improved error handling and logging across metric computation functions.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Details
Conversational Coherence Score: whether the conversation session felt like a natural, adaptive, helpful interaction.
The metric calculated as following:
The
ConversationalCoherenceMetricfirst constructs a sliding windows of turns for each turn, before using an LLM to determine whether the last turn in each sliding window has an "assistant" content that is relevant to the previous conversational context found in the sliding windowTesting
Added related unit tests
Documentation
Added related docstrings