A web application that allows you to compare responses from multiple AI models side by side. Send a single query and get responses from GPT-4, Claude, and Gemini simultaneously.
This tool helps you evaluate and compare the performance of different AI models by:
- Sending the same prompt to multiple models simultaneously
- Viewing responses side-by-side for easy comparison
- Rating responses across different categories (accuracy, clarity, etc.)
- Tracking your rating history and model performance
- Analyzing which models perform better in specific categories
- Flexible storage options with PostgreSQL or localStorage
- Data validation and cleanup tools
- Bulk data deletion capabilities
The main interface where you can enter queries and view responses from different AI models side by side.
Compare responses from different models in a clean, organized layout.
Interactive rating controls for evaluating responses across different categories.
Access and manage your past queries and their responses.
Comprehensive view of all ratings, organized by model.
Comprehensive view of all ratings, organized by category.
Add the ability to support different storage providers: Local Storage & PostgreSQL.
Add the ability to migrate data from one storage to another, validate the data in a storage, delete all the data in a storage layer.
- Compare responses from multiple AI models in real-time
- View response latency for each model
- Markdown formatting support for responses
- Query history with easy access to past comparisons
- Flexible storage options:
- PostgreSQL database for persistent storage
- Browser localStorage for offline usage
- Easy switching between storage types
- Data management tools:
- Data validation and automatic fixing
- Bulk data deletion
- Storage-specific operations
- Responsive design for desktop and mobile
- Dark/light mode support
- Enhanced UI with:
- Sticky header for easy navigation
- Light blue response sections for better readability
- Consistent card styling with subtle borders
- Full-width layout for better space utilization
- Improved query history display
- Unified storage selector in sidebar
- Rating system for responses:
- Thumbs up/down ratings for multiple categories
- Rating summary tables by model and category
- Visual indicators with color-coded backgrounds
- Interactive rating controls with hover effects
- Percentage-based rating summaries
-
GPT-4 (OpenAI)
- Most capable OpenAI model
- Version: gpt-4
-
Claude 3 Opus (Anthropic)
- Latest Claude model
- Version: claude-3-opus-20240229
-
Gemini Flash (Google)
- Quick response model
- Version: gemini-2.0-flash
- Node.js 18.x or higher
- npm or yarn
- PostgreSQL database (optional, for persistent storage)
- API keys for the following services:
- OpenAI API key
- Anthropic API key
- Google AI (Gemini) API key
- Clone the repository:
git clone https://github.com/artynuts/ai-model-comparison.git
cd ai-model-comparison- Install dependencies:
npm install- Create a
.env.localfile in the root directory with your API keys:
OPENAI_API_KEY=your_openai_key
ANTHROPIC_API_KEY=your_anthropic_key
GEMINI_API_KEY=your_google_api_key
DATABASE_URL=your_postgres_connection_string # Optional, for PostgreSQL storage- Run the development server:
npm run dev- Open http://localhost:3000 in your browser
-
Make sure you have Docker installed on your system.
-
Create a
.envfile in the root directory with your database configuration:
POSTGRES_USER=your_db_user
POSTGRES_PASSWORD=your_db_password
POSTGRES_DB=aimc_db
DATABASE_URL=postgresql://your_db_user:your_db_password@localhost:5432/aimc_db- Start the PostgreSQL container:
docker run --name aimc-postgres \
-e POSTGRES_USER=your_db_user \
-e POSTGRES_PASSWORD=your_db_password \
-e POSTGRES_DB=aimc_db \
-p 5432:5432 \
-d postgres:latest- Verify the container is running:
docker ps- To stop the container:
docker stop aimc-postgres- To start the container again:
docker start aimc-postgres- To remove the container:
docker rm aimc-postgresNote: Make sure to replace your_db_user and your_db_password with secure values of your choice. The database will be accessible on port 5432.
The application includes tools for managing your data:
-
Data Validation
- Located in the Settings page
- Checks for missing IDs, incorrect ordering, and other issues
- Automatically fixes common problems
- Works with both PostgreSQL and localStorage
-
Data Deletion
- Located in the Settings page
- Allows bulk deletion of all data
- Storage-specific deletion (PostgreSQL or localStorage)
- Confirmation dialog to prevent accidental deletion
- Enter your query in the text area
- Click "Compare Models" to send the query to all AI models
- View the responses side by side
- Rate responses using thumbs up/down for different categories
- Access your query history in the sidebar
- View rating summaries in the dedicated ratings page
- Click on past queries to view previous comparisons
- Choose your preferred storage method (PostgreSQL or localStorage) from the sidebar
- Next.js 14
- React
- TypeScript
- Tailwind CSS
- DaisyUI
- PostgreSQL (optional)
- OpenAI API
- Anthropic API
- Google Generative AI API
ai-model-comparison/
├── app/
│ ├── api/ # API routes
│ ├── components/ # React components
│ │ ├── ThumbsIcon.tsx # Shared rating icon component
│ │ ├── ThumbsRating.tsx # Rating control component
│ │ ├── StorageSelector.tsx # Storage type selector
│ │ ├── DataValidation.tsx # Data validation component
│ │ ├── DataDeletion.tsx # Data deletion component
│ │ └── ...
│ ├── context/ # React context
│ │ ├── StorageContext.tsx # Storage management
│ │ └── ...
│ ├── lib/
│ │ ├── storage/ # Storage providers
│ │ │ ├── PostgresStorageProvider.ts
│ │ │ ├── LocalStorageProvider.ts
│ │ │ └── StorageProvider.ts
│ │ └── utils/ # Utility functions
│ │ └── storage.ts
│ ├── history/ # History page
│ ├── ratings/ # Ratings summary page
│ ├── settings/ # Settings page
│ ├── types/ # TypeScript types
│ └── page.tsx # Main page
├── docker/ # Docker configuration files
├── public/ # Static assets
├── docker-compose.yml # Docker Compose configuration
└── package.json # Dependencies and scripts
The application supports rating responses across multiple categories:
- Accuracy: Were there any factual errors?
- Relevance: Did it fully answer the question?
- Completeness: Was anything missing?
- Conciseness: Was the response straight to the point?
- Unbiased: Did you detect any bias in the response?
Each category can be rated with thumbs up or down, and ratings are summarized in:
- Per-model view: Shows ratings grouped by AI model
- Per-category view: Shows ratings grouped by category
The application supports two storage methods:
-
PostgreSQL Database
- Persistent storage across sessions
- Suitable for production use
- Requires database setup
-
Browser localStorage
- Works offline
- No setup required
- Data persists in the browser
Switch between storage options using the selector in the sidebar. Your preference is remembered across sessions.
The application provides functionality to migrate data between PostgreSQL and localStorage:
-
Manual Migration
- Use the Data Migration tool in the Settings page
- Select the target storage type
- Click "Migrate Data" to start the process
- The tool will:
- Copy data from the source storage to the target storage
- Skip any duplicate entries
- Show progress and results of the migration
- List any queries that were skipped due to duplicates
-
Data Validation
- After migration, use the Data Validation tool to ensure:
- All records were transferred correctly
- Data integrity is maintained
- No duplicate entries exist
- All required fields are present
- After migration, use the Data Validation tool to ensure:
-
Troubleshooting
- If migration fails, check the browser console for error messages
- Use the Data Validation tool to identify and fix any issues
- The original data remains intact in the source storage
Note: When migrating large datasets, the process may take a few moments. The application will show a progress indicator during migration.
This project enforces 100% test coverage for all code. This ensures high code quality and reliability.
# Run all tests
npm test
# Run tests with coverage report
npm run test:coverage
# Check if coverage meets 100% threshold
npm run check-coverage- All new code must have 100% test coverage for:
- Statements
- Branches
- Functions
- Lines
The project uses Husky to enforce test coverage requirements:
- Pre-commit hook automatically runs tests and checks coverage
- Commits will be blocked if coverage is below 100%
- The hook will show which files need additional test coverage
- Fork the repository
- Create your feature branch (
git checkout -b feature/amazing-feature) - Commit your changes (
git commit -m 'Add amazing feature') - Push to the branch (
git push origin feature/amazing-feature) - Open a Pull Request
This project is licensed under the MIT License - see the LICENSE file for details.
Never commit your .env.local file or expose your API keys. They should be kept secret and secure.