A 3D AI meeting assistant that joins your video calls, responds to questions in real-time, and takes meeting notes automatically.
Demo-480p.mov
(A)Intern features a 3D virtual AI assistant named Iris that can:
Join meetings on platforms like Google Meet by simply providing a URL
Answer questions in real-time with natural voice responses and lifelike expressions
Access and reference uploaded company documents for contextual responses
Take automated meeting notes and generate summaries
The project consists of four main components:
/avatar- 3D avatar rendering and animation system/backend- Express server handling AI, meeting management, and data processing/dashboard- React-based user interface for managing meetings and documents/video-stream- Video streaming service for meeting integration
- Ready Player Me for 3D modeling
- Mixamo for natural animations
- ElevenLabs for voice synthesis
- Rhubarb Lip Sync for lip synchronization
- Three.js for 3D rendering
- Recall.ai API for meeting bot capabilities
- Custom video streaming solution for Three.js compatibility
- Real-time transcription and response system
- OpenAI for natural language processing
- React/Vite frontends
- Express.js backend
- Supabase database
- WebSocket communication
The system works through the following flow:
- User submits a meeting URL through the dashboard
- Meeting bot joins the call via Recall.ai
- Questions are transcribed in real-time
- Backend processes queries with relevant document context
- AI generates responses with voice synthesis and lip-sync
- Avatar stream is piped back into the meeting
The current system has approximately 10 seconds of latency due to our unique technical solution:
- Recall's headless browser can't directly render Three.js
- Solution: Convert avatar to video stream first
- Trade-off: Higher latency for better visual experience
- Node.js
- pnpm
- ngrok (for local development with Recall.ai)
When testing the meeting integration locally, you'll need ngrok to expose your local server to the internet. This is required because Recall.ai needs to communicate with publicly accessible endpoints.
- Install ngrok: https://ngrok.com/download
- Start ngrok on port 3001 (or your backend port):
ngrok http 3001
- Update your environment variables to use the ngrok URL:
- In
backend/.env, setPUBLIC_URLto your ngrok URL - In
video-stream/.env, update the WebSocket URL accordingly
- In
- Clone the repository
- Install dependencies in each directory:
cd avatar && pnpm install cd ../backend && pnpm install cd ../dashboard && pnpm install cd ../video-stream && pnpm install
- Configure environment variables (see
.env.examplein each directory) - Start the development servers