This repository contains code and documentation from research on Human-AI Interaction, focusing on eye-tracking data analysis using VTNet.
- Eye-tracking Data Processing: Transform raw Tobii data (120 Hz) into VTNet-compatible formats
- Cross-validation Frameworks: Specialized CV approaches for different classification scenarios
- HPC Integration: Ready-to-use scripts for high-performance computing environments
- Documentation: Documents what files to look for in the lab server, how to use VTNet, HPCs, and more
Contains documentation on locating the necessary data files within the lab repository.
Tools for transforming raw eye-tracking data:
- 🧹 Data Cleaning & Formatting: Scripts to clean and standardize raw Tobii data
- 📊 Exploratory Data Analysis: Functions for visualizing and understanding eye-tracking patterns
- 🏗️ Dataset Construction: Tools for building within-task and across-task datasets
- 🛠️ Utility Functions: Including scanpath extraction and cyclic data splitting
Resources for high-performance computing:
- 📝 Slurm Templates: Ready-to-use templates for the
ubc_mlcluster - 📈 Statistical Analysis: Scripts for computing performance metrics from VTNet outputs
Implementations of VTNet for the within-task dataset:
- ⏱️ Multiple sequence length configurations (14s, 29s, and length 1000)
- 📄 Documentation on required VTNet modifications
⚠️ Important: Thest_pickle_loaderfunction'smax_lengthparameter is critical for adjusting sequence lengths
VTNet implementations for the evaluating per user VTNet dataset:
- 🔄 Cross-task prediction configurations
- 📄 Documentation on required model modifications
- 📝 The extra preprocessing done to make the within-tasks dataset to per user
While multiple directory structures are possible, here is our recommended setup:
VTNet/
├── dataset/ # Dataset pre-processed via our functions in `preprocessing`
├── utils.py # Utility functions used in VTNet
├── vislit.pickle # Pickle file for grouped CV in classifying visual literacy
├── readp.pickle # Pickle file for grouped CV in classifying reading proficiency
├── verbalwm.pickle # Pickle file for grouped CV in classifying verbal working memory
└── vtenv/ # vtenv environments, one per new length cut-off (ex. vtenv14)
├── VisLit/
│ ├── VTNet.py # VTNet modified to be trained on VisLit
│ └── run.sh # Bash script to execute VTNet for Visual Literacies tasks
├── ReadP/
│ ├── VTNet.py # VTNet modified to be trained on ReadP
│ └── run.sh # Bash script to execute VTNet for Reading Proficiency tasks
└── VerbalWM/
├── VTNet.py # VTNet modified to be trained on VerbalWM
└── run.sh # Bash script to execute VTNet for Verbal Working Memory tasks
- Python 3.8 or higher
- Conda (for environment setup)
-
Clone the repository
git clone https://github.com/ethanwongca/hai_work
-
Navigate to the project directory
cd hai_work -
Create and activate the Conda environment
conda env create -f environment.yml conda activate hai-env
VTNet is a neural architecture that processes eye-tracking data to predict user confusion and Alzheimer disease. It transforms sequential eye movement data into features useful for classification tasks.
When using VTNet, pay attention to the max_length parameter in the st_pickle_loader function. This parameter adjusts the model for different sequence lengths (e.g., from 1000 to 3000).
For a deeper understanding of the methods used in this repository, we recommend these papers:
-
Barral, O., Lallé, S., Guz, G., Iranpour, A., & Conati, C. (2020). Eye‑tracking to predict user cognitive abilities and performance for user‑adaptive narrative visualizations. In Proceedings of the 2020 International Conference on Multimodal Interaction (pp. 163–173).
-
Sims, S. D., & Conati, C. (2020). A neural architecture for detecting user confusion in eye‑tracking data. In Proceedings of the 2020 International Conference on Multimodal Interaction (pp. 15–23).
-
Sriram, H., Conati, C., & Field, T. (2023). Classification of Alzheimer's disease with deep learning on eye‑tracking data. In Proceedings of the 25th International Conference on Multimodal Interaction (pp. 104–113).
- UBC's Human Artificial Intelligence Interaction Lab