Computer Vision Framework for Human Motion Analysis
Vibrio is a computer vision framework designed to analyze human motion in video footage. The framework implements a modular pipeline approach for detecting, tracking, and analyzing humans in videos, with specific focus on motion analysis through multiple optical methods.
The Vibrio framework is structured as a modular pipeline with the following components:
┌───────────────┐ ┌───────────────┐ ┌───────────────┐ ┌───────────────┐
│ Human │ │ Object │ │ Speed │ │ Physics │
│ Detection ├─────►│ Tracking ├─────►│ Estimation ├─────►│ Verification │
└───────────────┘ └───────────────┘ └───────────────┘ └───────────────┘
│
┌────────────────────────────────────────────────────────────────────┘
│
▼
┌───────────────┐ ┌───────────────┐ ┌───────────────┐
│ Optical │ │ Pose │ │ Result │
│ Analysis │◄─────┤ Detection │ │ Visualization│
└───────────────┘ └───────────────┘ └───────────────┘
The framework processes videos through sequential stages:
- Input: Video frames are extracted and processed one by one
- Detection: Humans are detected in each frame using deep learning models
- Tracking: Detected humans are tracked across frames to maintain identity
- Analysis: Multiple analyses are performed on tracked objects:
- Speed estimation based on positional changes
- Physics-based verification of calculated speeds
- Optical analysis for detailed motion characteristics
- Optional pose detection for biomechanical analysis
- Output: Results are saved to disk and can be visualized
The detection module uses YOLOv8, an object detection network, specifically configured to identify humans in video frames.
Technical Details:
- Model: YOLOv8 (nano or larger variants)
- Input: RGB image frame
- Output: List of bounding boxes with confidence scores
- Performance:
- Inference speed: ~30-100 FPS depending on hardware and model size
- Detection threshold configurable (default: 0.5)
Implementation:
detector = HumanDetector(
model_path="yolov8n.pt", # Smaller model for speed
conf_threshold=0.5,
device="cuda:0" # Uses GPU if available
)
detections = detector.detect(frame)The tracking module implements a Kalman filter-based tracking system to maintain the identity of detected objects across frames.
Technical Details:
- Algorithm: Kalman filter with Hungarian algorithm for assignment
- State Space Model: Position, velocity, and bounding box dimensions
- Assignment: IoU (Intersection over Union) based matching
Mathematical Foundation:
State representation in the Kalman filter:
Where:
-
$c_x, c_y$ are the center coordinates of the bounding box -
$w, h$ are the width and height of the bounding box -
$\dot{c_x}, \dot{c_y}$ are the velocity components -
$\dot{s}$ is the scale change rate
Implementation:
tracker = HumanTracker(
max_age=30, # Maximum frames to keep a track without matching
min_hits=3, # Minimum detections before track is established
iou_threshold=0.3 # Threshold for detection-track association
)
tracks = tracker.update(detections, frame_idx)The speed estimator calculates the velocity of tracked objects based on positional changes between frames.
Technical Details:
- Input: Track history with positions and current frame index
- Method: Pixel displacement with optional calibration
- Output: Estimated speed in km/h
Mathematical Foundation:
The basic speed calculation follows:
Where:
-
$v$ is velocity in m/s (converted to km/h) -
$d_{pixels}$ is distance in pixels -
$r_{calibration}$ is the pixel-to-meter ratio -
$fps$ is frames per second of the video
For uncalibrated video, a default pixel-to-meter ratio is used, which may introduce systematic error in the absolute speed values.
Implementation:
speed_estimator = SpeedEstimator(calibration=None)
tracks = speed_estimator.estimate(tracks, frame_idx=frame_idx, fps=video_fps)The physics verifier applies domain-specific constraints to validate speed estimates.
Technical Details:
- Method: Rule-based verification using physical constraints
- Context sensitivity: Different constraints for different activities
Constraint Examples:
| Activity | Max Speed | Max Acceleration | Notes |
|---|---|---|---|
| Walking | 7 km/h | 2 m/s² | Normal walking pace |
| Running | 45 km/h | 10 m/s² | Elite sprinters |
| Cycling | 80 km/h | 4 m/s² | Racing cyclists on flat terrain |
Implementation:
verifier = PhysicsVerifier(context='general')
tracks = verifier.verify(tracks)The optical analyzer implements advanced methods for detailed motion analysis using computer vision techniques.
Implemented Methods:
-
Optical Flow Analysis
- Algorithm: Farneback dense optical flow
- Outputs: Motion magnitude, direction, coherence
- Applications: Detecting direction and intensity of movement
-
Motion Energy Analysis
- Algorithm: Motion History Images (MHI)
- Outputs: Motion energy, active regions, temporal signatures
- Applications: Activity detection, motion segmentation
-
Neuromorphic Camera Simulation
- Algorithm: Event-based threshold crossing detection
- Outputs: Events density, polarity ratio
- Applications: High temporal precision motion detection
-
Texture and Gradient Analysis
- Algorithm: Gabor filters and Local Binary Patterns
- Outputs: Texture scores, dominant orientations
- Applications: Surface and micro-motion detection
-
Shadow and Illumination Analysis
- Algorithm: HSV-based shadow detection
- Outputs: Shadow area, movement direction
- Applications: Additional motion cues from lighting changes
Mathematical Foundations:
For optical flow, the algorithm solves:
Through Taylor expansion and additional constraints, this yields a flow field:
Where:
-
$\nabla I$ is the spatial gradient -
$\mathbf{v}$ is the velocity vector -
$I_t$ is the temporal gradient
Implementation:
analyzer = OpticalAnalyzer(output_dir='results/optical_analysis')
results = analyzer.analyze_video(
video_path,
methods=['optical_flow', 'motion_energy']
)# Basic analysis of a video
python main.py --input video.mp4 --output results/
# Batch processing of multiple videos
python batch_analyze.py --input_dir videos/ --output_dir results/batch --use_optical
# Analysis of results
python analyze_results.py --results_dir results/batch# Analyze with specific optical methods
python batch_analyze.py --input_dir videos/ --use_optical --optical_methods optical_flow motion_energy
# Perform dedicated optical analysis
python analyze_optical.py --video_path video.mp4 --methods optical_flow neuromorphic texture_analysis| Parameter | Description | Default |
|---|---|---|
--detector |
Human detector model | yolov8n |
--confidence |
Detection confidence threshold | 0.4 |
--use_pose |
Enable pose estimation | False |
--use_optical |
Enable optical analysis | False |
--device |
Processing device | cuda:0 or cpu |
Vibrio generates several types of outputs that can be analyzed:
-
Annotated Videos
- Bounding boxes around detected humans
- Track IDs and estimated speeds
- Visual overlays of optical analysis results
-
Quantitative Data
- CSV files with frame-by-frame tracking data
- Speed profiles for each tracked object
- Optical analysis metrics time series
-
Visualizations
- Speed distribution graphs
- Motion heatmaps
- Comparative analysis between subjects
The plot above shows an example speed profile from tracking analysis, demonstrating the acceleration, steady-state, and deceleration phases of human motion. Statistical analysis of such profiles enables quantitative comparison between different subjects or activities.
Optical flow analysis reveals motion patterns not immediately obvious in the raw video. The color-coded visualization shows direction (hue) and magnitude (saturation) of motion, providing insights into the distribution and coherence of movement.
The following visualizations demonstrate the framework's capabilities on real-world footage:
For the downhill mountain biking footage, we performed a comprehensive multi-method analysis, combining visual annotations, optical analysis methods, and quantitative metrics:
The optical flow analysis reveals detailed motion patterns:
The plots show:
- Top: Flow magnitude (mean, max, and standard deviation)
- Middle: Motion direction in radians
- Bottom: Motion coherence indicating coordinated movement
The motion energy metrics provide insights into activity intensity:
The plots demonstrate:
- Top: Motion energy over time, showing intensity variations
- Bottom: Number of active regions, correlating with movement complexity
The frequency domain analysis quantifies motion patterns:
This chart shows dominant motion frequencies and amplitudes in the rider's movement.
This visualization shows the normalized metrics from different analysis methods, allowing direct comparison between optical flow, motion coherence, and energy measurements, highlighting how these metrics correlate during different motion phases.
Vibrio has been tested on a variety of high-speed human activities, demonstrating its versatility:
Key Findings: Smooth acceleration phases followed by high-G turns with distinctive weight shifts. Analysis detected characteristic patterns in the skier's weight distribution during turns and rapid direction changes.
Key Findings: Extreme speeds (100+ km/h) with minimal lateral movement and tightly controlled body position. Optical analysis revealed exceptionally high motion coherence values indicating the precise technique required at these speeds.
These visualizations demonstrate how multiple optical methods can extract rich information about motion characteristics that go beyond simple speed measurements, revealing the biomechanical aspects of human performance across different activities.
The framework's performance depends on several factors:
-
Hardware Requirements
- CPU-only: Analysis runs at reduced speed (5-10 FPS)
- GPU-accelerated: Real-time or faster analysis possible (30+ FPS)
-
Model Selection Trade-offs
- Smaller models (yolov8n): Faster but less accurate detection
- Larger models (yolov8x): More accurate but slower detection
-
Processing Resolution
- Higher resolution: Better accuracy but slower processing
- Lower resolution: Faster processing but reduced accuracy
-
Speed Estimation Accuracy
- Without camera calibration, absolute speed values are approximate
- Best used for relative comparisons between subjects in the same video
-
Tracking Challenges
- Performance degrades with occlusions and crowded scenes
- Rapid camera movement can disrupt tracking continuity
-
Optical Analysis Requirements
- Some methods require minimum video quality and resolution
- Performance is scene-dependent (lighting, background complexity)
-
Jocher, G., Chaurasia, A., & Qiu, J. (2023). YOLO by Ultralytics [Computer software]. https://github.com/ultralytics/ultralytics
-
Welch, G., & Bishop, G. (1995). An introduction to the Kalman filter. University of North Carolina at Chapel Hill, Department of Computer Science.
-
Farnebäck, G. (2003). Two-frame motion estimation based on polynomial expansion. In Scandinavian conference on Image analysis (pp. 363-370).
-
Bradski, G., & Kaehler, A. (2008). Learning OpenCV: Computer vision with the OpenCV library. O'Reilly Media, Inc.
-
Kuhn, H. W. (1955). The Hungarian method for the assignment problem. Naval research logistics quarterly, 2(1‐2), 83-97.
This project is licensed under the MIT License - see the LICENSE file for details.