High-performance optimization engine for Large Language Models (LLMs) on Apple Silicon, combining Rust's performance with Python's ease of use.
Welcome to EvaOpt! This guide will help you get started with our project, even if you're new to programming.
EvaOpt is a tool that helps make artificial intelligence models (specifically, Large Language Models like ChatGPT) run faster and use less memory on Apple computers with M1/M2/M3 chips. Think of it like a compression tool that makes these AI models more efficient without losing their capabilities.
You'll need:
- A Mac computer with Apple Silicon (M1, M2, or M3 chip)
- Internet connection
- Basic familiarity with using the Terminal (don't worry, we'll guide you!)
-
Install Required Software
-
Install Python (version 3.9 or newer)
- Download from: https://www.python.org/downloads/
- During installation, make sure to check "Add Python to PATH"
-
Install Rust
- Open Terminal
- Copy and paste this command:
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
- Follow the on-screen instructions
-
-
Get the Project
- Open Terminal
- Run these commands one by one:
# Download the project git clone https://github.com/yourusername/evaopt.git # Go to project folder cd evaopt # Create a virtual environment (like a separate space for the project) python3 -m venv venv # Activate the virtual environment source venv/bin/activate # Install required packages pip install -r requirements.txt pip install -e .
Here's a simple example to get started:
# Create a new file called 'first_test.py' and add this code:
from evaopt import Optimizer, ModelConfig
# Set up basic configuration
config = ModelConfig(
model_type="llama2", # Type of AI model
quantization_bits=8, # How much to compress
use_fp16=True # Use faster processing
)
# Create an optimizer
optimizer = Optimizer(config)
# This is where you would optimize your model
# For example:
# optimized_model = optimizer.optimize_model(your_model)-
"Command not found" errors
- Problem: Python or Rust commands aren't recognized
- Solution: Make sure you've installed Python and Rust correctly and restart your Terminal
-
Installation errors
- Problem: Packages fail to install
- Solution: Try running:
pip install --upgrade pip pip install -r requirements.txt --no-cache-dir
-
Memory errors
- Problem: Process uses too much memory
- Solution: Try reducing the model size or increasing quantization_bits in the config
-
Import errors
- Problem: Can't import evaopt
- Solution: Make sure you're in the virtual environment:
source venv/bin/activate
If you encounter any issues not covered here, please:
- Check if your Mac has Apple Silicon (M1/M2/M3)
- Make sure all software is up to date
- Try restarting your computer
- Create an issue on our GitHub page
- 🚀 High-performance Rust core engine
- 🔢 Matrix optimization methods (SVD, Low-rank, Sparse)
- 📊 Model quantization (INT4/INT8)
- 💾 Smart memory management
- 🍎 Apple Silicon optimization
- macOS with Apple Silicon (M1/M2/M3)
- Python 3.9+
- Rust 1.75+
from evaopt import Optimizer, ModelConfig
# Configure optimizer
config = ModelConfig(
model_type="llama2",
quantization_bits=8,
use_fp16=True
)
# Create optimizer
optimizer = Optimizer(config)
# Optimize model
optimized_model = optimizer.optimize_model(model)MIT License
# Block-sparse optimization (1000x1000)
optimizer = Optimizer(config)
result = optimizer.optimize(matrix)
# Compression: 40.36%, Error: 0.000003# GPT-2 (Demo)
model = GPT2LMHeadModel(config)
# Llama-2-7b-chat
model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-chat-hf")- Block-sparse: 40% memory reduction, 0.001s processing time
- Matrix Compression: Up to 97.74% for rank-10 approximation
- LLM Inference: 26% speed improvement
- Memory Usage: Stable under 2GB for optimized models
- Block sizes: 16x16 to 128x128
- Adaptive threshold selection
- Fast block-wise processing
- Minimal accuracy loss
- SVD (Full/Truncated/Randomized)
- Low-rank approximation
- Sparse optimization
- Block-sparse compression
For more examples, check examples/ directory.