Description
PowerInfer currently optimizes for LLMs (Large Language Models) that utilize the ReLU activation function, leveraging their internal activation locality. However, many of the trending models do not use ReLU activation, creating a significant gap in PowerInfer's applicability.
This ongoing issue tracks our efforts to onboard new LLMs, particularly those in high demand within the community, and to continually enhance our existing ReLU-based LLMs.
Onboarding Progress
We're actively fine-tuning models into ReLU sparse models:
- Mistral 7B (Now released as Bamboo)
Inviting broader participation, we're also:
- Releasing guidelines and reference implementations for converting LLMs to ReLU-based models.
- Open-sourcing our predictor training code post and during ReLU LLM fine-tuning.
Onboarding New Models
We recognize that fine-tuning upstream models is computationally intensive, and the requirement for high-quality data often surpasses our current capabilities. As such, we are actively seeking industrial collaborations to unlock more of PowerInfer's potential and bring state-of-the-art models to a wider audience. For direct inquiries and partnership discussions, please contact us at yzmizeyu@sjtu.edu.cn.
We will also focus on models that have garnered significant interest in our community 🌟. Your input and feedback are highly valued and encouraged! 💬👍
Metadata
Metadata
Assignees
Type
Projects
Status