https://github.com/thu-ml/SpargeAttn

\n","updatedAt":"2025-02-26T05:57:34.166Z","author":{"_id":"66c0a08bac74db25de8427ec","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/66c0a08bac74db25de8427ec/9D-piDBZqSt6KNkHImmkv.jpeg","fullname":"Jintao Zhang","name":"jt-zhang","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":12}},"numEdits":1,"identifiedLanguage":{"language":"en","probability":0.8365098237991333},"editors":["jt-zhang"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/66c0a08bac74db25de8427ec/9D-piDBZqSt6KNkHImmkv.jpeg"],"reactions":[],"isReport":false}},{"id":"67bfc0e73859b01651fab8c6","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":242},"createdAt":"2025-02-27T01:33:27.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"This is an automated message from the [Librarian Bot](https://huggingface.co/librarian-bots). I found the following papers similar to this paper. \n\nThe following papers were recommended by the Semantic Scholar API \n\n* [LServe: Efficient Long-sequence LLM Serving with Unified Sparse Attention](https://huggingface.co/papers/2502.14866) (2025)\n* [Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity](https://huggingface.co/papers/2502.01776) (2025)\n* [SparAMX: Accelerating Compressed LLMs Token Generation on AMX-powered CPUs](https://huggingface.co/papers/2502.12444) (2025)\n* [MoBA: Mixture of Block Attention for Long-Context LLMs](https://huggingface.co/papers/2502.13189) (2025)\n* [HALO: Hadamard-Assisted Lower-Precision Optimization for LLMs](https://huggingface.co/papers/2501.02625) (2025)\n* [A Proximal Operator for Inducing 2:4-Sparsity](https://huggingface.co/papers/2501.18015) (2025)\n* [AttentionEngine: A Versatile Framework for Efficient Attention Mechanisms on Diverse Hardware Platforms](https://huggingface.co/papers/2502.15349) (2025)\n\n\n Please give a thumbs up to this comment if you found it helpful!\n\n If you want recommendations for any Paper on Hugging Face checkout [this](https://huggingface.co/spaces/librarian-bots/recommend_similar_papers) Space\n\n You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: `@librarian-bot recommend`","html":"

This is an automated message from the Librarian Bot. I found the following papers similar to this paper.

The following papers were recommended by the Semantic Scholar API

Please give a thumbs up to this comment if you found it helpful!

If you want recommendations for any Paper on Hugging Face checkout this Space

You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: \n\n@librarian-bot\n\t recommend

\n","updatedAt":"2025-02-27T01:33:27.430Z","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":242}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.7143100500106812},"editors":["librarian-bot"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2502.18137","authors":[{"_id":"67be8443ed8e258c0f70063a","user":{"_id":"66c0a08bac74db25de8427ec","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/66c0a08bac74db25de8427ec/9D-piDBZqSt6KNkHImmkv.jpeg","isPro":false,"fullname":"Jintao Zhang","user":"jt-zhang","type":"user"},"name":"Jintao Zhang","status":"claimed_verified","statusLastChangedAt":"2025-02-26T08:25:57.704Z","hidden":false},{"_id":"67be8443ed8e258c0f70063b","user":{"_id":"6329bdbbde087eac2921e6a9","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1663679904323-noauth.jpeg","isPro":false,"fullname":"Xiangchendong","user":"Xiang-cd","type":"user"},"name":"Chendong Xiang","status":"admin_assigned","statusLastChangedAt":"2025-02-26T08:49:29.341Z","hidden":false},{"_id":"67be8443ed8e258c0f70063c","name":"Haofeng Huang","hidden":false},{"_id":"67be8443ed8e258c0f70063d","name":"Jia Wei","hidden":false},{"_id":"67be8443ed8e258c0f70063e","user":{"_id":"65d5a000ec7e31555e4db57e","avatarUrl":"/avatars/aab8319fbaffdd53faff59a40ca5a5ea.svg","isPro":false,"fullname":"Neil Shi","user":"shineil","type":"user"},"name":"Haocheng Xi","status":"admin_assigned","statusLastChangedAt":"2025-02-26T08:49:45.446Z","hidden":false},{"_id":"67be8443ed8e258c0f70063f","name":"Jun Zhu","hidden":false},{"_id":"67be8443ed8e258c0f700640","user":{"_id":"65fcad0ba0d7adc40b54fac2","avatarUrl":"/avatars/7564b5642378fddb46ec3b5ae57c0402.svg","isPro":false,"fullname":"Jianfei Chen","user":"surfingtomchen","type":"user"},"name":"Jianfei Chen","status":"admin_assigned","statusLastChangedAt":"2025-02-26T08:49:52.550Z","hidden":false}],"publishedAt":"2025-02-25T12:02:17.000Z","submittedOnDailyAt":"2025-02-26T00:34:57.351Z","title":"SpargeAttn: Accurate Sparse Attention Accelerating Any Model Inference","submittedOnDailyBy":{"_id":"66c0a08bac74db25de8427ec","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/66c0a08bac74db25de8427ec/9D-piDBZqSt6KNkHImmkv.jpeg","isPro":false,"fullname":"Jintao Zhang","user":"jt-zhang","type":"user"},"summary":"An efficient attention implementation is essential for large models due to\nits quadratic time complexity. Fortunately, attention commonly exhibits\nsparsity, i.e., many values in the attention map are near zero, allowing for\nthe omission of corresponding computations. Many studies have utilized the\nsparse pattern to accelerate attention. However, most existing works focus on\noptimizing attention within specific models by exploiting certain sparse\npatterns of the attention map. A universal sparse attention that guarantees\nboth the speedup and end-to-end performance of diverse models remains elusive.\nIn this paper, we propose SpargeAttn, a universal sparse and quantized\nattention for any model. Our method uses a two-stage online filter: in the\nfirst stage, we rapidly and accurately predict the attention map, enabling the\nskip of some matrix multiplications in attention. In the second stage, we\ndesign an online softmax-aware filter that incurs no extra overhead and further\nskips some matrix multiplications. Experiments show that our method\nsignificantly accelerates diverse models, including language, image, and video\ngeneration, without sacrificing end-to-end metrics. The codes are available at\nhttps://github.com/thu-ml/SpargeAttn.","upvotes":58,"discussionId":"67be8447ed8e258c0f70075f","projectPage":"https://github.com/thu-ml/SageAttention","githubRepo":"https://github.com/thu-ml/SpargeAttn","ai_summary":"SpargeAttn is a universal sparse and quantized attention method that accelerates diverse models by predicting the attention map and using an online softmax-aware filter to skip unnecessary matrix multiplications.","ai_keywords":["attention","sparse attention","quantized attention","matrix multiplications","online filter","softmax-aware filter","language generation","image generation","video generation"],"githubStars":655},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"66c0a08bac74db25de8427ec","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/66c0a08bac74db25de8427ec/9D-piDBZqSt6KNkHImmkv.jpeg","isPro":false,"fullname":"Jintao Zhang","user":"jt-zhang","type":"user"},{"_id":"6705282fb019958d654d5c3d","avatarUrl":"/avatars/4db7d07344f10beafea26d042f38c531.svg","isPro":false,"fullname":"F sixxx6","user":"sixxx6","type":"user"},{"_id":"67052ac55c89251d79ebef91","avatarUrl":"/avatars/d65c44b4d17a33fd6cbc8461fb402ad4.svg","isPro":false,"fullname":"tsingte","user":"thu-zzte","type":"user"},{"_id":"67be8630e7b05f9e43b21204","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/c7jrbuqdaZfWZlj1ASJoN.png","isPro":false,"fullname":"ice","user":"akjhnh","type":"user"},{"_id":"67be87b95a29257e4850edb8","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/RLrVf_p1KdbIrlcyUp31O.png","isPro":false,"fullname":"Yuwei Huang","user":"hyw498169842","type":"user"},{"_id":"67441ead33e10fc6d42bbfab","avatarUrl":"/avatars/166019aa1971666ed27f425e65a166f2.svg","isPro":false,"fullname":"夏雨阳","user":"SunnyXia3579","type":"user"},{"_id":"66c09891e61ccd71d7660447","avatarUrl":"/avatars/890afa8794737fd8452a9fb60348cfd9.svg","isPro":false,"fullname":"Yuanli","user":"Macly","type":"user"},{"_id":"67be8bfad1e0f62eb8edba08","avatarUrl":"/avatars/c5c046fa4f87a630382eb575a6f3efd5.svg","isPro":false,"fullname":"jiaqi Tian","user":"jiaqiTian","type":"user"},{"_id":"67be8ba6c57d8197f71dbfce","avatarUrl":"/avatars/2d4ce9188170767ac0d5ea2d4de17115.svg","isPro":false,"fullname":"JiachiWang","user":"jc-wang","type":"user"},{"_id":"646ce31ffe25e5f8d973333d","avatarUrl":"/avatars/bc73b91ba23ad18bb38100d3e5ff0e31.svg","isPro":false,"fullname":"Wenbi Li","user":"hnjylwb","type":"user"},{"_id":"6303c5041dd5d3c624836739","avatarUrl":"/avatars/7dbc3d6e894c2eed9a2fe4cef7c1ce4a.svg","isPro":false,"fullname":"Ayami I","user":"Ayakinokiki","type":"user"},{"_id":"670536e2516ab475d8bb68c7","avatarUrl":"/avatars/bd687566e1c738192f05285084e19fd0.svg","isPro":false,"fullname":"jiaye","user":"Tianqimingxing","type":"user"}],"acceptLanguages":["*"],"dailyPaperRank":3}">

Papers

arxiv:2502.18137

SpargeAttn: Accurate Sparse Attention Accelerating Any Model Inference

Published on Feb 25

· Submitted by

jt-zhang on Feb 26

#3 Paper of the day

Upvote

Authors:

Jintao Zhang ,

Chendong Xiang ,

Haocheng Xi ,

Jianfei Chen

Abstract

SpargeAttn is a universal sparse and quantized attention method that accelerates diverse models by predicting the attention map and using an online softmax-aware filter to skip unnecessary matrix multiplications.

AI-generated summary

An efficient attention implementation is essential for large models due to its quadratic time complexity. Fortunately, attention commonly exhibits sparsity, i.e., many values in the attention map are near zero, allowing for the omission of corresponding computations. Many studies have utilized the sparse pattern to accelerate attention. However, most existing works focus on optimizing attention within specific models by exploiting certain sparse patterns of the attention map. A universal sparse attention that guarantees both the speedup and end-to-end performance of diverse models remains elusive. In this paper, we propose SpargeAttn, a universal sparse and quantized attention for any model. Our method uses a two-stage online filter: in the first stage, we rapidly and accurately predict the attention map, enabling the skip of some matrix multiplications in attention. In the second stage, we design an online softmax-aware filter that incurs no extra overhead and further skips some matrix multiplications. Experiments show that our method significantly accelerates diverse models, including language, image, and video generation, without sacrificing end-to-end metrics. The codes are available at https://github.com/thu-ml/SpargeAttn.