Concepedia

Attention Head Pruning

Attention Head Pruning is a model compression technique that selectively removes redundant or less informative attention heads from a transformer architecture to improve computational efficiency while preserving model performance.

16

Publications

2K

Citations

52

Authors

23

Institutions

Publications per year

2017–2026

16

Parents

Publications

16

Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned

Elena Voita, David Talbot, Fédor Moiseev et al. · 2019 · 1.1K citations · Full text

, , +12

SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruning

Hanrui Wang, Zhekai Zhang, Song Han · 2021 · 372 citations

, , +16

When BERT Plays the Lottery, All Tickets Are Winning

Sai Prasanna, Anna Rogers, Anna Rumshisky · 2020 · 115 citations · Full text

, , +9

Multi-Head Attention: Collaborate Instead of Concatenate

Jean-Baptiste Cordonnier, Andreas Loukas, Martin Jaggi · arXiv (Cornell University) · 2020 · 76 citations · Full text

, , +11

Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned

Elena Voita, David Talbot, Fédor Moiseev et al. · Zurich Open Repository and Archive (University of Zurich) · 2019 · 70 citations

, , +12

1–5 of 16

Authors

52

  1. SP
    Sai Prasanna

    University of Massachusetts Lowell, United States

    2 publications · 151 citations

    2

    h-index

  2. AR
    Anna Rogers

    University of Massachusetts Lowell, United States

    2 publications · 151 citations

    2

    h-index

  3. ZZ
    Zhekai Zhang

    Massachusetts Institute of Technology, United States

    2 publications · 384 citations

    2

    h-index

  4. EV
    Elena Voita

    University of Amsterdam, Netherlands

    2 publications · 1.1K citations

    2

    h-index

  5. HW
    Hanrui Wang

    Massachusetts Institute of Technology, United States

    2 publications · 384 citations

    2

    h-index

1–5 of 52

Institutions

23

  1. University of Amsterdam

    Amsterdam, The Netherlands

    2 h-index

    4 publications

    2.3K citations

  2. University of Edinburgh

    Edinburgh, United Kingdom

    2 h-index

    4 publications

    2.2K citations

  3. Moscow Institute of Physics and Technology

    Dolgoprudnyy, Russia

    2 h-index

    2 publications

    1.1K citations

  4. Massachusetts Institute of Technology

    Cambridge, United States

    2 h-index

    6 publications

    1.2K citations

  5. University of Zurich

    Zurich, Switzerland

    2 h-index

    2 publications

    1.1K citations

  6. Google (United States)

    Mountain View, United States

    1 h-index

    5 publications

    160 citations

  7. University of Massachusetts Lowell

    Lowell, United States

    1 h-index

    3 publications

    345 citations

  8. Harvard University

    Cambridge, United States

    1 h-index

    5 publications

    160 citations

1–8 of 23

Venues

  1. arXiv (Cornell University)

    Repository · 5 publications · 190 citations

    5

    h-index

  2. Journal of Energy Storage

    1 publications · 34 citations

    1

    h-index

  3. AI Open

    1 publications · 23 citations

    1

    h-index

  4. IEEE Transactions on Visualization and Computer Graphics

    TVCG · 1 publications · 41 citations

    1

    h-index

  5. Zurich Open Repository and Archive (University of Zurich)

    Repository · 1 publications · 70 citations

    1

    h-index

1–5