arXiv (Cornell University) · 2021 · 11 citations · 10 references
ComputingComputer ScienceArtificial IntelligenceGenderMachine LearningAttention NetworksBiasDeep LearningGender BiasGender DifferencesAttentionAttention EnhancementAttention MechanismsBias DetectionGender Bias MitigationNormalizationTransformer ModelsTransformer-based Prediction Models
In this paper, we propose a novel gender bias detection method by utilizing attention map for transformer-based models. We 1) give an intuitive gender bias judgement method by comparing the different relation degree between the genders and the occupation according to the attention scores, 2) design a gender bias detector by modifying the attention module, 3) insert the gender bias detector into different positions of the model to present the internal gender bias flow, and 4) draw the consistent gender bias conclusion by scanning the entire Wikipedia, a BERT pretraining dataset. We observe that 1) the attention matrices, Wq and Wk introduce much more gender bias than other modules (including the embedding layer) and 2) the bias degree changes periodically inside of the model (attention matrix Q, K, V, and the remaining part of the attention layer (including the fully-connected layer, the residual connection, and the layer normalization module) enhance the gender bias while the averaged attentions reduces the bias).
10
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov et al. · arXiv (Cornell University) · 2020 · 21.6K citations · Full text
Language Models are Few-Shot Learners
T. B. Brown, Benjamin Mann, Nick Ryder et al. · arXiv (Cornell University) · 2020 · 3K citations · Full text
Deformable DETR: Deformable Transformers for End-to-End Object Detection
Xizhou Zhu, Weijie Su, Lewei Lu et al. · arXiv (Cornell University) · 2020 · 1.9K citations · Full text