2024 · 10 citations · 17 references
ComputingComputer ScienceComputational LinguisticsMachine LearningNatural Language ProcessingAbstract Syntax TreesDeep LearningGrammarLanguage ModelingSyntactic AnalysisSyntax ParsingAbstract Syntax Tree RepresentationCode TranslationEncoder-decoder ArchitecturesLanguage Model PretrainingNeural Language ModelsSummarization
Pretrained models for code have exhibited promising performance across various code-related tasks, such as code summarization, code completion, code translation, and bug detection. However, despite their success, the majority of current models still represent code as a token sequence, which may not adequately capture the essence of the underlying code structure.
17
Yue Wang, Weishi Wang, Shafiq Joty et al. · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · 2021 · 1.3K citations · Full text
ZeRO: Memory optimizations Toward Training Trillion Parameter Models
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase et al. · 2020 · 741 citations
Unified Pre-training for Program Understanding and Generation
Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray et al. · 2021 · 581 citations · Full text