Shijie Wu

Member of Technical Staff, Anthropic

I work on large language models at Anthropic. Before that I spent 2.5 years at Bloomberg AI, where I worked on BloombergGPT.

I did my PhD at Johns Hopkins University (CLSP) with Mark Dredze, on how multilingual encoders like mBERT learn cross-lingual representations without cross-lingual supervision. Along the way I interned at Facebook AI and worked on morphology and character-level transduction with Ryan Cotterell.

Google Scholar GitHub Publications

wushijie5 at gmail dot com

News

2024-05Joined Anthropic after 2.5 years at Bloomberg AI.
2023-03Released BloombergGPT, a 50B-parameter language model for finance.
2022-07Thesis, How Do Multilingual Encoders Learn Cross-lingual Representation?, is on arXiv.
2022-01Joined Bloomberg AI to work on large language models.
2021-11Defended my PhD thesis at Johns Hopkins.
2021-09Everything Is All It Takes accepted at EMNLP 2021.
2021-04Applying the Transformer to Character-level Transduction received an honorable mention for best short paper at EACL 2021.
2020-09Two papers accepted at EMNLP 2020: on explicit cross-lingual alignment and a survey of contextualized encoders.
2020-07Are All Languages Created Equal in Multilingual BERT? won best long paper at RepL4NLP 2020.
2020-04Passed my thesis proposal. Emerging Cross-lingual Structure in Pretrained Language Models, from my Facebook AI internship, accepted at ACL 2020.
2019-08Beto, Bentz, Becas, on the cross-lingual effectiveness of multilingual BERT, accepted at EMNLP 2019.
2019-05Two papers accepted at ACL 2019.

Publications

2024

Academics Can Contribute to Domain-Specialized Language Models
Mark Dredze, Genta Indra Winata, Prabhanjan Kambadur, Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, David S Rosenberg, Sebastian Gehrmann
EMNLP 2024 · paper

2023

BloombergGPT: A Large Language Model for Finance
Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David Rosenberg, Gideon Mann
arXiv 2023 · paper
MixCE: Training Autoregressive Language Models by Mixing Forward and Reverse Cross-Entropies
Shiyue Zhang, Shijie Wu, Ozan Irsoy, Steven Lu, Mohit Bansal, Mark Dredze, David Rosenberg
ACL 2023 · paper
Overcoming Catastrophic Forgetting in Massively Multilingual Continual Learning
Genta Winata, Lingjue Xie, Karthik Radhakrishnan, Shijie Wu, Xisen Jin, Pengxiang Cheng, Mayank Kulkarni, Daniel Preotiuc-Pietro
Findings of ACL 2023 · paper
Towards a Unified Multi-Domain Multilingual Named Entity Recognition Model
Mayank Kulkarni, Daniel Preotiuc-Pietro, Karthik Radhakrishnan, Genta Indra Winata, Shijie Wu, Lingjue Xie, Shaohua Yang
EACL 2023 · paper

2022

How Do Multilingual Encoders Learn Cross-lingual Representation?
Shijie Wu
PhD thesis, Johns Hopkins University · paper
Zero-shot Cross-lingual Transfer is Under-specified Optimization
Shijie Wu, Benjamin Van Durme, Mark Dredze
RepL4NLP 2022 · paper
Bernice: A Multilingual Pre-trained Encoder for Twitter
Alexandra DeLucia, Shijie Wu, Aaron Mueller, Carlos Aguirre, Philip Resnik, Mark Dredze
EMNLP 2022 · paper
Cross-lingual Few-Shot Learning on Unseen Languages
Genta Winata, Shijie Wu, Mayank Kulkarni, Thamar Solorio, Daniel Preotiuc-Pietro
AACL 2022 · paper

2021

Everything Is All It Takes: A Multipronged Strategy for Zero-Shot Cross-Lingual Information Extraction
Mahsa Yarmohammadi, Shijie Wu, Marc Marone, Haoran Xu, Seth Ebner, Guanghui Qin, Yunmo Chen, Jialiang Guo, Craig Harman, Kenton Murray, Aaron Steven White, Mark Dredze, Benjamin Van Durme
EMNLP 2021 · paper · code
Applying the Transformer to Character-level Transduction
Shijie Wu, Ryan Cotterell, Mans Hulden
EACL 2021 · paper · code · talk Honorable mention, best short paper
Differentiable Generative Phonology
Shijie Wu, Edoardo Maria Ponti, Ryan Cotterell
arXiv 2021 · paper
SIGMORPHON 2021 Shared Task on Morphological Reinflection: Generalization Across Languages
Tiago Pimentel, Maria Ryskina, Sabrina J. Mielke, Shijie Wu, et al.
SIGMORPHON 2021 · paper

2020

Do Explicit Alignments Robustly Improve Multilingual Encoders?
Shijie Wu, Mark Dredze
EMNLP 2020 · paper · code · talk
Which *BERT? A Survey Organizing Contextualized Encoders
Patrick Xia, Shijie Wu, Benjamin Van Durme
EMNLP 2020 · paper
Are All Languages Created Equal in Multilingual BERT?
Shijie Wu, Mark Dredze
RepL4NLP 2020 · paper · talk Best long paper
Emerging Cross-lingual Structure in Pretrained Language Models
Alexis Conneau, Shijie Wu, Haoran Li, Luke Zettlemoyer, Veselin Stoyanov
ACL 2020 · paper
The Paradigm Discovery Problem
Alexander Erdmann, Micha Elsner, Shijie Wu, Ryan Cotterell, Nizar Habash
ACL 2020 · paper
The SIGMORPHON 2020 Shared Task on Multilingual Grapheme-to-Phoneme Conversion
Kyle Gorman, Lucas F.E. Ashby, Aaron Goyzueta, Arya McCarthy, Shijie Wu, Daniel You
SIGMORPHON 2020 · paper
SIGMORPHON 2020 Shared Task 0: Typologically Diverse Morphological Inflection
Ekaterina Vylomova, Jennifer White, Elizabeth Salesky, Sabrina J. Mielke, Shijie Wu, et al.
SIGMORPHON 2020 · paper

2019

Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT
Shijie Wu, Mark Dredze
EMNLP 2019 · paper
Exact Hard Monotonic Attention for Character-Level Transduction
Shijie Wu, Ryan Cotterell
ACL 2019 · paper · code
Morphological Irregularity Correlates with Frequency
Shijie Wu, Ryan Cotterell, Timothy O’Donnell
ACL 2019 · paper
A Simple Joint Model for Improved Contextual Neural Lemmatization
Chaitanya Malaviya, Shijie Wu, Ryan Cotterell
NAACL 2019 · paper
The SIGMORPHON 2019 Shared Task: Morphological Analysis in Context and Cross-Lingual Transfer for Inflection
Arya D. McCarthy, Ekaterina Vylomova, Shijie Wu, et al.
SIGMORPHON 2019 · paper

2018

Hard Non-Monotonic Attention for Character-Level Transduction
Shijie Wu, Pamela Shapiro, Ryan Cotterell
EMNLP 2018 · paper

Old posts from X, 2019–2024