Yichu Zhou  
(周 逸初)

selected publications

1. 

   A Closer Look at How Fine-tuning Changes BERT

   Yichu Zhou, and Vivek Srikumar

   In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), May 2022

   Abs Bib PDF Code

   Given the prevalence of pre-trained contextualized representations in today’s NLP, there have been many efforts to understand what information they contain, and why they seem to be universally successful. The most common approach to use these representations involves fine-tuning them for an end task. Yet, how fine-tuning changes the underlying embedding space is less studied. In this work, we study the English BERT family and use two probing techniques to analyze how fine-tuning changes the space. We hypothesize that fine-tuning affects classification performance by increasing the distances between examples associated with different labels. We confirm this hypothesis with carefully designed experiments on five different NLP tasks. Via these experiments, we also discover an exception to the prevailing wisdom that “fine-tuning always improves performance”. Finally, by comparing the representations before and after fine-tuning, we discover that fine-tuning does not introduce arbitrary changes to representations; instead, it adjusts the representations to downstream tasks while largely preserving the original spatial structure of the data points.

   @inproceedings{zhou-srikumar-2022-closer,
     title = {A Closer Look at How Fine-tuning Changes {BERT}},
     author = {Zhou, Yichu and Srikumar, Vivek},
     booktitle = {Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
     month = may,
     year = {2022},
     address = {Dublin, Ireland},
     publisher = {Association for Computational Linguistics},
     doi = {10.18653/v1/2022.acl-long.75},
     pages = {1046--1061}
   }

2. 

   DirectProbe: Studying Representations without Classifiers

   Yichu Zhou, and Vivek Srikumar

   In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Jun 2021

   Abs Bib PDF Code

   Understanding how linguistic structure is encoded in contextualized embedding could help explain their impressive performance across NLP. Existing approaches for probing them usually call for training classifiers and use the accuracy, mutual information, or complexity as a proxy for the representation’s goodness. In this work, we argue that doing so can be unreliable because different representations may need different classifiers. We develop a heuristic, DirectProbe, that directly studies the geometry of a representation by building upon the notion of a version space for a task. Experiments with several linguistic tasks and contextualized embeddings show that, even without training classifiers, DirectProbe can shine lights on how an embedding space represents labels and also anticipate the classifier performance for the representation.

   @inproceedings{zhou-srikumar-2021-directprobe,
     title = {{D}irect{P}robe: Studying Representations without Classifiers},
     author = {Zhou, Yichu and Srikumar, Vivek},
     booktitle = {Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
     month = jun,
     year = {2021},
     address = {Online},
     publisher = {Association for Computational Linguistics},
     doi = {10.18653/v1/2021.naacl-main.401},
     pages = {5070--5083},
   }