Shailee Jain, University of Texas Austin: “Using NLP Systems to Investigate Language Processing in the Human Brain”

Position:  PhD Candidate

Current Institution: University of Texas Austin

Abstract: Using NLP Systems to Investigate Language Processing in the Human Brain

Natural language contains diverse semantic information at multiple timescales- from individual words to compositional phrases. Here we discuss how the human cortex and NLP models encode semantic and temporal information while processing language. Our approach revolves around building encoding models that learn to predict fMRI responses to natural language using neural language models (LMs). However LMs do not explicitly separate information at different timescales or provide insight into the types of semantics encoded. To this end we also develop tools to control and interpret LMs that in turn reveal brain function. Our first work focussed on building interpretable multi-timescale (MT) representations by forcing individual units in an LSTM LM to integrate information over specific timescales. On using probing techniques we found that short-timescale units capture token-level information like part-of-speech while longer timescales capture discourse-level information like narrative topic. Next we used MT LSTM to explicitly and directly map the processing timescale of regions in the brain. While past work has used controlled experiments to provide coarse estimates the MT model could reveal a fine-grained map of timescales across the human language pathway. In the second work we considered how language systems selectively encode semantic and temporal information. To do this we first built encoding models using transformer LMs that simulated phrase-level processing in the brain. By predicting the phrases that drive each brain region this model provided a nuanced map of how semantic information is organized across the cortex. Next we ablated the transformer LM to investigate how different regions integrate information in a phrase. Overall this framework highlighted the distinct patterns of semantic integration across the brain. This built over our understanding of how the brain selectively processes information at the level of individual words.


Shailee Jain is a fourth year PhD student at the University of Texas at Austin advised by Alexander Huth. Her research lies at the intersection of ML NLP and cognitive neuroscience. Specifically she is interested in modeling biological and artificial systems in tandem to understand how they process language. Her approach relies on combining neuroimaging experiments with deep networks to build predictive models of the brain. By controlling and probing the linguistic information encoded in the artificial networks these predictive models can then reveal how the brain organizes and computes the same information. In the past Shailee has co-organized workshops on jointly modeling the brain and NLP systems at NeurIPS’19 and ICLR’20. Prior to joining UT Austin she received her bachelor’s degree from NITK Surathkal India.