Connecting Language to Actions & the World @ CMU


Just a big 'ol list. You can use the buttons to filter or text box to search.

EGL: You can't learn language ...
... from the radio
    Text → Perception
... by watching TV
    Perception → Action
... by yourself
    Action → Social


Click a category


Publications


Quanting Xie
Embodied-RAG: General non-parametric Embodied Memory for Retrieval and Generation
ArXiv (2024)
Quanting Xie
,
So Yeon Min
, Tianyi Zhang, Aarav Bajaj, Ruslan Salakhutdinov, Matthew Johnson-Roberson,
Yonatan Bisk

Website
MotIF: Motion Instruction Fine-tuning
ArXiv (2024)
Minyoung Hwang, Joey Hejna, Dorsa Sadigh,
Yonatan Bisk

Website

Vidhi Jain
ANAVI: Audio Noise Awareness using Visuals of Indoor environments for NAVIgation
Conference on Robot Learning (2024)
Vidhi Jain
, Rishi Veerapaneni,
Yonatan Bisk

Website

So Yeon Min
Situated Instruction Following
European Conference on Computer Vision (2024)
So Yeon Min
, Xavi Puig, Devendra Singh Chaplot, Tsung-Yen Yang, Akshara Rai, Priyam Parashar, Ruslan Salakhutdinov,
Yonatan Bisk
, Roozbeh Mottaghi

Website

Jared Fernandez
Gradient Localization Improves Lifelong Pretraining of Language Models
Findings of the Conference on Empirical Methods in Natural Language Processing (2024)
Jared Fernandez
,
Yonatan Bisk
, Emma Strubell

How to Train Your Fact Verifier: Knowledge Transfer with Multimodal Open Models
Findings of the Conference on Empirical Methods in Natural Language Processing (2024)
Jaeyoung Lee, Ximing Lu, Jack Hessel, Faeze Brahman, Youngjae Yu,
Yonatan Bisk
, Yejin Choi, Saadia Gabriel


Jimin Sun
Tools Fail: Detecting Silent Errors in Faulty Tools
Conference on Empirical Methods in Natural Language Processing (2024)
Jimin Sun
,
So Yeon Min
,
Yingshan Chang
,
Yonatan Bisk

DegustaBot: Zero-Shot Visual Preference Estimation for Personalized Multi-Object Rearrangement
ArXiv (2024)
Benjamin A. Newman, Pranay Gupta, Kris Kitani,
Yonatan Bisk
, Henny Admoni, Chris Paxton


Yingshan Chang
Language Models Need Inductive Biases to Count Inductively
ArXiv (2024)
Yingshan Chang
,
Yonatan Bisk


Yingshan Chang
DiffusionPID: Interpreting Diffusion via Partial Information Decomposition
Thirty-Eighth Annual Conference on Neural Information Processing Systems (2024)
Shaurya Dewan, Rushikesh Zawar, Prakanshul Saxena,
Yingshan Chang
, Andrew Luo,
Yonatan Bisk

Dialogue with Robots: Proposals for Broadening Participation and Research in the SLIVAR Community
ArXiv (2024)
Casey Kennington, Malihe Alikhani, Heather Pon-Barry, Katherine Atwell,
Yonatan Bisk
, Daniel Fried, Felix Gervits, Zhao Han, Mert Inan, Michael Johnston, Raj Korpan, Diane Litman, Matthew Marge, Cynthia Matuszek, Ross Mead, Shiwali Mohan, Raymond Mooney, Natalie Parde, Jivko Sinapov, Angela Stewart, Matthew Stone, Stefanie Tellex, Tom Williams


Yingshan Chang
VISREAS: Complex Visual Reasoning with Unanswerable Questions
Findings of the 62nd Annual Meeting of the Association for Computational Linguistics (2024)
Syeda Nahida Akter, Sangwu Lee,
Yingshan Chang
,
Yonatan Bisk
, Eric Nyberg

AgentKit: Flow Engineering with Graphs, not Coding
Conference on Language Modeling (2024)
Yue Wu, Yewen Fan,
So Yeon Min
, Shrimai Prabhumoye, Stephen McAleer,
Yonatan Bisk
, Ruslan Salakhutdinov, Yuanzhi Li, Tom Mitchell


Yingshan Chang
Skews in the Phenomenon Space Hinder Generalization in Text-to-Image Generation
European Conference on Computer Vision (2024)
Yingshan Chang
, Yasi Zhang, Zhiyuan Fang, Yingnian Wu,
Yonatan Bisk
, Feng Gao

Direct Preference Optimization of Video Large Multimodal Models from Language Model Reward
ArXiv (2024)
Ruohong Zhang, Liangke Gui, Zhiqing Sun, Yihao Feng, Keyang Xu, Yuanhan Zhang, Di Fu, Chunyuan Li, Alexander Hauptmann,
Yonatan Bisk
, Yiming Yang


Vidhi Jain
Vid2Robot: End-to-end Video-conditioned Policy Learning with Cross-Attention Transformers
Robotics: Science and Systems (2024)
Vidhi Jain
, Maria Attarian, Nikhil J Joshi, Ayzaan Wahid, Danny Driess, Quan Vuong, Pannag R Sanketi, Pierre Sermanet, Stefan Welker, Christine Chan, Igor Gilitschenski,
Yonatan Bisk
, Debidatta Dwibedi

Website

Hao Zhu
SOTOPIA-π: Interactive Learning of Socially Intelligent Language Agents
The 62nd Annual Meeting of the Association for Computational Linguistics (2024)
Ruiyi Wang, Haofei Yu, Wenxin Zhang, Zhengyang Qi, Maarten Sap, Graham Neubig,
Yonatan Bisk
, Hao Zhu

Website
OpenEQA: Embodied Question Answering in the Era of Foundation Models
Conference on Computer Vision and Pattern Recognition (2024)
Arjun Majumdar, Anurag Ajay, Xiaohan Zhang, Pranav Putta, Sriram Yenamandra, Mikael Henaff, Sneha Silwal, Paul Mcvay, Oleksandr Maksymets, Sergio Arnaud, Karmesh Yadav, Qiyang Li, Ben Newman, Mohit Sharma, Vincent Berges, Shiqi Zhang, Pulkit Agrawal,
Yonatan Bisk
, Dhruv Batra, Mrinal Kalakrishnan, Franziska Meier, Chris Paxton, Sasha Sax, Aravind Rajeswaran

PDF Website Blog

Quanting Xie
Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysis
ArXiv (2023)
Yafei Hu,
Quanting Xie
,
Vidhi Jain
, Jonathan Francis, Jay Patrikar, Nikhil Keetha, Seungchan Kim, Yaqi Xie, Tianyi Zhang, Shibo Zhao, Yu Quan Chong, Chen Wang, Katia Sycara, Matthew Johnson-Roberson, Dhruv Batra, Xiaolong Wang, Sebastian Scherer, Zsolt Kira, Fei Xia,
Yonatan Bisk

Website

Hao Zhu
SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agents
The Twelfth International Conference on Learning Representations (2024)
Xuhui Zhou, Hao Zhu,
Leena Mathur
, Ruohong Zhang, Haofei Yu, Zhengyang Qi, Louis-Philippe Morency,
Yonatan Bisk
, Daniel Fried, Graham Neubig, Maarten Sap


Vidhi Jain
Open X-Embodiment: Robotic Learning Datasets and RT-X Models: Open X-Embodiment Collaboration
IEEE International Conference on Robotics and Automation (ICRA) (2024)
Abby O’Neill, Abdul Rehman, Abhiram Maddukuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acorn Pooley, Agrim Gupta, Ajay Mandlekar, Ajinkya Jain, Albert Tung, Alex Bewley, Alex Herzog, Alex Irpan, Alexander Khazatsky, Anant Rai, Anchit Gupta, Andrew Wang, Anikait Singh, Animesh Garg, Aniruddha Kembhavi, Annie Xie, Anthony Brohan, Antonin Raffin, Archit Sharma, Arefeh Yavary, Arhan Jain, Ashwin Balakrishna, Ayzaan Wahid, Ben Burgess-Limerick, Beomjoon Kim, Bernhard Schölkopf, Blake Wulfe, Brian Ichter, Cewu Lu, Charles Xu, Charlotte Le, Chelsea Finn, Chen Wang, Chenfeng Xu, Cheng Chi, Chenguang Huang, Christine Chan, Christopher Agia, Chuer Pan, Chuyuan Fu, Coline Devin, Danfei Xu, Daniel Morton, Danny Driess, Daphne Chen, Deepak Pathak, Dhruv Shah, Dieter Büchler, Dinesh Jayaraman, Dmitry Kalashnikov, Dorsa Sadigh, Edward Johns, Ethan Foster, Fangchen Liu, Federico Ceola, Fei Xia, Feiyu Zhao, Freek Stulp, Gaoyue Zhou, Gaurav S. Sukhatme, Gautam Salhotra, Ge Yan, Gilbert Feng, Giulio Schiavi, Glen Berseth, Gregory Kahn, Guanzhi Wang, Hao Su, Hao-Shu Fang, Haochen Shi, Henghui Bao, Heni Ben Amor, Henrik I Christensen, Hiroki Furuta, Homer Walke, Hongjie Fang, Huy Ha, Igor Mordatch, Ilija Radosavovic, Isabel Leal, Jacky Liang, Jad Abou-Chakra, Jaehyung Kim, Jaimyn Drake, Jan Peters, Jan Schneider, Jasmine Hsu, Jeannette Bohg, Jeffrey Bingham, Jeffrey Wu, Jensen Gao, Jiaheng Hu, Jiajun Wu, Jialin Wu, Jiankai Sun, Jianlan Luo, Jiayuan Gu, Jie Tan, Jihoon Oh, Jimmy Wu, Jingpei Lu, Jingyun Yang, Jitendra Malik, João Silvério, Joey Hejna, Jonathan Booher, Jonathan Tompson, Jonathan Yang, Jordi Salvador, Joseph J. Lim, Junhyek Han, Kaiyuan Wang, Kanishka Rao, Karl Pertsch, Karol Hausman, Keegan Go, Keerthana Gopalakrishnan, Ken Goldberg, Kendra Byrne, Kenneth Oslund, Kento Kawaharazuka, Kevin Black, Kevin Lin, Kevin Zhang, Kiana Ehsani, Kiran Lekkala, Kirsty Ellis, Krishan Rana, Krishnan Srinivasan, Kuan Fang, Kunal Pratap Singh, Kuo-Hao Zeng, Kyle Hatch, Kyle Hsu, Laurent Itti, Lawrence Yunliang Chen, Lerrel Pinto, Li Fei-Fei, Liam Tan, Linxi Jim Fan, Lionel Ott, Lisa Lee, Luca Weihs, Magnum Chen, Marion Lepert, Marius Memmel, Masayoshi Tomizuka, Masha Itkina, Mateo Guaman Castro, Max Spero, Maximilian Du, Michael Ahn, Michael C. Yip, Mingtong Zhang, Mingyu Ding, Minho Heo, Mohan Kumar Srirama, Mohit Sharma, Moo Jin Kim, Naoaki Kanazawa, Nicklas Hansen, Nicolas Heess, Nikhil J Joshi, Niko Suenderhauf, Ning Liu, Norman Di Palo, Nur Muhammad MahiShafiullah, Oier Mees, Oliver Kroemer, Osbert Bastani, Pannag R Sanketi, Patrick Tree Miller, Patrick Yin, Paul Wohlhart, Peng Xu, Peter David Fagan, Peter Mitrano, Pierre Sermanet, Pieter Abbeel, Priya Sundaresan, Qiuyu Chen, Quan Vuong, Rafael Rafailov, Ran Tian, Ria Doshi, Roberto Martín-Martín, Rohan Baijal, Rosario Scalise, Rose Hendrix, Roy Lin, Runjia Qian, Ruohan Zhang, Russell Mendonca, Rutav Shah, Ryan Hoque, Ryan Julian, Samuel Bustamante, Sean Kirmani, Sergey Levine, Shan Lin, Sherry Moore, Shikhar Bahl, Shivin Dass, Shubham Sonawani, Shuran Song, Sichun Xu, Siddhant Haldar, Siddharth Karamcheti, Simeon Adebola, Simon Guist, Soroush Nasiriany, Stefan Schaal, Stefan Welker, Stephen Tian, Subramanian Ramamoorthy, Sudeep Dasari, Suneel Belkhale, Sungjae Park, Suraj Nair, Suvir Mirchandani, Takayuki Osa, Tanmay Gupta, Tatsuya Harada, Tatsuya Matsushima, Ted Xiao, Thomas Kollar, Tianhe Yu, Tianli Ding, Todor Davchev, Tony Z. Zhao, Travis Armstrong, Trevor Darrell, Trinity Chung, Vidhi Jain, Vincent Vanhoucke, Wei Zhan, Wenxuan Zhou, Wolfram Burgard, Xi Chen, Xiaolong Wang, Xinghao Zhu, Xinyang Geng, Xiyuan Liu, Xu Liangwei, Xuanlin Li, Yao Lu, Yecheng Jason Ma, Yejin Kim, Yevgen Chebotar, Yifan Zhou, Yifeng Zhu, Yilin Wu, Ying Xu, Yixuan Wang, Yonatan Bisk, Yoonyoung Cho, Youngwoon Lee, Yuchen Cui, Yue Cao, Yueh-Hua Wu, Yujin Tang, Yuke Zhu, Yunchu Zhang, Yunfan Jiang, Yunshuang Li, Yunzhu Li, Yusuke Iwasawa, Yutaka Matsuo, Zehan Ma, Zhuo Xu, Zichen Jeff Cui, Zichen Zhang, Zipeng Lin
Website

Quanting Xie
Reasoning about the Unseen for Efficient Outdoor Object Navigation
ArXiv (2023)
Quanting Xie
, Tianyi Zhang, Kedi Xu, Matthew Johnson-Roberson,
Yonatan Bisk

MOSAIC: Learning Unified Multi-Sensory Object Property Representations for Robot Perception
IEEE International Conference on Robotics and Automation (2024)
Gyan Tatiya, Jonathan Francis, Ho-Hsiang Wu,
Yonatan Bisk
, Jivko Sinapov


Hao Zhu
WebArena: A Realistic Web Environment for Building Autonomous Agents
The Twelfth International Conference on Learning Representations (2024)
Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou,
Yonatan Bisk
, Daniel Fried, Uri Alon, Graham Neubig

Website

Vidhi Jain
Spatial-Language Attention Policies for Efficient Robot Learning
Conference on Robot Learning (2023)
Priyam Parashar,
Vidhi Jain
, Xiaohan Zhang, Jay Vakil, Sam Powers,
Yonatan Bisk
, Chris Paxton

Website
SPAE: Semantic Pyramid AutoEncoder for Multimodal Generation with Frozen LLMs
Thirty-seventh Conference on Neural Information Processing Systems (2023)
Lijun Yu, Yong Cheng, Zhiruo Wang, Vivek Kumar, Wolfgang Macherey, Yanping Huang, David A. Ross, Irfan Essa,
Yonatan Bisk
, Ming-Hsuan Yang, Kevin Murphy, Alexander Hauptmann, Lu Jiang


Vidhi Jain
HomeRobot: Open-Vocabulary Mobile Manipulation
Conference on Robot Learning (2023)
Sriram Yenamandra, Arun Ramachandran, Karmesh Yadav, Austin Wang, Mukul Khanna, Theophile Gervet, Tsung-Yen Yang,
Vidhi Jain
, Alexander William Clegg, John Turner, Zsolt Kira, Manolis Savva, Angel Chang, Devendra Singh Chaplot, Dhruv Batra, Roozbeh Mottaghi,
Yonatan Bisk
, Chris Paxton

Website Video

So Yeon Min
SPRING: GPT-4 Out-performs RL Algorithms by Studying Papers and Reasoning
Thirty-seventh Conference on Neural Information Processing Systems (2023)
Yue Wu, Shrimai Prabhumoye,
So Yeon Min
,
Yonatan Bisk
, Ruslan Salakhutdinov, Amos Azaria, Tom Mitchell, Yuanzhi Li


So Yeon Min
Plan, Eliminate, and Track -- Language Models are Good Teachers for Embodied Agents
ArXiv (2023)
Yue Wu,
So Yeon Min
,
Yonatan Bisk
, Ruslan Salakhutdinov, Amos Azaria, Yuanzhi Li, Tom Mitchell, Shrimai Prabhumoye


Vidhi Jain
Towards Open-World Mobile Manipulation in Homes: Lessons from the Neurips 2023 HomeRobot Open Vocabulary Mobile Manipulation Challenge
Thirty-seventh Conference on Neural Information Processing Systems: Competition Track (2023)
Sriram Yenamandra, Arun Ramachandran, Mukul Khanna, Karmesh Yadav, Jay Vakil, Andrew Melnik, Michael Büttner, Leon Harz, Lyon Brown, Gora Chand Nandi, Arjun PS, Gaurav Kumar Yadav, Rahul Kala, Robert Haschke, Yang Luo, Jinxin Zhu, Yansen Han, Bingyi Lu, Xuan Gu, Qinyuan Liu, Yaping Zhao, Qiting Ye, Chenxiao Dou, Yansong Chua, Volodymyr Kuzma, Vladyslav Humennyy, Ruslan Partsey, Jonathan Francis, Devendra Singh Chaplot, Gunjan Chhablani, Alexander Clegg, Theophile Gervet,
Vidhi Jain
, Ram Ramrakhya, Andrew Szot, Austin Wang, Tsung-Yen Yang, Aaron Edsinger, Charlie Kemp, Binit Shah, Zsolt Kira, Dhruv Batra, Roozbeh Mottaghi,
Yonatan Bisk
, Chris Paxton

Website

Hao Zhu
EXCALIBUR: Encouraging and Evaluating Embodied Exploration
Conference on Computer Vision and Pattern Recognition (2023)
Hao Zhu, Raghav Kapoor,
So Yeon Min
, Winson Han, Jiatai Li, Kaiwen Geng, Graham Neubig,
Yonatan Bisk
, Aniruddha Kembhavi, Luca Weihs

PDF Video Tasks

Jared Fernandez
The Framework Tax: Disparities Between Inference Efficiency in Research and Deployment
The 2023 Conference on Empirical Methods in Natural Language Processing (2023)
Jared Fernandez
, Jacob Kahn, Clara Na,
Yonatan Bisk
, Emma Strubell


Hao Zhu
Computational Language Acquisition with Theory of Mind
The Eleventh International Conference on Learning Representations (2023)
Andy Liu, Hao Zhu, Emmy Liu,
Yonatan Bisk
, Graham Neubig

PDF

So Yeon Min
Self-Supervised Object Goal Navigation with In-Situ Finetuning
IEEE/RSJ International Conference on Intelligent Robots and Systems (2023)
So Yeon Min
, Yao-Hung Hubert Tsai, Wei Ding, Ali Farhadi, Ruslan Salakhutdinov,
Yonatan Bisk
, Jian Zhang

Video

Vidhi Jain
MAEA: Multimodal Attribution for Embodied AI
Progress and Challenges in Building Trustworthy Embodied AI at NeurIPS 2022 (2022)
Vidhi Jain
, Jayant Sravan Tamarapalli, Sahiti Yerramilli,
Yonatan Bisk

PDF

So Yeon Min
Tackling AlfWorld with Action Attention and Common Sense from Language Models
Language and Reinforcement Learning Workshop at NeurIPS 2022 (2022)
Yue Wu,
So Yeon Min
,
Yonatan Bisk
, Ruslan Salakhutdinov, Shrimai Prabhumoye

PDF
Retrospectives on the Embodied AI Workshop
ArXiv (2022)
Matt Deitke, Dhruv Batra,
Yonatan Bisk
, Tommaso Campari, Angel X. Chang, Devendra Singh Chaplot, Changan Chen, Claudia Pérez D'Arpino, Kiana Ehsani, Ali Farhadi, Li Fei-Fei, Anthony Francis, Chuang Gan, Kristen Grauman, David Hall, Winson Han, Unnat Jain, Aniruddha Kembhavi, Jacob Krantz, Stefan Lee, Chengshu Li, Sagnik Majumder, Oleksandr Maksymets, Roberto Martín-Martín, Roozbeh Mottaghi, Sonia Raychaudhuri, Mike Roberts, Silvio Savarese, Manolis Savva, Mohit Shridhar, Niko Sünderhauf, Andrew Szot, Ben Talbot, Joshua B. Tenenbaum, Jesse Thomason, Alexander Toshev, Joanne Truong, Luca Weihs, Jiajun Wu


So Yeon Min
Don't Copy the Teacher: Data and Model Challenges in Embodied Dialogue
Conference on Empirical Methods in Natural Language Processing (2022)
So Yeon Min
, Hao Zhu, Ruslan Salakhutdinov,
Yonatan Bisk

Video
EvEntS ReaLM: Event Reasoning of Entity States via Language Models
Conference on Empirical Methods in Natural Language Processing (2022)
Evangelia Spiliopoulou, Artidoro Pagnoni,
Yonatan Bisk
, Eduard Hovy

On Advances in Text Generation from Images Beyond Captioning: A Case Study in Self-Rationalization
Findings of the Conference on Empirical Methods in Natural Language Processing (2022)
Shruti Palaskar, Akshita Bhagia,
Yonatan Bisk
, Florian Metze, Alan W Black, Ana Marasovic


Vidhi Jain
Transformers are Adaptable Task Planners
Conference on Robot Learning (2022)
Vidhi Jain
, Yixin Lin, Eric Undersander,
Yonatan Bisk
, Akshara Rai

Video

Yingshan Chang
Training Vision-Language Transformers from Captions
Transactions on Machine Learning Research (2023)
Liangke Gui,
Yingshan Chang
, Qiuyuan Huang, Subhojit Som, Alexander Hauptmann, Jianfeng Gao,
Yonatan Bisk

Symmetric Machine Theory of Mind
39th International Conference on Machine Learning (2022)
Melanie Sclar, Graham Neubig,
Yonatan Bisk

PDF Video
A Framework for Learning to Request Rich and Contextually Useful Information from Humans
39th International Conference on Machine Learning, (2022)
Khanh Nguyen,
Yonatan Bisk
, Hal Daume III

Video

Hao Zhu
Simulated Language Learning from Communicative Goals and Linguistic Input
Annual Meeting of the Cognitive Science Society (2022)
Hao Zhu,
Yonatan Bisk
, Graham Neubig

PDF
HEAR 2021: Holistic Evaluation of Audio Representations
Proceedings of Machine Learning Research (PMLR): NeurIPS 2021 Competition Track (2022)
Joseph Turian, Jordie Shier, Humair Raj Khan, Bhiksha Raj, Björn W. Schuller, Christian J. Steinmetz, Colin Malloy, George Tzanetakis, Gissel Velarde, Kirk McNally, Max Henry, Nicolas Pinto, Camille Noufi, Christian Clough, Dorien Herremans, Eduardo Fonseca, Jesse Engel, Justin Salamon, Philippe Esling, Pranay Manocha, Shinji Watanabe, Zeyu Jin,
Yonatan Bisk

Website
KAT: A Knowledge Augmented Transformer for Vision-and-Language
Annual Conference of the North American Chapter of the Association for Computational Linguistics (2022)
Liangke Gui, Borui Wang, Qiuyuan Huang, Alexander Hauptmann,
Yonatan Bisk
, Jianfeng Gao


So Yeon Min
FILM: Following Instructions in Language with Modular Methods
The Tenth International Conference on Learning Representations (2022)
So Yeon Min
, Devendra Singh Chaplot, Pradeep Ravikumar,
Yonatan Bisk
, Ruslan Salakhutdinov

Website Video

Yingshan Chang
WebQA: Multihop and Multimodal QA
Conference on Computer Vision and Pattern Recognition (2022)
Yingshan Chang
, Mridu Narang, Hisami Suzuki, Guihong Cao, Jianfeng Gao,
Yonatan Bisk

Project Leaderboard Challenge Summary

Hao Zhu
Dependency Induction Through the Lens of Visual Perception
The SIGNLL Conference on Computational Natural Language Learning (2021)
Ruisi Su, Shruti Rijhwani, Hao Zhu, Junxian He, Xinyu Wang,
Yonatan Bisk
, Graham Neubig

Language Grounding with 3D Objects
Conference on Robot Learning (2021)
Jesse Thomason, Mohit Shridhar,
Yonatan Bisk
, Chris Paxton, Luke Zettlemoyer

TACo: Token-aware Cascade Contrastive Learning for Video-Text Alignment
International Conference on Computer Vision (2021)
Jianwei Yang,
Yonatan Bisk
, Jianfeng Gao

Worst of Both Worlds: Biases Compound in Pre-trained Vision-and-Language Models
4th Workshop on Gender Bias in Natural Language Processing (2022)
Tejas Srinivasan,
Yonatan Bisk

KB-VLP: Knowledge Based Vision and Language Pretraining
ICML21 Workshop on Self-Supervised Learning (2021)
Kezhen Chen, Qiuyuan Huang,
Yonatan Bisk
, Daniel McDuff, Jianfeng Gao

PDF

Hao Zhu
Few-shot Language Coordination by Modeling Theory of Mind
The Thirty-eighth International Conference on Machine Learning (2021)
Hao Zhu, Graham Neubig,
Yonatan Bisk

PDF
Grounding `Grounding' in NLP
Findings of The 2021 Conference of the Association for Computational Linguistics (2021)
Khyathi Raghavi Chandu,
Yonatan Bisk
, Alan W Black

An Empirical Study on the Generalization Power of Neural Representations Learned via Visual Guessing Games
The 2021 Conference of the European Chapter of the Association for Computational Linguistics (2021)
Alessandro Suglia,
Yonatan Bisk
, Ioannis Konstas, Antonio Vergari, Emanuele Bastianelli, Andrea Vanzo, Oliver Lemon

Knowledge-driven Data Construction for Zero-shot Evaluation in Commonsense Question Answering
35th AAAI Conference on Artificial Intelligence (2021)
Kaixin Ma, Filip Ilievski, Jonathan Francis,
Yonatan Bisk
, Eric Nyberg, Alessandro Oltramari

ALFWorld: Aligning Text and Embodied Environments for Interactive Learning
International Conference on Learning Representations (2021)
Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté,
Yonatan Bisk
, Adam Trischler, Matthew Hausknecht

Website
Imagining Grounded Conceptual Representations from Perceptual Information in Situated Guessing Games
The 28th International Conference on Computational Linguistics (2020)
Alessandro Suglia, Antonio Vergari, Ioannis Konstas,
Yonatan Bisk
, Emanuele Bastianelli, Andrea Vanzo, Oliver Lemon


Hao Zhu
The Return of Lexical Dependencies: Neural Lexicalized PCFGs
Transactions of the Association for Computational Linguistics (2020)
Hao Zhu,
Yonatan Bisk
, Graham Neubig

Experience Grounds Language
Conference on Empirical Methods in Natural Language Processing (2020)
Yonatan Bisk
, Ari Holtzman, Jesse Thomason, Jacob Andreas, Yoshua Bengio, Joyce Chai, Mirella Lapata, Angeliki Lazaridou, Jonathan May, Aleksandr Nisnevich, Nicolas Pinto, Joseph Turian

Video Slides
RMM: A Recursive Mental Model for Dialog Navigation
Conference on Empirical Methods in Natural Language Processing: Findings (2020)
Homero Roman Roman,
Yonatan Bisk
, Jesse Thomason, Asli Celikyilmaz, Jianfeng Gao

A Benchmark for Structured Procedural Knowledge Extraction from Cooking Videos
Workshop on NLP Beyond Text (2020)
Frank F. Xu, Lei Ji, Botian Shi, Junyi Du, Graham Neubig,
Yonatan Bisk
, Nan Duan

Multi-View Learning for Vision-and-Language Navigation
ArXiv (2020)
Qiaolin Xia, Xiujun Li, Chunyuan Li,
Yonatan Bisk
, Zhifang Sui, Jianfeng Gao, Yejin Choi, Noah A. Smith

ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks
The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2020)
Mohit Shridhar, Jesse Thomason, Daniel Gordon,
Yonatan Bisk
, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, Dieter Fox

Website
PIQA: Reasoning about Physical Commonsense in Natural Language
Thirty-Fourth AAAI Conference on Artificial Intelligence (2020)
Yonatan Bisk
, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, Yejin Choi

Slides Leaderboard
Robust Navigation with Language Pretraining and Stochastic Sampling
Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing (2019)
Xiujun Li, Chunyuan Li, Qiaolin Xia,
Yonatan Bisk
, Asli Celikyilmaz, Jianfeng Gao, Noah A. Smith, Yejin Choi

Defending Against Neural Fake News
Thirty-third Conference on Neural Information Processing Systems (2019)
Rowan Zellers, Ari Holtzman, Hannah Rashkin,
Yonatan Bisk
, Ali Farhadi, Franziska Roesner, Yejin Choi

Demo Models
FIND: Identifying Functionally and Structurally Important Features in Protein Sequences with Deep Neural Networks
bioRxiv (2019)
Ranjani Murali, James Hemp, Victoria Orphan,
Yonatan Bisk

PDF
Improving Robot Success Detection using Static Object Data
IEEE/RSJ International Conference on Intelligent Robots and Systems (2019)
Rosario Scalise, Jesse Thomason,
Yonatan Bisk
, Siddhartha Srinivasa

Video
Early Fusion for Goal Directed Robotic Vision
2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (2019)
Aaron Walsman,
Yonatan Bisk
, Saadia Gabriel, Dipendra Misra, Yoav Artzi, Yejin Choi, Dieter Fox

HellaSwag: Can a Machine Really Finish Your Sentence?
Association for Computational Linguistics (2019)
Rowan Zellers, Ari Holtzman,
Yonatan Bisk
, Ali Farhadi, Yejin Choi

From Recognition to Cognition: Visual Commonsense Reasoning
The IEEE Conference on Computer Vision and Pattern Recognition (2019)
Rowan Zellers,
Yonatan Bisk
, Ali Farhadi, Yejin Choi

URL
Tactical Rewind: Self-Correction via Backtracking in Vision-and-Language Navigation
The IEEE Conference on Computer Vision and Pattern Recognition (2019)
Liyiming Ke, Xiujun Li,
Yonatan Bisk
, Ari Holtzman, Zhe Gan, Jingjing Liu, Jianfeng Gao, Yejin Choi, Siddhartha Srinivasa

Video
Prospection: Interpretable Plans From Language By Predicting the Future
International Conference on Robotics and Automation (ICRA) (2019)
Chris Paxton,
Yonatan Bisk
, Jesse Thomason, Arunkumar Byravan, Dieter Fox

Shifting the Baseline: Single Modality Performance on Visual Navigation & QA
Annual Conference of the North American Chapter of the Association for Computational Linguistics (2019)
Jesse Thomason, Daniel Gordon,
Yonatan Bisk

Benchmarking Hierarchical Script Knowledge
Annual Conference of the North American Chapter of the Association for Computational Linguistics (2019)
Yonatan Bisk
, Jan Buys, Karl Pichotta, Yejin Choi

PDF
Character-based Surprisal as a Model of Reading Difficulty in the Presence of Errors
The 41st Annual Meeting of the Cognitive Science Society (2019)
Michael Hahn, Frank Keller,
Yonatan Bisk
, Yonatan Belinkov

SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference
Conference on Empirical Methods in Natural Language Processing (2018)
Rowan Zellers,
Yonatan Bisk
, Roy Schwartz, Yejin Choi

Website Video
Balancing Shared Autonomy with Human-Robot Communication
ArXiv (2018)
Rosario Scalise,
Yonatan Bisk
, Maxwell Forbes, Daqing Yi, Yejin Choi, Siddhartha Srinivasa

Inducing Grammars with and for Neural Machine Translation
2nd Workshop on Neural Machine Translation (2018)
Ke Tran,
Yonatan Bisk

Bridging HMMs and RNNs through Architectural Transformations
32nd Conference on Neural Information Processing Systems (NIPS 2018), IRASL workshop (2018)
Jan Buys,
Yonatan Bisk
, Yejin Choi

PDF
Synthetic and Natural Noise Both Break Neural Machine Translation
6th International Conference on Learning Representations (2018)
Yonatan Belinkov,
Yonatan Bisk

CHALET: Cornell House Agent Learning Environment
ArXiv Preprint 1801.07357 (2018)
Claudia Yan, Dipendra Misra, Andrew Bennett, Aaron Walsman,
Yonatan Bisk
, Yoav Artzi

Simulator
Learning Interpretable Spatial Operations in a Rich 3D Blocks World 
Thirty-Second Conference on Artificial Intelligence (AAAI-18) (2018)
Yonatan Bisk
, Kevin Shih, Yejin Choi, Daniel Marcu

Data Poster
Natural Language Inference from Multiple Premises
Eighth International Joint Conference on Natural Language Processing (2017)
Alice Lai,
Yonatan Bisk
, Julia Hockenmaier

Natural Language Communication with Robots
15th Annual Conference of the North American Chapter of the Association for Computational Linguistics (2016)
Yonatan Bisk
, Deniz Yuret, Daniel Marcu

PDF Data Slides
Supertagging with LSTMs
15th Annual Conference of the North American Chapter of the Association for Computational Linguistics (Short Papers) (2016)
Ashish Vaswani,
Yonatan Bisk
, Kenji Sagae, Ryan Musa

PDF
Towards a Dataset for Human Computer Communication via Grounded Language Acquisition
AAAI 2016 Workshop on Symbiotic Cognitive Systems (2016)
Yonatan Bisk
, Daniel Marcu, William Wong

PDF
Unsupervised Neural Hidden Markov Models
Workshop on Structured Prediction for NLP (2016)
Ke Tran,
Yonatan Bisk
, Ashish Vaswani, Daniel Marcu, Kevin Knight

PDF Slides
Evaluating Induced CCG Parsers on Grounded Semantic Parsing
2016 Conference on Empirical Methods in Natural Language Processing (2016)
Yonatan Bisk
, Siva Reddy, John Blitzer, Julia Hockenmaier, Mark Steedman

PDF
Probing the Linguistic Strengths and Limitations of Unsupervised Grammar Induction
53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) (2015)
Yonatan Bisk
, Julia Hockenmaier

PDF
Labeled Grammar Induction with Minimal Supervision
53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers) (2015)
Yonatan Bisk
, Christos Christodoulopoulos, Julia Hockenmaier

PDF Poster
An HDP Model for Inducing Combinatory Categorial Grammars
Transactions of the Association for Computational Linguistics (2013)
Yonatan Bisk
, Julia Hockenmaier

PDF Slides
Simple Robust Grammar Induction with Combinatory Categorial Grammar
Twenty-Sixth Conference on Artificial Intelligence (AAAI-12) (2012)
Yonatan Bisk
, Julia Hockenmaier

PDF
Document-Topic Hierarchies from Document Graphs
21st ACM international conference on Information and knowledge management (CIKM 2012) (2012)
Tim Weninger,
Yonatan Bisk
, Jiawei Han

PDF
Induction of Linguistic Structure with Combinatory Categorial Grammars
NAACL HLT Workshop on Induction of Linguistic Structure (2012)
Yonatan Bisk
, Julia Hockenmaier

PDF
Normal-form parsing for CCGs with generalized composition and type-raising
23rd International Conference on Computational Linguistics (Coling 2010) (2010)
Julia Hockenmaier,
Yonatan Bisk

PDF