Connecting Language to Actions & the World @ CMU


Just a big 'ol list. You can use the buttons to filter or text box to search.

EGL: You can't learn language ...
... from the radio
    Text → Perception
... by watching TV
    Perception → Action
... by yourself
    Action → Social

or

Journals and Conferences

HomeRobot Open Vocab Mobile Manipulation Challenge
Thirty-seventh Conference on Neural Information Processing Systems: Competition Track
Sriram Yenamandra, Arun Ramachandran, Mukul Khanna, Karmesh Yadav, Devendra Singh Chaplot, Gunjan Chhablani, Alexander Clegg, Theophile Gervet,
Vidhi Jain
, Ruslan Partsey, Ram Ramrakhya, Andrew Szot, Tsung-Yen Yang, Aaron Edsinger, Charlie Kemp, Binit Shah, Zsolt Kira, Dhruv Batra, Roozbeh Mottaghi,
Yonatan Bisk
, Chris Paxton

Paper BibTeX
Hao Zhu
EXCALIBUR: Encouraging and Evaluating Embodied Exploration
Conference on Computer Vision and Pattern Recognition
Hao Zhu
, Raghav Kapoor,
So Yeon Min
, Winson Han, Jiatai Li, Kaiwen Geng, Graham Neubig,
Yonatan Bisk
, Aniruddha Kembhavi, Luca Weihs

Paper BibTeX
Computational Language Acquisition with Theory of Mind
The Eleventh International Conference on Learning Representations
Andy Liu,
Hao Zhu
, Emmy Liu,
Yonatan Bisk
, Graham Neubig

Paper BibTeX   Code
So Yeon Min
Don't Copy the Teacher: Data and Model Challenges in Embodied Dialogue
Conference on Empirical Methods in Natural Language Processing
So Yeon Min
,
Hao Zhu
, Ruslan Salakhutdinov,
Yonatan Bisk

Paper BibTeX
EvEntS ReaLM: Event Reasoning of Entity States via Language Models
Conference on Empirical Methods in Natural Language Processing
Evangelia Spiliopoulou, Artidoro Pagnoni,
Yonatan Bisk
, Eduard Hovy

Paper BibTeX
On Advances in Text Generation from Images Beyond Captioning: A Case Study in Self-Rationalization
Findings of the Conference on Empirical Methods in Natural Language Processing
Shruti Palaskar, Akshita Bhagia,
Yonatan Bisk
, Florian Metze, Alan W Black, Ana Marasovic

Paper BibTeX
Vidhi Jain
Transformers are Adaptable Task Planners
Conference on Robot Learning
Vidhi Jain
, Yixin Lin, Eric Undersander,
Yonatan Bisk
, Akshara Rai

Paper BibTeX   Video
Symmetric Machine Theory of Mind
39th International Conference on Machine Learning
Melanie Sclar, Graham Neubig,
Yonatan Bisk

Paper BibTeX   Talk
A Framework for Learning to Request Rich and Contextually Useful Information from Humans
39th International Conference on Machine Learning,
Khanh Nguyen,
Yonatan Bisk
, Hal Daume III

Paper BibTeX   Talk
Hao Zhu
Simulated Language Learning from Communicative Goals and Linguistic Input
Annual Meeting of the Cognitive Science Society
Hao Zhu
,
Yonatan Bisk
, Graham Neubig

Paper BibTeX
HEAR 2021: Holistic Evaluation of Audio Representations
Proceedings of Machine Learning Research (PMLR): NeurIPS 2021 Competition Track
Joseph Turian, Jordie Shier, Humair Raj Khan, Bhiksha Raj, Björn W. Schuller, Christian J. Steinmetz, Colin Malloy, George Tzanetakis, Gissel Velarde, Kirk McNally, Max Henry, Nicolas Pinto, Camille Noufi, Christian Clough, Dorien Herremans, Eduardo Fonseca, Jesse Engel, Justin Salamon, Philippe Esling, Pranay Manocha, Shinji Watanabe, Zeyu Jin,
Yonatan Bisk

Paper BibTeX
KAT: A Knowledge Augmented Transformer for Vision-and-Language
Annual Conference of the North American Chapter of the Association for Computational Linguistics
Liangke Gui, Borui Wang, Qiuyuan Huang, Alex Hauptmann,
Yonatan Bisk
, Jianfeng Gao

Paper BibTeX   Code
So Yeon Min
FILM: Following Instructions in Language with Modular Methods
The Tenth International Conference on Learning Representations
So Yeon Min
, Devendra Singh Chaplot, Pradeep Ravikumar,
Yonatan Bisk
, Ruslan Salakhutdinov

Paper BibTeX   Code
Yingshan Chang
WebQA: Multihop and Multimodal QA
Conference on Computer Vision and Pattern Recognition
Yingshan Chang
, Mridu Narang, Hisami Suzuki, Guihong Cao, Jianfeng Gao,
Yonatan Bisk

Paper BibTeX   Project   Leaderboard   Challenge Summary
Dependency Induction Through the Lens of Visual Perception
The SIGNLL Conference on Computational Natural Language Learning
Ruisi Su, Shruti Rijhwani,
Hao Zhu
, Junxian He, Xinyu Wang,
Yonatan Bisk
, Graham Neubig

Paper BibTeX
Language Grounding with 3D Objects
Conference on Robot Learning
Jesse Thomason, Mohit Shridhar,
Yonatan Bisk
, Chris Paxton, Luke Zettlemoyer

Paper BibTeX   Data
TACo: Token-aware Cascade Contrastive Learning for Video-Text Alignment
International Conference on Computer Vision
Jianwei Yang,
Yonatan Bisk
, Jianfeng Gao

Paper BibTeX
Hao Zhu
Few-shot Language Coordination by Modeling Theory of Mind
The Thirty-eighth International Conference on Machine Learning
Hao Zhu
, Graham Neubig,
Yonatan Bisk

Paper BibTeX   Code
Grounding `Grounding' in NLP
Findings of The 2021 Conference of the Association for Computational Linguistics
Khyathi Raghavi Chandu,
Yonatan Bisk
, Alan W Black

Paper BibTeX
An Empirical Study on the Generalization Power of Neural Representations Learned via Visual Guessing Games
The 2021 Conference of the European Chapter of the Association for Computational Linguistics
Alessandro Suglia,
Yonatan Bisk
, Ioannis Konstas, Antonio Vergari, Emanuele Bastianelli, Andrea Vanzo, Oliver Lemon

Paper BibTeX
Knowledge-driven Data Construction for Zero-shot Evaluation in Commonsense Question Answering
35th AAAI Conference on Artificial Intelligence
Kaixin Ma, Filip Ilievski, Jonathan Francis,
Yonatan Bisk
, Eric Nyberg, Alessandro Oltramari

Paper BibTeX   Code
ALFWorld: Aligning Text and Embodied Environments for Interactive Learning
International Conference on Learning Representations
Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté,
Yonatan Bisk
, Adam Trischler, Matthew Hausknecht

Paper BibTeX   Code
Imagining Grounded Conceptual Representations from Perceptual Information in Situated Guessing Games
The 28th International Conference on Computational Linguistics
Alessandro Suglia, Antonio Vergari, Ioannis Konstas,
Yonatan Bisk
, Emanuele Bastianelli, Andrea Vanzo, Oliver Lemon

Paper BibTeX
Hao Zhu
The Return of Lexical Dependencies: Neural Lexicalized PCFGs
Transactions of the Association for Computational Linguistics
Hao Zhu
,
Yonatan Bisk
, Graham Neubig

Paper BibTeX   Code
Experience Grounds Language
Conference on Empirical Methods in Natural Language Processing (EMNLP)
Yonatan Bisk
, Ari Holtzman, Jesse Thomason, Jacob Andreas, Yoshua Bengio, Joyce Chai, Mirella Lapata, Angeliki Lazaridou, Jonathan May, Aleksandr Nisnevich, Nicolas Pinto, Joseph Turian

Paper BibTeX   Video   Slides
RMM: A Recursive Mental Model for Dialog Navigation
Conference on Empirical Methods in Natural Language Processing: Findings
Homero Roman Roman,
Yonatan Bisk
, Jesse Thomason, Asli Celikyilmaz, Jianfeng Gao

Paper BibTeX   Code
ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks
The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
Mohit Shridhar, Jesse Thomason, Daniel Gordon,
Yonatan Bisk
, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, Dieter Fox

Paper BibTeX
PIQA: Reasoning about Physical Commonsense in Natural Language
Thirty-Fourth AAAI Conference on Artificial Intelligence
Yonatan Bisk
, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, Yejin Choi

Paper BibTeX   Slides
Robust Navigation with Language Pretraining and Stochastic Sampling
Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing
Xiujun Li, Chunyuan Li, Qiaolin Xia,
Yonatan Bisk
, Asli Celikyilmaz, Jianfeng Gao, Noah A. Smith, Yejin Choi

Paper BibTeX
Defending Against Neural Fake News
Thirty-third Conference on Neural Information Processing Systems
Rowan Zellers, Ari Holtzman, Hannah Rashkin,
Yonatan Bisk
, Ali Farhadi, Franziska Roesner, Yejin Choi

Paper BibTeX   Demo   Models
Improving Robot Success Detection using Static Object Data
IEEE/RSJ International Conference on Intelligent Robots and Systems
Rosario Scalise, Jesse Thomason,
Yonatan Bisk
, Siddhartha Srinivasa

Paper BibTeX   Video
Early Fusion for Goal Directed Robotic Vision
2019 IEEE/RSJ International Conference on Intelligent Robots and Systems
Aaron Walsman,
Yonatan Bisk
, Saadia Gabriel, Dipendra Misra, Yoav Artzi, Yejin Choi, Dieter Fox

Paper BibTeX
HellaSwag: Can a Machine Really Finish Your Sentence?
Association for Computational Linguistics
Rowan Zellers, Ari Holtzman,
Yonatan Bisk
, Ali Farhadi, Yejin Choi

Paper BibTeX
From Recognition to Cognition: Visual Commonsense Reasoning
The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
Rowan Zellers,
Yonatan Bisk
, Ali Farhadi, Yejin Choi

Paper BibTeX
Tactical Rewind: Self-Correction via Backtracking in Vision-and-Language Navigation
The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
Liyiming Ke, Xiujun Li,
Yonatan Bisk
, Ari Holtzman, Zhe Gan, Jingjing Liu, Jianfeng Gao, Yejin Choi, Siddhartha Srinivasa

Paper BibTeX   Video
Prospection: Interpretable Plans From Language By Predicting the Future
International Conference on Robotics and Automation (ICRA)
Chris Paxton,
Yonatan Bisk
, Jesse Thomason, Arunkumar Byravan, Dieter Fox

Paper BibTeX
Shifting the Baseline: Single Modality Performance on Visual Navigation & QA
Annual Conference of the North American Chapter of the Association for Computational Linguistics
Jesse Thomason, Daniel Gordon,
Yonatan Bisk

Paper BibTeX
Benchmarking Hierarchical Script Knowledge
Annual Conference of the North American Chapter of the Association for Computational Linguistics
Yonatan Bisk
, Jan Buys, Karl Pichotta, Yejin Choi

Paper BibTeX   Code
Character-based Surprisal as a Model of Reading Difficulty in the Presence of Errors
The 41st Annual Meeting of the Cognitive Science Society
Michael Hahn, Frank Keller,
Yonatan Bisk
, Yonatan Belinkov

Paper BibTeX
SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference
Conference on Empirical Methods in Natural Language Processing (EMNLP)
Rowan Zellers,
Yonatan Bisk
, Roy Schwartz, Yejin Choi

Paper BibTeX   Talk
Synthetic and Natural Noise Both Break Neural Machine Translation
6th International Conference on Learning Representations
Yonatan Belinkov,
Yonatan Bisk

Paper BibTeX
Learning Interpretable Spatial Operations in a Rich 3D Blocks World 
Thirty-Second Conference on Artificial Intelligence (AAAI-18)
Yonatan Bisk
, Kevin Shih, Yejin Choi, Daniel Marcu

Paper BibTeX   Data   Poster
Natural Language Inference from Multiple Premises
Eighth International Joint Conference on Natural Language Processing
Alice Lai,
Yonatan Bisk
, Julia Hockenmaier

Paper BibTeX
Natural Language Communication with Robots
15th Annual Conference of the North American Chapter of the Association for Computational Linguistics
Yonatan Bisk
, Deniz Yuret, Daniel Marcu

Paper BibTeX   Data   Slides
Supertagging with LSTMs
15th Annual Conference of the North American Chapter of the Association for Computational Linguistics (Short Papers)
Ashish Vaswani,
Yonatan Bisk
, Kenji Sagae, Ryan Musa

Paper BibTeX
Evaluating Induced CCG Parsers on Grounded Semantic Parsing
2016 Conference on Empirical Methods in Natural Language Processing
Yonatan Bisk
, Siva Reddy, John Blitzer, Julia Hockenmaier, Mark Steedman

Paper BibTeX
Probing the Linguistic Strengths and Limitations of Unsupervised Grammar Induction
53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)
Yonatan Bisk
, Julia Hockenmaier

Paper BibTeX
Labeled Grammar Induction with Minimal Supervision
53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)
Yonatan Bisk
, Christos Christodoulopoulos, Julia Hockenmaier

Paper BibTeX   Poster
An HDP Model for Inducing Combinatory Categorial Grammars
Transactions of the Association for Computational Linguistics
Yonatan Bisk
, Julia Hockenmaier

Paper BibTeX   Slides
Simple Robust Grammar Induction with Combinatory Categorial Grammar
Twenty-Sixth Conference on Artificial Intelligence (AAAI-12)
Yonatan Bisk
, Julia Hockenmaier

Paper BibTeX
Document-Topic Hierarchies from Document Graphs
21st ACM international conference on Information and knowledge management (CIKM 2012)
Tim Weninger,
Yonatan Bisk
, Jiawei Han

Paper BibTeX
Normal-form parsing for CCGs with generalized composition and type-raising
23rd International Conference on Computational Linguistics (Coling 2010)
Julia Hockenmaier,
Yonatan Bisk

Paper BibTeX

Preprints, Tech reports, and Workshops

SPRING: GPT-4 Out-performs RL Algorithms by Studying Papers and Reasoning
ArXiv
Yue Wu, Shrimai Prabhumoye,
So Yeon Min
,
Yonatan Bisk
, Ruslan Salakhutdinov, Amos Azaria, Tom Mitchell, Yuanzhi Li

Paper BibTeX
Plan, Eliminate, and Track -- Language Models are Good Teachers for Embodied Agents
ArXiv
Yue Wu,
So Yeon Min
,
Yonatan Bisk
, Ruslan Salakhutdinov, Amos Azaria, Yuanzhi Li, Tom Mitchell, Shrimai Prabhumoye

Paper BibTeX
Jared Fernandez
The Framework Tax: Disparities Between Inference Efficiency in Research and Deployment
ArXiv
Jared Fernandez
, Jacob Kahn, Clara Na,
Yonatan Bisk
, Emma Strubell

Paper BibTeX
So Yeon Min
Self-Supervised Object Goal Navigation with In-Situ Finetuning
ArXiv
So Yeon Min
, Yao-Hung Hubert Tsai, Wei Ding, Ali Farhadi, Ruslan Salakhutdinov,
Yonatan Bisk
, Jian Zhang

Paper BibTeX
Vidhi Jain
MAEA: Multimodal Attribution for Embodied AI
Progress and Challenges in Building Trustworthy Embodied AI at NeurIPS 2022
Vidhi Jain
, Jayant Sravan Tamarapalli, Sahiti Yerramilli,
Yonatan Bisk

Paper BibTeX
Tackling AlfWorld with Action Attention and Common Sense from Language Models
Language and Reinforcement Learning Workshop at NeurIPS 2022
Yue Wu,
So Yeon Min
,
Yonatan Bisk
, Ruslan Salakhutdinov, Shrimai Prabhumoye

Paper BibTeX
Retrospectives on the Embodied AI Workshop
ArXiv
Matt Deitke, Dhruv Batra,
Yonatan Bisk
, Tommaso Campari, Angel X. Chang, Devendra Singh Chaplot, Changan Chen, Claudia Pérez D'Arpino, Kiana Ehsani, Ali Farhadi, Li Fei-Fei, Anthony Francis, Chuang Gan, Kristen Grauman, David Hall, Winson Han, Unnat Jain, Aniruddha Kembhavi, Jacob Krantz, Stefan Lee, Chengshu Li, Sagnik Majumder, Oleksandr Maksymets, Roberto Martín-Martín, Roozbeh Mottaghi, Sonia Raychaudhuri, Mike Roberts, Silvio Savarese, Manolis Savva, Mohit Shridhar, Niko Sünderhauf, Andrew Szot, Ben Talbot, Joshua B. Tenenbaum, Jesse Thomason, Alexander Toshev, Joanne Truong, Luca Weihs, Jiajun Wu

Paper BibTeX
Training Vision-Language Transformers from Captions Alone
ArXiv
Liangke Gui, Qiuyuan Huang, Alex Hauptmann,
Yonatan Bisk
, Jianfeng Gao

Paper BibTeX   Code
Worst of Both Worlds: Biases Compound in Pre-trained Vision-and-Language Models
4th Workshop on Gender Bias in Natural Language Processing
Tejas Srinivasan,
Yonatan Bisk

Paper BibTeX
KB-VLP: Knowledge Based Vision and Language Pretraining
ICML21 Workshop on Self-Supervised Learning
Kezhen Chen, Qiuyuan Huang,
Yonatan Bisk
, Daniel McDuff, Jianfeng Gao

Paper BibTeX
A Benchmark for Structured Procedural Knowledge Extraction from Cooking Videos
Workshop on NLP Beyond Text
Frank F. Xu, Lei Ji, Botian Shi, Junyi Du, Graham Neubig,
Yonatan Bisk
, Nan Duan

Paper BibTeX   Code
Multi-View Learning for Vision-and-Language Navigation
ArXiv
Qiaolin Xia, Xiujun Li, Chunyuan Li,
Yonatan Bisk
, Zhifang Sui, Jianfeng Gao, Yejin Choi, Noah A. Smith

Paper BibTeX
FIND: Identifying Functionally and Structurally Important Features in Protein Sequences with Deep Neural Networks
bioRxiv
Ranjani Murali, James Hemp, Victoria Orphan,
Yonatan Bisk

Paper BibTeX
Inducing Grammars with and for Neural Machine Translation
2nd Workshop on Neural Machine Translation
Ke Tran,
Yonatan Bisk

Paper BibTeX
Bridging HMMs and RNNs through Architectural Transformations
32nd Conference on Neural Information Processing Systems (NIPS 2018), IRASL workshop
Jan Buys,
Yonatan Bisk
, Yejin Choi

Paper BibTeX
CHALET: Cornell House Agent Learning Environment
ArXiv Preprint 1801.07357
Claudia Yan, Dipendra Misra, Andrew Bennett, Aaron Walsman,
Yonatan Bisk
, Yoav Artzi

Paper BibTeX   Simulator
Towards a Dataset for Human Computer Communication via Grounded Language Acquisition
AAAI 2016 Workshop on Symbiotic Cognitive Systems
Yonatan Bisk
, Daniel Marcu, William Wong

Paper BibTeX
Unsupervised Neural Hidden Markov Models
Workshop on Structured Prediction for NLP
Ke Tran,
Yonatan Bisk
, Ashish Vaswani, Daniel Marcu, Kevin Knight

Paper BibTeX   Slides
Unsupervised Grammar Induction with Combinatory Categorial Grammars
University of Illinois at Urbana-Champaign
Yonatan Bisk

Paper BibTeX
Induction of Linguistic Structure with Combinatory Categorial Grammars
NAACL HLT Workshop on Induction of Linguistic Structure
Yonatan Bisk
, Julia Hockenmaier

Paper BibTeX