Research Statement (2022)

I am broadly interested in human-robot interaction (HRI), robotics, AI, augmented reality, and HCI. My overarching goal is to develop capable and understandable robotic systems that communicate efficiently and explain themselves so they fluently collaborate and interact with humans. As robots stand to benefit society, providing solutions to industry, our daily life, and the aging population, my goal is essential for robots to gain more acceptance and retain these benefits.

I reach my goal with a multidisciplinary approach:

1. I draw insights from psychology and cognitive science to enable natural human-robot communication.
2. I develop computational cognitive models and AI algorithms to structure robot explanations.
3. I investigate natural language generation techniques and novel visualization methods for effective robot communication, such as concise referring forms and projector-based augmented reality.
4. I conduct user studies using quantitative and qualitative methods in realistic real-world task settings.

I. Accomplished Work

During my Ph.D. study, I have independently proposed and developed expertise loosely related to the funded grants of my advisor. My current postdoctoral research further broadens my horizons.

A. Fluent Collaboration via Proactive Handover Release

I started my career working on enabling robots to hand objects, an essential daily activity for collaborative and assistive tasks. Seemingly straightforward, it is challenging for robots to achieve the human level of fluency, especially during object transfer. In my HRI paper [C4], I attached force sensors to a robot’s gripers and developed proactive release algorithms to analyze the haptic data to detect object grasp and pull. A user study revealed a reduction of handover time from 3.7s (threshold approach) to 2.7s, achieving higher fluency and ease of taking.

B. Humans’ Preferences in Robot Explanations

During the handover study, participants wanted the robot to explain when it refused to release. In my ICRA handover workshop [W2] and ACM THRI [J3] journal papers, I examined the need and properties of preferred robot explanations with a failure scenario where a robot was asked to hand an almost-reachable cup. The robot conveyed the unreachability non-verbally: doing nothing, looking at it, looking with repeated pointing. Results showed that non-verbal behaviors must accompany verbal explanations and participants prefer them to be in situ and concise.

C. Explanation Generation Algorithms with Behavior Tree

Given these user preferences, I then proposed algorithms with behavior tree (BT) to generate the structure of explanations. BT encapsulates behaviors in a more interpretable tree structure compared to finite state machines with intertwined states, and scales well with its modular and reusable behavior nodes. However, the free-form BT does not have the human explanation structure. In my THRI journal paper [J1], I focused on organizing and representing complex tasks to make them readily explainable. I added structure to BT by framing it as a set of semantic sets and proposed hierarchical explanation generation algorithms. I also proposed algorithms to explain failures and correct them by extracting and inserting a self-contained, dependency-free subtree.

D. Real-World Collaborative Mobile Manipulation Testbed

To improve the generalization of my algorithms, I evaluated them with a multi-task, multi-step collaborative mobile manipulation kitting task, in which a Fetch robot builds kits of gearbox parts for human coworkers to assemble. The robot navigates in a confined environment and collects irregular parts from different stations, with machining involved. As a competition task, my team won second place and published the effort at ICRA 2020 [C5].

With both manipulation and navigation subtasks, the task is rich in challenges that can lead to realistic and robust robotic solutions. Compared to open-space navigation, it helps tackle autonomous near-obstacle navigation (e.g., table) required for proximity manipulation. It also helps develop robust detection algorithms for objects in manipulation. Finally, it advances obstacle avoidance algorithms in confined spaces, such as gearing machines. Besides the ICRA paper, I collaborated with Tufts University to replace the task sequence component with a cognitive architecture to prove the architecture’s adaptability, published at IROS [C6].

E. Explaining Past Behavior in Changing Environment via Replay

The situated and temporal nature of human-robot interaction creates unique challenges for robot explanation generation. Robots and humans physically move objects that will no longer be present at explanation time. In my THRI journal paper [J5], I proposed robot replays and ways to indicate the missing objects by physically replaying past behaviors, with verbal and projection indicators. Participants were asked to tell where a misrecognized object was picked, where a ground obstacle was to explain the detour, and where an object was misplaced. Results showed that physical replay with verbal and projection indicators best conveyed all three missing information.

F. Projector-Based Augmented Reality (AR) for Scalability

For the replay experiment, I found projection markers alone remarkably efficient in communicating navigation goals. I have architected and implemented this spatial AR technology on a Fetch robot with a projector and a pan-tilt unit overhead [C9]. It can also visualize manipulation intent [W3]. Compared to head-worn AR, projector-based AR does not require wearing special hardware, and the visualizations are visible in group and crowd contexts.

G. Cognitive Status-Informed Referential Choice Model

In my postdoc research, I have focused on concise referring forms [W4, C10, C11] for easy-to-follow language and explanations. Precisely, I computationally model and evaluate cognitive status-informed referring form selection. For data collection and human evaluation, I went beyond simple tabletop scenarios and designed a novel interaction task where an instructor teaches a learner a series of construction tasks in four separate quadrants, leading to repeated references to a mixture of present and non-present objects. The task design [W4] won the Best LBR Award, Third Prize at HRI 2022. The human evaluation [C10] won the Best Long-Paper Award at INLG 2022.

H. Reality Mismatch in Augmented Reality (AR) Robot Appendage

Besides projector-based AR, I have expanded to head-worn AR to add virtual body parts to physically limited robots, e.g., enabling armless robots to gesture. Yet, they might have negative impacts due to being virtual. With student researchers, we evaluated AR and physical arms for gesturing [W7, C13]. Results showed that AR body parts have the same benefits as physical morphological components in task performance and subjective experience.

References