john so

johnso <at> stanford <dot> edu

{ github | twitter | linkedin | cv | scholar }

I am an MS CS at Stanford University, where I'm fortunate to be advised by Shuran Song as a member of Stanford REAL (Robotics and Embodied Artificial Intelligence Lab).


I received my BS EECS from UC Berkeley, where I was advised by Pieter Abbeel, Stephen James, and Xingyu Lin as a part of Berkeley RLL (Robot Learning Lab). I most recently spent time as an AI Resident at 1X, where I focused on training foundation policy models for humanoid robots.

research

My dream is for robots to become an everyday household occurrence; an important step is to enable robots to quickly adapt prior knowledge to new scenes or tasks. Towards this, I aim to answer two questions:

  1. How do we pre-train using out-of-distribution data, such as cross-embodiment and actionless data?
  2. How do we scale in-domain data collection to pre-train multi-task and generalizable policies?

My recent interests are in capturing useful priors from large unstructured datasets for robot learning, such as motion and skills. Long term, I hope to leverage perspectives from cognitive science to inform how robots can learn from and like humans.

Any-point Trajectory Modeling for Policy Learning

Chuan Wen*, Xingyu Lin*, John So*, Qi Dou, Kai Chen, Yang Gao, Pieter Abbeel

We learn to generate trajectories of arbitrary points from large actionless video datasets and condition a policy on these learned point trajectories, enabling sample-efficient policy learning and positive cross-embodiment transfer.

RSS 2024 { paper | arXiv | website | code }

SpawnNet: Learning Generalizable Visuomotor Skills from Pre-trained Networks

Xingyu Lin*, John So*, Sashwat Mahalingam, Fangchen Liu, Pieter Abbeel

We propose a novel method to adapt dense features from pre-trained vision backbones for sample-efficienct policy learning and generalization to unseen objects.

ICRA 2024 { paper | arXiv | website | code }

Sim-to-Real via Sim-to-Seg: End-to-end Off-road Autonomous Driving Without Real Data

John So*, Amber Xie*, Sunggoo Jung, Jeffrey Edlund, Rohan Thakker, Ali Agha-mohammadi, Pieter Abbeel, Stephen James

We learn a segmentation model, train a navigation policy in simulation with RL in learned segmentation space, and deploy zero-shot to a real vehicle.

CoRL 2022 { paper | arXiv | website | code }

miscellaneous

As an undergrad, I spent the majority of my time outside of research helping to build, organize, and lead Machine Learning at Berkeley (ML@B), serving as the organization's president in Fall 2022. We presented a white paper about our structure and initiatives at the NeurIPS 2022 Broadening Research Collaborations in ML Workshop; you may find a preprint here.


I like to think about how to best teach, learn, and optimize for fulfillment. Shoot me an email or DM if you'd like to chat :)