john so

aspiring full-stack roboticist!

[johnso backwards] <at> gmail <dot> com

{ twitter, linkedin, github, scholar }

I received my MS CS at Stanford University, where I was fortunate to be advised by Shuran Song as a member of Stanford REAL (Robotics and Embodied Artificial Intelligence Lab), and my BS EECS from UC Berkeley, where I was advised by Pieter Abbeel, Stephen James, and Xingyu Lin as a part of Berkeley RLL (Robot Learning Lab).

I previously spent time as an intern at Tesla Optimus, where I focused on policy learning from non-embodied data, and as an AI resident at 1X, where I focused on data and training foundation policy models.

research

My dream is for robots to become an everyday household occurrence, and I believe that large scale data is fundamental to bridge this gap between dexterity and robust generalization to unseen scenarios. My recent interests include:

  1. Generalization: How do we elicit task knowledge from pre-trained models via prompting and conditioning? How do we ground pre-trained internet representations for robotic control?
  2. Data: How can we learn priors for control and skills from non-embodied data, such as human videos and simulation? How do we scale, curate, and augment embodied data?

Long term, I hope to leverage perspectives from cognitive science and developmental psychology to inform how robots can learn from and like humans.

Any-point Trajectory Modeling for Policy Learning

Chuan Wen*, Xingyu Lin*, John So*, Qi Dou, Kai Chen, Yang Gao, Pieter Abbeel

TL;DR: We condition a policy on arbitrary point trajectories learned from actionless videos, enabling sample-efficient policy learning and positive cross-embodiment transfer.

RSS 2024 { paper, arXiv, website, code }

SpawnNet: Learning Generalizable Visuomotor Skills from Pre-trained Networks

Xingyu Lin*, John So*, Sashwat Mahalingam, Fangchen Liu, Pieter Abbeel

TL;DR: We propose a method to adapt dense features from pre-trained vision backbones for visuomotor policies, enabling sample-efficient generalization to unseen objects.

ICRA 2024 { paper, arXiv, website, code }

Sim-to-Real via Sim-to-Seg: End-to-end Off-road Autonomous Driving Without Real Data

John So*, Amber Xie*, Sunggoo Jung, Jeffrey Edlund, Rohan Thakker, Ali Agha-mohammadi, Pieter Abbeel, Stephen James

TL;DR: We train a navigation policy in simulation with RL in learned segmentation space, and deploy zero-shot to a real vehicle.

CoRL 2022 { paper, arXiv, website, code }

miscellaneous

I'm forever grateful to the communities which raised me, notably Machine Learning at Berkeley (ML@B), 61A Staff, Accel Scholars, and Felicis Fellows.

In particular, I spent the majority of my undergrad learning from and organizing ML@B, serving as the organization's president in Fall 2022. We presented a white paper about our structure and initiatives at the NeurIPS 2022 Broadening Research Collaborations in ML Workshop; you may find a preprint here.

Sometimes, I write my thoughts down! Generally, I like to think about how to best teach, learn, and optimize for fulfillment; shoot me an email or DM if you'd like to chat :)