We live in a world where robots are being used to finish a growing number of everyday tasks, from the mundane folding of clothes to the intricate task of tying a knot. But, as it turns out, teaching a robot to fold a shirt or tie a knot isn’t as easy as it might seem.
The challenge of training robots to manipulate fabrics has stumped researchers for years. Most current approaches lean heavily on imitation learning, where robots are trained using videos or motion capture footage of humans performing the tasks. While this technique has shown promise, it comes with a hefty price tag. Gathering large volumes of high-quality human demonstration data is both time-consuming and expensive.
Learning from humans
Taking a book out of human reality, researchers have turned the problem on its head with a novel approach that could revolutionise how robots learn to manipulate fabrics. Their secret weapon? The vast ocean of tutorial videos posted online every day.
“This work begins with a simple idea, that of building a system that allows robots to utilise the countless human demonstration videos online to learn complex manipulation skills,” explains study author Weikun Peng. “In other words, given an arbitrary human demonstration video, we wanted the robot to complete the same task shown in the video.”
Previous studies have used domain-specific videos for robot training, meaning the robots were trained using videos shot in the same environment where they would later perform the tasks. But Peng and his team took a bolder step, creating a framework that allows robots to learn from any video, regardless of the setting.
Tying the knot
Their approach consists of three main components: Real2Sim, Learn@Sim, and Sim2Real. The magic begins with Real2Sim, which replicates the human demonstration in a simulated environment, producing a sequence of object meshes that represent the object’s trajectory. Next comes Learn@Sim, where the robot learns the optimal grasping and placing points through reinforcement learning. Finally, Sim2Real takes the learned policies and applies them to a real dual-arm robot, with a residual policy to bridge the gap between simulation and reality.
The team put their approach to the test by teaching a robot to knot a tie, a task notoriously difficult for machines. Remarkably, the robot succeeded, showcasing the potential of this new training method.
Looking ahead, Peng and his colleagues envision expanding this Real2Sim approach to other complex tasks. The ultimate goal is to replicate the real world in simulation, addressing the data scarcity problem plaguing the robotics community.
“If we can replicate an object’s motion in simulation, could we replicate the real world in simulation?” Peng muses.
As robots become increasingly adept at tasks once thought impossible, the day when they seamlessly fold our clothes and knot our ties is not far off. And it all started with a simple idea and the wealth of human demonstrations freely available on the internet.
The findings of this research have been published in a preprint journal.
