[00:00] SkillGen: AI system for robotic learning and automation.
[00:19] Core Innovation: Automated dataset generation from minimal human input.
[01:12] Skill Segmentation: Smart system for breaking down and adapting complex tasks.
[01:59] Hybrid Skill Policy: Framework for controlling robot actions and task completion.
[02:50] Performance Results: 75.4% success rate, generating 24,000+ demonstrations.
[04:18] Real-World Testing: 35% success in direct simulation-to-reality transfer.
[05:01] Current Limitations: Preset sequences and object tracking requirements.
[06:47] HSP Variants: Different approaches to robot control and motion planning.
[07:43] Practical Applications: Successful implementation in pick-and-place tasks.
Authors: Caelan Garrett, Ajay Mandlekar, Bowen Wen, Dieter Fox
Affiliation: NVIDIA
Abstract: Imitation learning from human demonstrations is an effective paradigm for robot manipulation, but acquiring large datasets is costly and resource-intensive, especially for long-horizon tasks. To address this issue, we propose SkillMimicGen (SkillGen), an automated system for generating demonstration datasets from a few human demos. SkillGen segments human demos into manipulation skills, adapts these skills to new contexts, and stitches them together through free-space transit and transfer motion. We also propose a Hybrid Skill Policy (HSP) framework for learning skill initiation, control, and termination components from SkillGen datasets, enabling skills to be sequenced using motion planning at test-time. We demonstrate that SkillGen greatly improves data generation and policy learning performance over a state-of-the-art data generation framework, resulting in the capability to produce data for large scene variations, including clutter, and agents that are on average 24% more successful. We demonstrate the efficacy of SkillGen by generating over 24K demonstrations across 18 task variants in simulation from just 60 human demonstrations, and training proficient, often near-perfect, HSP agents. Finally, we apply SkillGen to 3 real-world manipulation tasks and also demonstrate zero-shot sim-to-real transfer on a long-horizon assembly task. Videos, and more at this https URL.
Link: https://arxiv.org/abs/2410.18907
