Real-world robot learning
Accepted to CoRL 2026
1 HKUST (Guangzhou)2 HKUST3 ETH Zurich
The full picture
While multi-fingered dexterous robot hands have the potential to provide human-level dexterity, using them for functional manipulation remains difficult because task success depends on precise contact, coordinated arm-hand motion, and reactive recovery from execution errors. Precise contact depends on complex contact dynamics that are difficult to model accurately in simulation, motivating real-world learning. However, real-world demonstrations in the full arm-hand control space are costly and noisy because each trajectory must coordinate arm motion with high-DoF multi-finger contact. Small execution deviations can further cause slipped contact or premature force application, making recovery from off-nominal contact states important for robust execution.
To deal with these challenges, we propose HiFun, a hierarchical real-world RL framework for functional dexterous manipulation tasks. Our key insight is to decouple the control complexity into a functional hand skill for precise contact execution and an arm-hand coordination policy for arm control and hand-skill activation. This design enables effective real-time human intervention for reactive recovery learning and efficient data collection with Dynamic Movement Primitive (DMP) augmentation. Across six real-world tasks on two dexterous hands, HiFun achieves 98.3% average success within one hour of online training and remains robust to dynamic disturbances.
01 / From challenges to design
HiFun structures what the robot explores, how actions are evaluated, and how human guidance enters learning.
01 / Challenge
Direct arm–hand RL must discover arm motion, coordinated finger contacts, and activation timing in a coupled action space. Successful contact sequences occupy only a small part of that space.
HiFun insight
Learn finger execution separately, then let the coordination policy explore 6D arm motion and hand-skill activation.
02 / Challenge
The same unsuccessful outcome can result from poor arm alignment or mistimed hand-skill activation. A terminal success signal alone does not distinguish these causes.
HiFun insight
Use distinct navigation and modulation targets, with action-subspace gradient routing, to guide arm motion and skill timing.
Method overview
Experimental evidence
03 / Challenge
Real-time correction of many coupled finger joints is difficult to deliver consistently, especially when human and robot hand morphologies differ.
HiFun insight
Operators correct arm placement and skill activation through the learned hand skill, reducing the need to teleoperate individual finger joints.
04 / Challenge
Human corrections can be noisy, delayed, or hesitant. Their usefulness also depends on what the current policy would do in the same state.
HiFun insight
IAW weights each correction relative to the current policy action, giving stronger imitation weight to corrections with higher critic-estimated advantage.
Method overview
successful evaluation trials
functional manipulation tasks
dexterous hand embodiments
online HIL per task*
98.3% mean success. 50 trials per task within the trained workspace. Online HIL is one training stage; the full pipeline averages approximately 109 min/task. Evaluation details ↗
02 / What makes these tasks hard
Precise tool use, contact-rich rotation, and constrained-space manipulation require precise fingertip placement and force regulation, synchronized arm–hand motion, and recovery after small errors. Here, a narrow opening blocks direct grasping. The robot must lever the exposed handle into reach, form a precision grasp, and extract it without slipping, trapping the tool, or colliding with the opening.
Place the fingers on the exposed, task-relevant part of the handle.
Synchronize arm motion and multi-finger levering to bring the handle within reach.
Transition to a precision grasp without losing contact.
Maintain coordinated arm motion and finger contact to avoid trapping or collision at the opening.
Task challengeSmall contact errors can break an entire manipulation sequence. Recovery must be learned alongside precise contact and coordination.
03 / Experimental evidence
Three views of the evidence: adapting finger contact, learning arm–skill coordination, and using corrections selectively.
Contact-aware hand skills
A collision rotates the key and breaks thumb contact. The residual hand skill adjusts the fingers to restore contact without arm re-alignment.
Fixed-arm Angle-Spreader skill. Toolbox yaw 0–30°; tool pitch/roll 0–20°.
Fixed-arm Bit Removal skill, with drill yaw up to 30°.
Fixed-arm Angle-Spreader skill under strong wrench perturbations.
These experiments support local recovery around a demonstrated contact structure. Additional evaluation, rebuttal ↗
Skill-conditioned coordination
Giving a baseline access to the same learned skill improves performance. Learning where to align the arm and when to activate that skill closes a further gap.
Separate from the six-task 98.3% headline result. Rebuttal comparison ↗
Value-guided learning
Separate value heads train spatial alignment and skill activation. Intervention advantage weighting (IAW) gives more imitation weight to corrections with higher estimated value than the current policy action.
Multi-head critic
95th-percentile TCP-to-target error at activation, measured on Angle-Spreader Retrieval.
Intervention advantage weighting
The same success rates are reported on each of Angle-Spreader Retrieval and Bit Removal.
04 / Method
A learned hand skill executes precise contact. A coordination policy controls arm motion and hand-skill activation through a compact interface.
RGB images · arm–hand state · contact feedback
Kinesthetic reference + residual RL
Learned arm–hand coordination
The activation value blends the hand-skill command with the neutral hand configuration.
05 / Real-world evaluation
Each task is evaluated over 50 trials with randomized object poses inside the trained workspace. Success requires completing the functional outcome.
Tighten the target screw while maintaining alignment and actuating the trigger.
Establish fingertip contact and fully extract the tool from its constrained tray.
Lever an exposed handle into reach, form a precision grasp and fully extract it.
Maintain multi-finger contact while loosening the chuck, then remove the drill bit.
Across hand embodiments
HiFun deploys on both Sharpa and XHand. On XHand, it performs precise pipette dispensing and retrieves a thin tool from a constrained space.
Three vial positions, one dispensing skill. Align the tip with each vial and dispense liquid without collision or leakage.
Use a fingertip contact sequence to fully retrieve a thin shaft embedded in foam.
Beyond nominal rollouts
HiFun adjusts its approach after a tray moves and reuses a learned hand skill across different tool sizes.
Long-horizon manipulation
Here, HiFun runs on XHand in a longer functional sequence. The policy turns the key, grasps the drill, and places it into the drawer by coordinating different hand skills. This shows that HiFun can deploy across dexterous hands and integrate multiple skills in one policy.
06 / Learning cost & autonomy
Online HIL fine-tuning averages 46.2 minutes per task. Including hand-skill learning and preparation, the reported full pipeline averages approximately 109 minutes per task.
Intervention rate complements success: it measures how much human correction is still needed. The original curves below show intervention rate and cycle time across the six tasks.
07 / Explore the work
Read the method and evaluation, watch the overview video, or explore the additional evidence introduced during review.
HiFun relies on task-relevant hand skills learned around kinesthetic priors. New contact structures may require additional priors or adaptation. The current evaluation establishes functional execution within trained workspaces and local contact variations, rather than language-conditioned or open-ended task generalization.
Reusable hand-skill libraries and integration with high-level task-selection models are research directions. They are not evaluated capabilities of the current system.
@inproceedings{huang2026hifun,
title = {HiFun: a Hierarchical Framework for Efficient
Functional Dexterous Manipulation Learning},
author = {Huang, Linyi and Huai, Guowei and Liu, Weibin and
Jiang, Shulong and Tan, Ping and Zhang, Hui and Song, Jie},
booktitle = {Conference on Robot Learning (CoRL)},
year = {2026}
}