Telerobotic Digital Twin for the xArm 1S
Hand-tracked XR robot control with ROS 2 communication and real-time Unity articulation
An XR telerobotics system connecting hand-tracked robot controls to an articulated Unity digital twin through ROS 2 communication.
- Role
- Software Engineer — XR & Robotics Systems
- Year
- 2025
Technologies
Project media
Technical context
Remote robotic manipulation requires coordination between operator input, communication middleware, and robot motion. Immersive interfaces offer an alternative to conventional controls, while digital twins provide a visual representation of commanded joint movements.
Problem
Connect an immersive robot-control interface to a separate articulated simulation while maintaining consistent joint identification, angular representations, and responsive motion across the communication pipeline.
Project overview
The system consists of two Unity applications connected through ROS 2 middleware running on Ubuntu under WSL. The Meta Quest 3 application provides a hand-tracked control panel with independent controls for six robot joints. User interactions update joint targets, which are periodically published as sensor_msgs/JointState messages through the Unity ROS TCP Connector.
The desktop application receives joint-state messages and maps their identifiers to the corresponding components of a URDF-based xArm 1S model. Incoming angles are converted from radians to degrees and applied through Unity ArticulationBody drives. Configurable drive properties and mirrored gripper targets govern movement within the simulation.
Technical approach
- Develop a Meta Quest 3 control interface using hand-tracked interactions and virtual joint controls
- Implement continuous joint adjustments and ROS 2 JointState message publishing
- Connect the XR application and desktop simulation through ROS 2 middleware running on Ubuntu
- Import the xArm 1S URDF model and configure its articulated hierarchy in Unity
- Map incoming joint positions to articulation drive targets with coordinated gripper movement
Technical focus
- XR interaction design and hand-tracked robot control
- ROS 2 communication between independent Unity applications
- Robot joint-state serialization and angular transformations
- URDF-based articulated modeling and physics-driven motion
- Real-time digital twin control and joint synchronization
Architecture notes
- Meta Quest 3 Unity client translating virtual button interactions into six joint targets
- ROS TCP Connector publishing sensor_msgs/JointState messages to the /joint_states topic
- ROS 2 middleware on Ubuntu providing the communication bridge between applications
- Desktop Unity subscriber mapping ROS joint identifiers to ArticulationBody components
- URDF-based robot hierarchy with configurable articulation drives and mirrored gripper targets
Engineering highlights
- Controller-free XR interface with press-and-hold controls for independent joint movement
- ROS 2 communication pipeline connecting separate immersive and desktop applications
- Six-joint command publishing with degree-to-radian and radian-to-degree conversion
- URDF-based robot simulation with configurable stiffness, damping, and drive force
- Coordinated gripper movement through mirrored articulation targets
Outcome
Implemented an end-to-end XR-to-ROS 2-to-Unity control pipeline linking hand-tracked input with a six-joint articulated robot simulation. The system demonstrates coordinated joint control across separate applications, with ROS 2 handling communication between the immersive interface and the digital twin.