Case Study · Telerobotics / ROS2
Telerobotic Digital Twin for the xArm 1S
Hand-tracked XR control connected through ROS2 to a real-time Unity digital twin.
A functional XR telerobotics prototype that connects a hand-tracked Meta Quest 3 control interface with ROS2 and a Unity-based xArm 1S digital twin. Joint commands are published through ROS2 and mapped in real time to the robot’s articulated joints, enabling intuitive remote control while providing a foundation for future integration with a physical xArm 1S robot.
- ROLE
- Lead Developer · XR & Robotics Engineer
- YEAR
- 2025
TECH_STACK
Technologies
MEDIA
Project media
CONTEXT
Technical context
Robotic systems are often controlled through conventional interfaces that can limit operator mobility and make complex multi-joint manipulation less intuitive. This project investigates how immersive XR interaction and ROS2 communication can provide a more natural control method while maintaining a synchronized digital representation of the robot. The system combines a hand-tracked Meta Quest 3 interface, ROS2 middleware running on Ubuntu and a Unity desktop application containing a 6-DOF xArm 1S digital twin. The prototype was designed as a foundation for future physical robot integration, allowing the same control pipeline to be extended from simulation to real-world telerobotic operation.
PROBLEM
Problem
Traditional robot control interfaces can be difficult to operate intuitively, particularly when managing multiple joints remotely. Physical controllers can also restrict the operator’s natural hand movement, while disconnected simulation and control systems make it harder to verify commands before applying them to a real robot. The challenge was to create a unified control pipeline that could translate hand-tracked XR interactions into ROS2 joint commands and reflect those commands immediately on a synchronized Unity digital twin. The system also needed to preserve robot joint limits, support all six degrees of freedom and remain extensible for future control of a physical xArm 1S.
OVERVIEW
Project overview
The system consists of two Unity applications connected through ROS2 middleware running on Ubuntu. The Meta Quest 3 application acts as the operator interface, while the desktop application provides a real-time digital representation of the xArm 1S robot.
Inside the XR application, the operator uses hand tracking to interact with a virtual control panel. Dedicated controls adjust the position of each of the robot’s six joints, eliminating the need for handheld controllers and enabling direct interaction through natural hand movements.
The XR application converts joint commands into ROS2 JointState messages and publishes them through the /joint_states topic. ROS2 serves as the communication layer that transfers the joint data from the immersive interface to the desktop simulation.
The desktop Unity application subscribes to the ROS2 joint-state stream through the Unity ROS TCP Connector. Received positions are mapped to the corresponding joints, converted from radians to degrees and applied to the ArticulationBody drive targets of the URDF-based xArm 1S model.
The resulting architecture keeps the six-degree-of-freedom digital twin synchronized with the commands issued from the Meta Quest 3 interface. Joint limits, articulation properties and mirrored gripper movement are preserved within the Unity simulation.
The prototype was designed as a modular foundation for future physical deployment. A ROS2-compatible physical xArm 1S could be connected to the same communication pipeline, extending the system from digital-twin validation to real-world telerobotic control.
APPROACH
Technical approach
- Designed a distributed architecture connecting a Meta Quest 3 XR application, ROS2 middleware on Ubuntu and a Unity desktop digital twin.
- Built a hand-tracked XR control panel with interactive buttons for independently controlling all six robotic joints.
- Published joint commands from the XR application as ROS2 JointState messages through the /joint_states topic.
- Configured ROS2 as the communication bridge between the XR interface and the desktop simulation.
- Imported and configured the xArm 1S as a URDF-based articulated robot model in Unity.
- Mapped incoming ROS2 joint names and positions to the corresponding Unity ArticulationBody joints.
- Converted received joint positions from radians to degrees and applied them as real-time articulation drive targets.
- Preserved joint limits, physical drive properties and synchronized movement of the robotic gripper.
- Structured the system so the digital twin can later be replaced or extended with a physical ROS2-connected xArm 1S robot.
FOCUS
Technical focus
- XR Telerobotics
- ROS2 Communication Architecture
- Hand-Tracked Robot Control
- Digital Twin Synchronization
- 6-DOF Joint Control
- ROS2 JointState Messaging
- URDF-Based Robot Simulation
- Unity ArticulationBody Physics
- Real-Time Joint Mapping
- Meta Quest 3 Interaction
- Distributed XR-to-Robot Control
- Physical Robot Integration Readiness
ARCHITECTURE
Architecture notes
- The system is divided into a Meta Quest 3 XR client, ROS2 middleware on Ubuntu and a Unity desktop digital-twin application.
- The XR client captures hand-tracked button interactions and converts them into target positions for six robotic joints.
- Joint commands are encoded as sensor_msgs/JointState messages and published through the /joint_states ROS2 topic.
- ROS2 acts as the middleware layer between the XR control interface and the desktop simulation.
- The Unity desktop application subscribes to /joint_states through the Unity ROS TCP Connector.
- Incoming joint names are mapped to the corresponding ArticulationBody components in the imported xArm 1S model.
- Joint positions are transmitted in radians, converted to degrees and applied as ArticulationDrive target values.
- The robot is represented as a URDF-based articulated hierarchy with configured joint limits and physical drive properties.
- The gripper uses mirrored target positions to keep its left and right components synchronized.
- The communication and control layers are modular, allowing the digital twin to be extended or replaced with a physical ROS2-connected xArm 1S.
HIGHLIGHTS
Engineering highlights
- Functional end-to-end XR telerobotics prototype
- Hand-tracked Meta Quest 3 control interface
- Controller-free interaction through virtual XR buttons
- Independent control of six robotic joints
- Real-time ROS2 JointState communication
- Unity-based xArm 1S digital twin
- URDF-based articulated robot model
- ROS2-to-Unity joint synchronization
- Joint-limit-aware ArticulationBody control
- Synchronized two-sided gripper movement
- Modular architecture prepared for physical robot integration
- Localhost-tested communication pipeline extensible to networked deployment
OUTCOME
Outcome
The project resulted in a functional end-to-end telerobotics prototype that successfully connects hand-tracked XR input, ROS2 communication and a real-time Unity digital twin. Operators can control all six joints of the xArm 1S through a Meta Quest 3 interface, with commands transmitted through ROS2 and reflected immediately in the articulated desktop simulation. The prototype validates the proposed communication architecture and demonstrates how immersive control can be integrated with robotic middleware and digital-twin technology. It also establishes a reusable foundation for future expansion toward physical xArm 1S control, additional safety mechanisms, feedback visualization and more advanced manipulation workflows.