EPREVAEngineer Paper
WEARABLESNewsElectronicsRobotics & automationComputing & AI for engineering

Cornell's EchoMotion tracks 12 body joints using a smartwatch's ultrasound

On an unmodified Samsung Galaxy Watch 6, mean per-joint position error was 5.99 cm across 15 participants; a 12-minute calibration cut error by more than one-third.

a wrist-worn smartwatch on a moving arm, a lab setting with monitoring equipment or a screen showing a 3D joint skeleton, the sense that sound/ultrasound is being emitted and reflected
AI-generated image

Researchers at Cornell University's Smart Computer Interfaces for Future Interactions (SciFi) Lab have developed EchoMotion, a system that estimates full-body pose using only a wrist-worn smartwatch. According to the peer-reviewed paper presented at UbiComp/ISWC in Shanghai on 13 October, the system uses the watch's built-in speaker and microphone to emit inaudible ultrasonic signals and process their reflections, then a lightweight deep learning pipeline infers the 3D positions of 12 body joints. Because it relies on sound rather than cameras, the approach preserves privacy for the wearer and bystanders and generates less data to process, the researchers say.

The researchers first validated the approach with 10 participants using a custom hardware prototype, reporting a mean per-joint position error (MPJPE) of 5.36 cm and a mean per-joint rotation error (MPJRE) of 11.68°. They then tested feasibility with 15 participants on a commercial smartwatch, reporting MPJPE of 5.99 cm and MPJRE of 11.92°. According to the Cornell press release, a one-time, 12-minute calibration cut the average error by more than one-third, to about 6.1 cm — a figure the peer-reviewed abstract does not report. The same AI model, without retraining, also worked on a Samsung Galaxy Watch 7 and a Google Pixel Watch 3.

In a semi-in-the-wild study with 10 participants performing 18 exercises in a small room, a narrow hallway and next to a noisy road, EchoMotion maintained comparable performance, with MPJPE of 6.41 cm and MPJRE of 13.18° — still a controlled study, not a deployment in daily life. The researchers report that furniture and ambient noise did not interfere, but that performance suffered slightly in the hallway because of echoes coming off other people. Lead author Saif Mahmud expects that more data, from thousands of users instead of dozens, would improve interpretation of movements in crowded environments such as the subway.

The prototype's sensing drew about 12 milliwatts, roughly 40 times less than a wearable camera, and a 30-minute session on a Galaxy Watch 6 used only 5% to 6% of the battery, according to the press release. The team says it is developing the technology for fitness tracking and exploring its combination with first-person vision on smart glasses.

Key facts

  • 5.36 cm mean per-joint position error on the custom prototype with 10 participants
  • 5.99 cm mean per-joint position error on a commercial smartwatch with 15 participants
  • 6.41 cm mean per-joint position error in the semi-in-the-wild study with 10 participants
  • About 12 mW sensing power, roughly 40 times less than a wearable camera
  • 5% to 6% of battery used in a 30-minute session on a Galaxy Watch 6

Sources

  1. EchoMotion: Towards Full-Body Pose Tracking with a Single Smartwatch Using Active Acoustic Sensing — Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies — Saif Mahmud, Jiwan Kim, Tianhong Catherine Yu et al.
  2. Ultrasound and AI turn off-the-shelf smartwatch into full-body tracker without cameras — Phys.org (Technology)

Discussion

Technical questions and corrections are welcome. Comments are signed and checked before they appear.

No comments yet.

Sign in to add a typed review comment. Reviews are signed with your name.

More news