AI might assist cut back damage threat in pianists

Researchers at Stanford Engineering have developed an AI-trained mannequin to precisely recreate the hand actions of elite-level pianists and the bodily stresses they endure whereas enjoying.

Skilled piano gamers spend numerous hours on the keys, perfecting their craft. For folks with smaller arms, this dedication can take a bodily toll. Repeated stretching to succeed in distant keys in a chord can pressure muscle tissues and joints and should result in tendonitis, carpal tunnel syndrome, and different accidents.

Researchers at Stanford are working to know the forces concerned within the hand actions of elite-level piano gamers, and the way these change with totally different hand sizes and keyboard widths. On Dec. 5 at SIGGRAPH Asia 2024, Stanford Engineering researchers offered an AI-trained mannequin that may recreate the hand actions required to play sophisticated items of music and the bodily stresses these arms are beneath.

This is step one in an effort to cut back the danger of long-term accidents in piano gamers and check options—resembling narrower keyboards—that would make piano enjoying extra equitable for everybody.

“We’d by no means anticipate a world-class athlete to compete with tools that doesn’t match their physique. But we ask pianists, notably ladies, to adapt to a one-size-fits-all design that was by no means constructed with them in thoughts,” mentioned Elizabeth Schumann, the Billie Bennet Achilles Director of Keyboard Research at Stanford and co-author on the paper. “We will use analysis like this to reshape the way forward for efficiency and make it extra sustainable.”

Capturing elite efficiency

The trendy piano keyboard was designed within the nineteenth century with the typical European man in thoughts. At this time, an estimated 87% of grownup ladies and 24% of grownup males have arms which are smaller than excellent for the usual piano. Utilizing conventional strategies to check the impacts of smaller arms would require following a cohort of piano gamers for many years, and it might nonetheless be tough to quantify damage dangers or check options.

As an alternative, the researchers have recorded the hand actions {of professional} piano gamers and are utilizing synthetic intelligence to foretell how totally different sizes of arms would transfer to play new music.

Schumann and her colleagues recruited 15 elite-level pianists to play a complete of 10 hours of music whereas cameras filmed their arms from each angle. The researchers could not put any sensors on the gamers with out doubtlessly interfering with their efficiency, in order that they used superior laptop imaginative and prescient strategies to mix the movies and reconstruct the gamers’ hand motions in three dimensions. They used further processing to be sure that these actions have been completely synced with the audio.

“The standard of information that we have been capable of obtain is unprecedented,” mentioned Karen Liu, a professor of laptop science at Stanford and lead writer on the paper. “It isn’t simply that the reconstructed 3D motions are correct and clear; the standard of the efficiency itself is really outstanding. We have been capable of seize various actions from a bunch of gifted musicians acting at a excessive degree—I’ve not seen a dataset like that previously.”

Roucheng Wang, a graduate pupil in Liu’s lab, and Pei Xu, a postdoctoral researcher, used this dataset to coach a mannequin to precisely generate new piano-playing information. After they gave the mannequin sheet music it had by no means seen, resembling Beethoven’s “Für Elise,” the mannequin might present the proper 3D hand actions wanted to play the piece, utilizing a simulated piano to generate sound.

“I used to be simply so surprised at how precisely this mannequin might simulate elite-level approach,” Schumann mentioned. “It is actually unimaginable.”

From robotic joints to actual muscle tissues

This mannequin is simply step one for Liu, Schumann, and their colleagues. In the meanwhile, the mannequin can bodily simulate hand actions, however it would not simulate the muscle tissues and tendons that create these actions or the pressure that they’re subjected to.

“We’re not on the biomechanics degree but, the place the actuation of the arms is generated by muscle tissues—at the moment our mannequin is extra like a robotic that immediately generates torque at joints,” Liu mentioned. “Ultimately, our mannequin wants to have the ability to predict the strain degree of muscle tissues and the potential of damage.”

The researchers are at the moment working so as to add these layers to their mannequin in order that they will use it to assist piano gamers—particularly these with small arms—keep away from accidents. They’re additionally adapting their work to mannequin the actions of different dexterous musicians. They’ve utilized a number of the identical strategies to an current dataset of hand motions of extremely expert guitar gamers.

Their outcomes, which have been additionally offered not too long ago at SIGGRAPH Asia 2024, confirmed that they might efficiently mannequin the distinct left- and right-hand actions wanted to play a wide range of each widespread and classical music.

“With high-quality information, we might mannequin the 3D actions wanted for different kinds of music efficiency as effectively,” Liu mentioned. “This is not a mannequin that’s changing folks—we’re working with musicians to assist perceive and remedy issues. This undertaking is about advancing folks, and AI is only a device for that.”