Google researchers have released a new artificial intelligence called SensorLM, which learns the “language” of smartwatch health sensors to bridge the gap between raw data and real-world contexts.
Have you ever looked at your smartwatch and not understood what those numbers meant? Your device tracks every step and every heartbeat, but it can’t tell you the story behind the data. A heart rate of 150 bpm might represent an energetic run or a stressful work presentation; your watch simply can’t tell the difference. That’s exactly what Google’s SensorLM aims to address.
The biggest challenge lies in the data itself. To understand the connection between sensor signals and everyday life, AI needs to learn from millions of hours of pre-annotated textual descriptions of sample data. Having people manually record millions of hours of sensor data is virtually impossible.
Therefore, the Google team developed a system that automatically generates descriptive captions for sensor data. This approach allowed them to build the largest known dataset of sensor language data, using 59.7 million hours of data from over 103,000 people.
SensorLM learns primarily through two methods:
1.It is trained through contrastive learning, becoming an excellent detective. Contrastive learning taught it to distinguish between similar but different activities; for example, it could correctly identify “light swimming” and “strength training” based solely on sensor signals.
2.It was trained through generative pre-training to become an excellent “storyteller.” In this process, the AI learned how to compose human-readable descriptions based on its observations of complex sensor data.
In tests classifying 20 different activities (without any specific preparation, i.e., a “zero-shot” task), SensorLM performed exceptionally well with extremely high accuracy. In contrast, some other powerful language models essentially had to guess.
Beyond categorizing activities, SensorLM generates precise summaries. From raw sensor data streams alone, it produces detailed and coherent descriptions of events. For example, it can detect an outdoor bike ride, a subsequent walk, and a period of sleep down to the minute.
Research shows that its performance continuously improves as the model size increases and training data grows. This paves the way for truly personalized digital health coaches, clinical monitoring tools, and health applications that can provide advice through natural conversation.
We are moving beyond an era focused solely on simple metrics. With innovative technologies like Google SensorLM, we are moving towards a future where wearable devices — like our smartwatches — truly understand the language of our bodies and transform massive amounts of data into personalized and actionable insights.