Leveraging mixed reality and food product composition data in order to increase saliency of nutritional quality of groceries at the point of sale.
Diet-related non-communicable diseases have become the leading cause of mortality globally, accounting for more deaths than non-diet-related mortality causes combined. We believe that machine learning and data science can support making health-beneficial decisions in our daily lives. Motivated through the advances of related technology and deep learning frameworks, we contribute to this field by piloting HoloSelecta, a mixed reality headset application in the field of health and nutrition.
HoloSelecta leverages existing retail data, including image and ingredient data, in order to visualize the degree of nutritional quality of groceries at the point of sale before making a purchase decision. In the case of HoloSelecta, the scenario of a vending machine was chosen, as it often is a source of unhealthy food items and also hinders checking ingredient data as products are locked behind the machine. With HoloSelecta, wearers of mixed-reality headsets can still ‘see’ the Nutri-Score of the products within the vending machine. HoloSelecta serves as an example for potentially very effective mixed reality mediated just-in-time adaptive intervention right in the moment when relevant activities such as buying a snack are detected.