Top Mixed Reality App Development and Immersive Technology Solution

This story is about JetStyle’s ambition to gain expertise in the areas which will potentially be relevant in the future

About

At JetStyle, we believe that extended reality technologies (AR & VR) are on their way to merge into Mixed reality; and we’ve found a lot of like-minded colleagues that support this vision as well. VR is a niche phenomenon, accessible only if you have a headset. AR is more mass-oriented, but it has its own equipment-related restrictions. Both technologies are developing to blend with each other to ensure a more immersive and affordable experience.

However, there is still no equipment to match the audience’s needs. To be accessible, MR gear has to be lightweight, comfortable for wearing and powerful enough to track users’ movements in real time. When wearables like this appear in the market, they will redefine the way we consume content and interact with the world. While hardware developers are taking huge steps forward and releasing more advanced models, production companies like us need to get ready for the new era too.

Our MR-VR-AR projects

JetXR (XR production subdivision of JetStyle) aims to keep up with the industry, so a few years ago we decided to try our hands with MR apps development — to make sure we step into the big technological leap full-armed. We needed a pet project to work with MR by trial and error and see what UX patterns are. The goal was to develop an app that would give the audience actual benefit from using it. At the same time, JetXR’s Creative Director Alex Markin was going to visit Cuba for the first time... What can be more Cuban than Salsa? While getting ready for the trip, Alex decided to take a few salsa classes — to absorb the local vibe.

Alexey Markin

Head of XR-direction
and co-founder of JetStyle

What I had absorbed first was pain! Offline lessons are fun, but they have their disadvantages too. When you are in class, there are about 20 other people in the room. The teacher’s attention is divided between everyone, and sometimes it’s not enough for your personal learning process. You try to keep up with the instructions, but if you’re not a born dancing king, you might end up feeling like a complete failure.

Challenge

Salsa is a social dance, so learning it with other people makes total sense. However, everybody has their own speed when it comes to acquiring a skill. In salsa, there are a few basic steps that form the foundation of the dance. It’s too early to dance salsa with a partner until you have mastered the basic steps. These elements take every student different amounts of time, simply because of a person’s individual cognitive features. For some, these basic steps are better learnt by yourself — taking your time at your own pace, with no external distractions. Or at a personal lesson with a teacher.

Additionally, it can be frustrating to fall behind the class. What’s the point of visiting dozens of paid classes if all you need is to have a few more quiet hours of mastering that Enchufla step?So we thought of an XR app that would solve a real-world problem and could help you master the basic salsa moves, before you start learning the social aspect of the dance.

Solution

What was the process like? We took a salsa teacher to a motion capture studio, wrapped him up with mo-cap sensors, recorded the steps and started developing the app prototype.

Initially the trainer app was created for Magic Leap headsets. A bit of background: Magic Leap was one of the first headsets for mixed reality. However, at the time it didn’t really meet the mainstream audience expectations (as any existing headset, actually). The headset is pretty costly ($3,299 per a base model) and is used primarily by developers.We used Magic Leap UI Kits and received a pretty unsophisticated app prototype, but it worked. Students could see a virtual dancing teacher or follow the steps highlighted on the floor in front of them.

During the development process it became clear that Magic Leap was not going to get popular in the near future. Their hardware is expensive and we don’t really see how it can get any cheaper. Mixed reality field of view is the size of a post stamp. We needed something affordable and consumer-oriented.

Then Oculus stepped up and released their Quest headset, the market’s first consumer headset. The picture resolution is a bit better than the Magic Leap’s one, and Quest has inside-out tracking. It is made possible with 4 high speed infrared cameras added onto the headset. And what is more important, all of the user’s field of view is renderable (i.e. we can use it for extending reality).

Later Oculus released Pass Through, an experimental API that allows users to see through the cameras. The visuals are black and white; but at that time the feature meant we could add up another layer to this background. Which we did after Oculus developers shared the access to this API to programmers. Basically, with Quest we instantly received a prototype of a mixed reality headset. It might not be the perfect solution, but it’s affordable and available immediately out of the box.

Next generation of Oculus Quest shows off their better quality cameras. They are still black and white, but Oculus Quest 2 has been the closest thing we have to a MR headset.

In early December 2022 we received the latest Meta Quest Pro with higher power and colorful pass through mode. We are planning to adapt the Salsa app to this hardware as well. Next year will bring us Oculus Quest 3 and new opportunities for mixed reality development. To be continued.

Xenia Chuklai, a JetStyle designer, has been one of those who practiced salsa with the app:

Xenia Chuklai

Designer of JetStyle

As a person who is familiar with salsa basics, I’ve been excited to use my experience in a new environment. ‘Salsa MR' is a great tool to fire up the basic steps technique. Offline practice is still a must, though, as salsa (or any other dance style) is so much more than a series of steps.

I love this app’s feature that enables you to see your coach right in front of you. This is super immersive and close to real-life learning experience. Or even better, as you can speed up or slow down your virtual teacher.

As a result,

Salsa MR is the most immersive MR training assistant we can get in the existing technical conditions. It provides users with a friendly interface and a virtual assistant that teaches you salsa moves. Your MR-teacher will not get tired or tell you “Time’s up” and finish the class.

The app patiently teaches you all the basic salsa moves, with whatever speed and pace is most convenient for you. The MR-assistant is here to support your process and eliminate any psychological discomfort.

With the new generation of headsets we will be able to update the application to be among the first contributors in the stores. By now we have made and fixed enough bugs and experimented so much that we can say that we DO know how to make apps for Mixed Reality. Additionally, now we have a solid collection of best practices to use them separately for different projects.