How Mixed Reality HoloLens User Experience Can Make Life Easier for Retailers

Taras Chaykivskyy
C++ developer at ELEKS

Digital transformation is now among the most rapidly evolving trends. It reshapes almost every aspect of business today, offering new ways to connect with customers, powered by advanced and memorable interactions. While organizations started actively adopting digital innovations, the tech landscape responded immediately, delivering new solutions to meet the emerging demand. Among the latest novelties entering the market over the last few years, solutions that enable virtual reality (VR), augmented reality (AR), and the newly-introduced mixed reality (MR) experience are gaining significant popularity. These tools can completely transform the way people interact with their surroundings, how they communicate, explore, and create new things.

One such solution, called HoloLens, has been recently released by Microsoft. This self-contained, holographic computer allows users to engage with digital content and interact with holograms in the real world around them. The gadget has been received with enthusiasm by the tech-savvy community and some experimental mixed reality HoloLens applications have started to appear.

As soon as we got HoloLens in our R&D Lab, we were very excited to try it out on some real-life use cases. So, we decided to find out the benefits of the HoloLens technology for Retail. Watch this short video to see the result of our experiment and read the article to find out the details.

 

Mixed reality HoloLens technology for Retail – our use case

One of the biggest advantages of mixed reality is that it combines the best features of VR and AR, allowing to lay digital information and believable virtual objects over real world environments. These objects can be anchored to certain points in the real space so that the person that experiences mixed reality treats them as a part of the world around. In other words, mixed reality applications allow visualizing any kind of digital information, to make it a part of a very natural and transparent user experience that can fit into various real-life use cases.
We decided to try to map the opportunities of the mixed reality HoloLens experience on the needs of the retail industry, focusing on range planning, as this part of merchandising is usually a very complex task.

Range planning is an ongoing iterative process and goes far beyond the building of visually-appealing planograms. Making planograms work in-store and maintaining their efficiency further – these are usually the biggest challenges. This process is very data-intensive and, apart from analytical effort, it requires a lot of in-store field work.

So, how can mixed reality fit into the range planning process and what benefits can this technology bring in? Imagine that all the statistics and information required for planogram analysis and monitoring can be transferred from the spreadsheets to brick-and-mortar, represented in the form of a 3D virtual diagram and made accessible at hand at any time for any particular product.
We decided to incorporate this concept into a real life application using Microsoft Hololens to enable a mixed reality experience.

How we mapped the benefits of mixed reality technology onto retailers’ needs

Apparently, store performance review requires the manager to spend a certain amount of time in-store. And, once a manager gets to the brick-and-mortar, there is the challenge of retrieving tons of data needed to perform the review. And with HoloLens, we can visualize all the business analytics and make it available right at the person’s fingertips.

The aim of our experiment was to provide the user with real-time statistical data concerning products available in-store or at the warehouse in a way most suitable to supporting efficient decision making. From the technology point, we wanted to create a system with an intuitive user interface that could assemble data, process it, and present in a form of interactive analytical charts, tables or any other way suitable for the user. To provide the user with information associated with a specific store, the system would scan the store’s wifi to recognize the location and get access to the relevant database.

To receive information about a certain product from this store, a person will only have to look at the store shelf and a corresponding product code will be recognized by the system, triggering holographic data visualization.

hololens holographic data visualization

In this way, the user can access information associated with any product, including the SKU (Stock Keeping Unit), brand and category sales, out-of-stock data, profits per SKU, or any other relevant data that can be pulled from the server. This information can be further compared against the real life state of things to identify cases when, for example, products with certain SKUs are out of stock on the shelf, however, they have significant stock within the system, therefore they are not ordered and supplied to the store frequently.

Actually, there are multiple ways for the manager to work with the visualized analytics. One of the key benefits here is that all the data is available at hand and can be analyzed and compared against the real state of things right on the spot, ensuring reliable input for decision making.

But before we proceed with the implementation process, let’s dive into the details describing the user experience we wanted to provide.

Speaking of HoloLens UX development…let us share some tricks

Now it’s time to mention a few interesting findings we discovered while creating our mixed reality HoloLens user experience. You may find these tips useful when working on your own HoloLens-powered UX.

The first interesting thing we’ve noticed associated with HoloLens UI is with regard to the distance from the user’s eyes to the object that needs to be recognized. We created the rectangular frame that defines for the user the virtual area where the QR code should be placed to get read.

hololens QR code recognition

Then we set up a distance from a virtual camera (the human eyes) to this frame being 50 cm. As we tested these settings with the device, we’ve found that, because with HoloLens the user sees visuals in a stereo mode, the person naturally recognizes the distance to the frame and understands that the QR code needs to be at the same distance from the eyes as the rectangle is to be captured and recognized by the system. So, the system receives some additional information concerning the space that a QR code should occupy within the frame, as well as the area where this code can be found. So, the computation time is reduced significantly.

Another unexpected finding about HoloLens UI was that if you want to have your 3D models projected the way it is done with regular 3D screens, you, most likely, won’t be able to make it work correctly. The thing is that, in case the model gets projected on the edge of one of the HoloLens screens, the user starts noticing the boundary between the real world and the virtual one. Therefore, you will have to do some post-processing on your side to blur the picture on the edges of the screens.

And how things looked from the code side

Technology-wise, our aim was to make our system perform the following key actions:

  1. identify the product;
  2. pull relevant information from the server and prepare statistics;
  3. visualize the statistics;
  4. communicate with the user and respond to interactions.

Speaking about functionality implementation, there was no difficulty associated with points #2 and #3 from our list above. These features are easily enforced with Unity3D, as it offers comprehensive, ready-to-use classes and components that you can employ. The item #4 is also easy to implement using the HoloLensInputModule and applying the business logic of interaction.

And what about #1? Well, this is where we had the most fun. So, I would like to get into the details a bit.

Whether it is about a warehouse or a store, the goods are identified and differentiated with the help of machine-readable codes. For our system, we chose QR code and used ZXing.Net library to enable code reading. As we started, it took up to 1.2 seconds for the system to take a picture, search for the QR code, and have it recognized. Well, things were definitely too slow, even for real-time processing. Since 0.4 seconds out of this time were used to take a picture, and this operation is asynchronous, we could only try to influence the remaining 0.8 seconds.

To speed up performance, we reduced the picture area that the system processed to search for the QR code by 9 times and implemented some optimizations for ZXing.Net, such as removed detection for other types of codes and added a range of the required QR code sizes. As a result, we made our system perform calculations within 0.103 seconds. And to ensure smooth animation, you need to have 30-60 FPS (33-16.7 ms).

Since namespace System.Threading is not available with Unity3D for HoloLens, you won’t be able to establish parallel processing. When looking for ready-to-use solutions, we found that Unity’s coroutines is the best fit. We managed to split method performance into several sub-calls. For example, picture processing was broken into three separate stages:

private void OnPhotoCaptured(PhotoCapture.PhotoCaptureResult result, PhotoCaptureFrame photoCaptureFrame)
{
    _photoProcessState.result = result;
    _photoProcessState.photoCaptureFrame = photoCaptureFrame;
    StartCoroutine(ProcessFrame());
}
private IEnumerator ProcessFrame()
{
    yield return null;

    PreprocessPhoto(_photoProcessState); // #1

    yield return null;

    DetectCodes(_photoProcessState); // #2

    // return;
}

The first yield was called to avoid the overuse of the frame time, since Unity3D performs some generic actions before it executes the callback function. Method #1 is performed over 0.03 seconds approximately, while #2 takes up to 0.06 seconds. This way, you can split performance into some shorter parts to bypass the exceeded frame time and avoid animation lags.

We can also divide the process of code detection into some smaller chunks and perform “heavy” calculations without the influence on high FPS. To make things even faster, use modules from the unity asset store or WinRT API, which is accessible from C# scripts only.

So, why it is worth bringing mixed reality technology to brick-and-mortar?

In short, with HoloLens managers can effectively combine the digital experience with the benefits of the in-store presence. Mixed reality HoloLens technology gives an immediate possibility to analyze the product-related information right when looking at the shelf. Therefore, retailers can benefit from a great combination of extensive amount of data and visualisation technologies used against the real state of things.

hololens visualisation technologies

This can also provide great advantages when negotiating with suppliers, as it is always better to discuss real-life cases and figures in actual stores.

Once the technology becomes more widespread, it can get even broader use in retail: for example, store managers could order stock replenishment right in front of the shelf, etc. And, of course, there are multiple opportunities to improve the in-store experience for shoppers as well. We already have a bunch of ideas here, so stay tuned…

tags

Comments