POST

Touchless: a new way to connect with museum interactives

Typically, when building interactive displays we have to solve many complex UX problems. In the wake of a global pandemic, there’s now a new challenge to consider — can interactive touchscreens cater to a COVID-19 conscious audience?

A remote control interface that resembles a joystick being displayed on a mobile phone with an iPad displaying a set of instructions for how to control the remote.
Touchless is a new way to control touchscreen based displays using your mobile device.

We love working with the GLAM (galleries, libraries, archives, and museums) sector. At Springlabs we’ve been prototyping a new way for people to safely connect and interact with touchscreens. 

It had to be hardware-less, easy to implement, and a “one-size fits all” solution. Our own handheld device seemed the ideal candidate. Almost everyone has one — that’s right, we’re talking about a mobile phone.

Prototyping Touchless

We started exploring how we might use mobile phones as a remote control to connect with touchscreens. We used Protopie to create working prototypes without the need of any coding. This helped us to validate the idea quickly as we could easily do some quick user testing. For more on this check out our post Bridging ideas and reality with Protopie.

One criterion for this project was that it had to be easy to apply to an existing HTML/Javascript interactive with no need to modify the source code. Our aim was to create a solution that worked by simple loading our Touchless script.

Creating the pieces of the puzzle

Data syncing with Socket.io

For our prototype, we picked socket.io for data syncing between the interactive and the user’s phone. The remote interface would be displayed as a simple in browser webpage.

Creating the server code with socket.io was surprisingly easy due to their great documentation. We managed to have two devices communicating with each other in no time.

Creating an accurate remote control

The remote resembles a joystick, therefore we created an input solution that is based on velocity and direction. The further the user moves the joystick out, the faster the cursor will move. This approach ensures that the user has fine grained control but is also able to move greater distances on the screen without taking too long. Getting the balance right proved to be fairly time-consuming and also dependent on the interactive itself. Having editable debug controls greatly helped to get that balance right.

Connecting to an interactive screen

The cursor script would be available by adding a simple script tag to the interactive page. Once loaded, it would offer the user a QR code that would allow them to load the remote control on their mobile phone. The client can keep track of all the devices connected and can show instructions to users once their mobile phone is connected.

Once a user has successfully connected to an interactive, a remote cursor will appear on top of the interactive display to give the user visual feedback of where the touch events would usually take place.

Adding a multiplayer option

If Touchless is considered when designing an interactive, it can be so much more than just the joystick! A quick proof of concept showed us the potential of using Touchless with multiple people at the same time. Imagine a content aware joystick, where the device can display different content based on the cursor position on the interactive. The opportunities are endless and we are looking forward to exploring Touchless further.

Our learnings from Touchless

Touchless was similar to a lot of project timelines — 80% of the work was done in 20% of the time, and the last stretch to get our prototype to 100% presented its own challenges.

Our main challenge was to differentiate between HTML objects and Canvas/WebGL objects. Our prototype project, Quake Nation, used a WebGL library called PixiJS to display a vast amount of graphic instances. 

For Touchless, we differentiated between HTML objects and Canvas/WebGL objects by mapping the event types from our remote control based on the element under the cursor location. This made it possible for our solution to work with both an external library and native HTML events without modifying the original code of the interactive.

Request a Touchless demo

Do you have interactive displays you’d like to transform? We’d love to chat. Request a demo of Touchless via our Springload contact page.


TAGS