Siro -  A multi-sensorial interface for tele-operative mining interventions

Brief

Design a professional-user interface in the system- and use-context of remote mining machines.

Approach

Enriching the sensorial experience during the remote mining operation and exploring sustainable and human-centred ways of automating systems. 

Impact

Siro enables the operator to remotely control an excavator in different mines around the globe. At
the core of the system are wearables which substitute different sensorial sensations from the actual location. 

Details

10-week group project with Anna Puchalska and Inna Zrajaeva at the Umeå Institute of Design in partnership with Komatsu Japan

Type

Proffesional-user Product

My Role

Ethnographic Research / 
UX/UI design / Filmmaking / Storytelling / Formgiving

Seam Concept-video, 2020

Sensing the mine

This project deal with the professional-user and system-context of remote teleoperated mining interventions. 

Our focus is on making the different sensations that can be sensed during the real excavation process on the actual physical site available as embodied experiences during the remote interventions while reducing the user's cognitive and physical load. In the next steps, we were exploring strategies for sustainable ways of human-machine-collaboration. 

“I use a lot of senses during the operation, sometimes I even use my nose to smell if there is no grease leaking from the machine” - Peter, Machine-operator

We went on a research-trip where we visited two different mines - the Aitik copper mine in Gällivare and the iron-ore mine in Kiruna

Key Features

Siro enables the operator to remotely control an excavator in different mines around the globe. Instead of a joystick, the operator uses a haptic positioning device combined with a table-top representation of the rock-pit.

 

At the core of the system are wearables which substitute different sensorial feedbacks that a remote operator would otherwise miss. The sensations are ranging from sound, haptics and even smell. The operator is able to teach the system a specific strategy of excavating in order to automate tasks if a mental rest is needed.

 

Meanwhile, the wearables are keeping the operator in the loop about what is happing during the automated process by the different sensorial wearables on the body. The operator is able to teach the system a specific strategy of excavating in order to automate the tasks. Meanwhile, the wearables are informing the operator about what is happening during

the automated process.

Siro is bringing embodied feedback like in the form of sound, haptics and even smell into the user-experience of semi-autonomous teleoperated mining systems

Tele-excavating materials

The excavator can be kilometres away from the workstation: the feedback of the excavation is realized by several wearables and input-devices.

Smelling different materials in the pit 

Different materials are represented through different smells.

Sensing the mine put with sound 

 If something is happening like falling rocks or approaching trucks, it can be sensed by subtle or intense sounds depending on how far away from the excavator it is happening.

Feeling the machine through haptics

When the excavator moves the acceleration can be felt by getting rhythmic vibrations on the back. Or wearables that are getting slightly warmer when the truck is getting closer to the excavator.

Programing the excavation process

Some things like the penetration depth or excavation trajectory are hard to determine for an AI. An operator can teach the system and put it on automated mode if he or she needs a break.

The operator can teach the system a strategy of excavating in order to automate the tasks - meanwhile, the wearables are informing the operator about what is happening during the automated process

Programming excavation process 

Analysing the penetration depth, the excavation trajectory & the material characteristics

Operating on auto-mode

Stay in the loop while not being at the workstation through wearables on the body

The Interface

The main interface consists of a tabletop representation of the mine-pit.

In which the topography, the different kinds of material and the positioning of the shovel/bucket as well the truck that is picking up the rocks can be seen. The front screen represents the first-person view as well as side-view of the excavation process. 

Screen

Table-top display

Haptic device

Finger pedals

At the core of the system are wearables that expand the senses of the operator - sensing more is highly valuable, especially when they are far away from the site

Haptic & olfactory wearables

Vibro-tacticle - The user can feel the swinging of the excavator 

The user can feel the swing of the excavator represented a  rhythmic vibration pattern

Vibration + pressure - translating the movement of the excavator 

The design of the back of the wearable system consists of vibrators along two lines

Olfactory systems - different materials are represented in different smell 

Olfactory systems enable to detect a presence or concentration of particular chemicals

Feeling the machine and mine pit through pseudo haptic weight and temperature one the hands and arms

Pseudo haptic weight

One arm is representing the state of the bucket while the other is representing the truck. When the actual bucket is filling up with materials the operator feels the weight on their arm

Temperature

The proximity of the bucket to the material or truck is represented in tactile vibrations on the hand. The position of the truck is shared with the user by temperature feedback

Force feedback

The rock penetration depth and obsticals in the excavation trajectory can be felt as force feedback of the haptic device

The Process

In our field trip that took place from the 4th to the 5th of October, we visited two different mines. The first one being the Aitik copper mine in Gällivare, a small city in the northern part of Sweden, the second one

the iron ore mine in Kiruna, a small city north of the polar circle.

Research trip to the Aitik copper mine in Gällivare

The Timelime of our process

Challenges that shaped the process

In automated systems, the human operator often gets degraded to an over-seer role where he or she as to act on problems that the-driven system cannot solve (black swan situation). That also leads to a deterioration of skills over a longer period of time.

Due to the lack of embodiment of sensorial experiences the operator relies on abstract simulation to in order to make decisions. This makes remote interventions inefficient and straining. This means that an operator can only work in 20 minutes shifts. 

Ethnographic research

We dippedinto the complexity of structures, responsibilities and tasks of miners, operators, truck drivers, supervisors and dispatchers. The data was gathered by using qualatative methods like interviews and observations.

Stina, coordinator

Stina, a former machine operator and know coordinator showed us around and explained the social structures between the employer and the organization 

Peter, machine operator

Peter agreed to a "follow-me-around" through his excavator where he explained to us the main functionalities and pain points

Sense-making, Sketching and Ideating

We’ve decided to conduct interviews with power experts researching on cognitive aspects of human-computer-interaction and human-in-the-loop computing, AI, HCI.

 

We have visited specialists from Oryx Simulators and Komatsu Forest. Material gathered during the research was processed in various ways: task analysis, user journeys, stakeholder’s map, recordings of interviews.

Task analysis with a map of different senses involved in the mining operation

Stakeholder’s map to discover the complexity of relationships

Quick weekly ideation sessions to keep us moving forward

Prototyping, Testing and Making

The prototyping phase started right after the milestone presentation with Komatsu on the 28th of October. After making decisions for our future concept-pathway we have started to build first physical prototypes that we tested with the wizard-of-oz technique.

Testing the set up (device, screen, tabletop)

Form-giving of the haptic device, wearables, tabletop

Lo-fidelity prototyping and mapping of the feedback sensors

Interested in learning more?

All works © Oliver Weglinski 2020