Siro 

Sensorial Interface for remote mining operations

Introduction
 

This project focuses on the professional-user,
system- and use-context of remote mining machines: more specifically on how different senses which are experienced during a maned excavation operation could be translated in sensorial sensations simulated in the teleoperation excavation experience.

Approach

We decided to focus on one excavator &
operator at a time. The main focus is on sensorial experiences and how they can enrich the teleoperated experience. Furthermore, we were exploring sustainable ways of automatization. 

Solution

​

Siro enables the operator to remotely control an excavator in different mines around the globe. Instead of a joystick, the operator uses a haptic positioning device combined with a table-top representation of the rock-pit.

At the core of the system are wearables which substitute different sensorial feedbacks that a remote operator would otherwise miss. The sensations are ranging from sound, haptics and even smell. The operator is able to teach the system a specific strategy of excavating in order to automate tasks if a mental rest is needed. Meanwhile, the wearables are keeping the operator in the loop about what is happing during the automated process by the different sensorial wearables on the body.

Details 

University: Umea Institute of Design

​

Type: Speculative Design

​

Year: 2019

​

Duration: 10 weeks

​

Team: Inna Zrajaeva, Anna Puchalska

​

Partner: Komatsu Japan

Key features

tele-excavating materials

The excavator can be kilometres away from the workstation:
the feedback of the excavation is realized by several wearables and input-devices. It consists of haptic feedback, like vibrations, pressure and motions as well as temperature and smell.

smelling different materials in the pit 

Different materials are represented through different smells.

sensing the environment by sound 

When trucks are moving around in the mine they create different sounds depending on the area. If something is happening like falling rocks, it can be heard subtle or intense depending on how far away from the excavator it is happening.

feeling the machine through haptics

For example, when the excavator moves the acceleration can be felt by getting rhythmic vibrations on the back. Or wearables that are getting slightly warmer when the truck is getting closer to the excavator.

programing the excavation process

Some excation interventions are hard to determine for an AI. That is why the operator 

programming excavation process 

activating auto-mode

analyse the penetration depth, the excavation trajectory & the material characteristics

stay in the loop  while leaving the workstation through wearables on the body

The interface

The main interface consists of a tabletop representation of the mine-pit. In which the topography, the different kinds of material and the positioning of the shovel/bucket as well the truck that is picking up the rocks can be seen. TThe front screen represents the first-person view as well as side-view of the excavation process. 

Sensorial Wearables

At the core of the system are wearables that expand the senses of the operator. The ability to actually feel more and sense more is highly valuable especially when the operator is far away from the work location.

vibro-tactile

The user can feel the swinging of the excavator represented in the rhythmic vibration pattern.

vibration + pressure

The design of the back of the wearable system consists of vibrators along two lines along the back, translating the movement of the excavator (forward, backward, left, right). The tilting of the machine is represented through pressure feedback elements.

pseudo haptic weight

One arm is representing the state of the bucket while the other arm is representing the truck. When the actual bucket is filling up with materials the operator feels the weight on their arm via faux weight sensations.

feeling the machine through haptics

For example, when the excavator moves the acceleration can be felt by getting rhythmic vibrations on the back. Or wearables that are getting slightly warmer when the truck is getting closer to the excavator.

temperature

The proximity of the bucket to the material or truck is represented in tactile vibrations on the hand. The position of the truck is shared with the user by temperature feedback. When the truck is close to the excavator the wearable on the hand is getting warmer.

Research trip to Gällivare & Kiruna 

In our field trip that took place from the 4th to the 4th of October, we visited two different mines. The first one being the Aitik copper mine in Gällivare, a small city in the northern part of Sweden, the second one the iron ore mine in Kiruna, a small city north of the polar circle.

Human-centred ethnographic research

Our task was to do ethnographic research on workers and try to get the most information by talking to them, video and audio documenting and getting to know more about their work and life from their own unique perspective. We dived into the complexity of structures, responsibilities and tasks of miners, operators, truck drivers, supervisors and dispatchers.

Stina, coordinator

​

Stina, a former machine operator and know coordinator showed us around and explained the social structures between the employer and the organization 

Peter, machine operator

Peter agreed to a "follow-me-around" through his excavator where he explained to us the main functionalities and pain points

Expert interviews - possible futures & visualizing material

We’ve decided to conduct interviews with power experts researching on cognitive aspects of human-computer-interaction and human-in-the-loop computing, AI, HCI. We have visited specialists from Oryx Simulators and Komatsu Forest. Material gathered during the research was processed in various ways: task analysis, user journeys, stakeholder’s map, trends map, recordings of interviews

Testing & prototyping

The prototyping phase started right after the milestone presentation with Komatsu on the 28th of October. After making decisions for our future concept-pathway we have started to build first physical prototypes that we tested with the wizard-of-oz technique.

Sketches of concepts that were used for creating a future journey map.