Siro - A multi-sensorial interface for tele-operative mining interventions
Brief
Design a professional-user interface in the system- and use-context of remote mining machines.
Approach
Enriching the sensorial experience during the remote mining operation and exploring sustainable and human-centred ways of automating systems.
Impact
Siro enables the operator to remotely control an excavator in different mines around the globe. At
the core of the system are wearables which substitute different sensorial sensations from the actual location.
Details
10-week group project with Anna Puchalska and Inna Zrajaeva at the Umeå Institute of Design in partnership with Komatsu Japan
Type
Proffesional-user Product
My Role
Ethnographic Research /
UX/UI design / Filmmaking / Storytelling / Formgiving
Seam Concept-video, 2020
Sensing the mine
This project deal with the professional-user and system-context of remote teleoperated mining interventions.
Our focus is on making the different sensations that can be sensed during the real excavation process on the actual physical site available as embodied experiences during the remote interventions while reducing the user's cognitive and physical load. In the next steps, we were exploring strategies for sustainable ways of human-machine-collaboration.
“I use a lot of senses during the operation, sometimes I even use my nose to smell if there is no grease leaking from the machine” - Peter, Machine-operator
We went on a research-trip where we visited two different mines - the Aitik copper mine in Gällivare and the iron-ore mine in Kiruna
Key Features
Siro enables the operator to remotely control an excavator in different mines around the globe. Instead of a joystick, the operator uses a haptic positioning device combined with a table-top representation of the rock-pit.
At the core of the system are wearables which substitute different sensorial feedbacks that a remote operator would otherwise miss. The sensations are ranging from sound, haptics and even smell. The operator is able to teach the system a specific strategy of excavating in order to automate tasks if a mental rest is needed.
Meanwhile, the wearables are keeping the operator in the loop about what is happing during the automated process by the different sensorial wearables on the body. The operator is able to teach the system a specific strategy of excavating in order to automate the tasks. Meanwhile, the wearables are informing the operator about what is happening during
the automated process.
Siro is bringing embodied feedback like in the form of sound, haptics and even smell into the user-experience of semi-autonomous teleoperated mining systems
Tele-excavating materials
The excavator can be kilometres away from the workstation: the feedback of the excavation is realized by several wearables and input-devices.
Smelling different materials in the pit
Different materials are represented through different smells.
Sensing the mine put with sound
If something is happening like falling rocks or approaching trucks, it can be sensed by subtle or intense sounds depending on how far away from the excavator it is happening.
Feeling the machine through haptics
When the excavator moves the acceleration can be felt by getting rhythmic vibrations on the back. Or wearables that are getting slightly warmer when the truck is getting closer to the excavator.
Programing the excavation process
Some things like the penetration depth or excavation trajectory are hard to determine for an AI. An operator can teach the system and put it on automated mode if he or she needs a break.
The operator can teach the system a strategy of excavating in order to automate the tasks - meanwhile, the wearables are informing the operator about what is happening during the automated process
Programming excavation process
Analysing the penetration depth, the excavation trajectory & the material characteristics
Operating on auto-mode
Stay in the loop while not being at the workstation through wearables on the body
The Interface
The main interface consists of a tabletop representation of the mine-pit.
In which the topography, the different kinds of material and the positioning of the shovel/bucket as well the truck that is picking up the rocks can be seen. The front screen represents the first-person view as well as side-view of the excavation process.
Screen
Table-top display
Haptic device
Finger pedals
At the core of the system are wearables that expand the senses of the operator - sensing more is highly valuable, especially when they are far away from the site
Haptic & olfactory wearables
Vibro-tacticle - The user can feel the swinging of the excavator
The user can feel the swing of the excavator represented a rhythmic vibration pattern
Vibration + pressure - translating the movement of the excavator
The design of the back of the wearable system consists of vibrators along two lines
Olfactory systems - different materials are represented in different smell
Olfactory systems enable to detect a presence or concentration of particular chemicals
Feeling the machine and mine pit through pseudo haptic weight and temperature one the hands and arms
Pseudo haptic weight
One arm is representing the state of the bucket while the other is representing the truck. When the actual bucket is filling up with materials the operator feels the weight on their arm
Temperature
The proximity of the bucket to the material or truck is represented in tactile vibrations on the hand. The position of the truck is shared with the user by temperature feedback
Force feedback
The rock penetration depth and obsticals in the excavation trajectory can be felt as force feedback of the haptic device
The Process
In our field trip that took place from the 4th to the 5th of October, we visited two different mines. The first one being the Aitik copper mine in Gällivare, a small city in the northern part of Sweden, the second one
the iron ore mine in Kiruna, a small city north of the polar circle.
Research trip to the Aitik copper mine in Gällivare
The Timelime of our process
Challenges that shaped the process
In automated systems, the human operator often gets degraded to an over-seer role where he or she as to act on problems that the-driven system cannot solve (black swan situation). That also leads to a deterioration of skills over a longer period of time.
Due to the lack of embodiment of sensorial experiences the operator relies on abstract simulation to in order to make decisions. This makes remote interventions inefficient and straining. This means that an operator can only work in 20 minutes shifts.
Ethnographic research
We dippedinto the complexity of structures, responsibilities and tasks of miners, operators, truck drivers, supervisors and dispatchers. The data was gathered by using qualatative methods like interviews and observations.
Stina, coordinator
Stina, a former machine operator and know coordinator showed us around and explained the social structures between the employer and the organization
Peter, machine operator
Peter agreed to a "follow-me-around" through his excavator where he explained to us the main functionalities and pain points
Sense-making, Sketching and Ideating
We’ve decided to conduct interviews with power experts researching on cognitive aspects of human-computer-interaction and human-in-the-loop computing, AI, HCI.
We have visited specialists from Oryx Simulators and Komatsu Forest. Material gathered during the research was processed in various ways: task analysis, user journeys, stakeholder’s map, recordings of interviews.
Task analysis with a map of different senses involved in the mining operation
Stakeholder’s map to discover the complexity of relationships
Quick weekly ideation sessions to keep us moving forward
Prototyping, Testing and Making
The prototyping phase started right after the milestone presentation with Komatsu on the 28th of October. After making decisions for our future concept-pathway we have started to build first physical prototypes that we tested with the wizard-of-oz technique.
Testing the set up (device, screen, tabletop)
Form-giving of the haptic device, wearables, tabletop
Lo-fidelity prototyping and mapping of the feedback sensors
