Siro
Sensorial Interface for remote mining operations
Introduction
This project focuses on the professional-user,
system- and use-context of remote mining machines: more specifically on how different senses which are experienced during a maned excavation operation could be translated in sensorial sensations simulated in the teleoperation excavation experience.
Approach
We decided to focus on one excavator &
operator at a time. The main focus is on sensorial experiences and how they can enrich the teleoperated experience. Furthermore, we were exploring sustainable ways of automatization.
Solution
Siro enables the operator to remotely control an excavator in different mines around the globe. Instead of a joystick, the operator uses a haptic positioning device combined with a table-top representation of the rock-pit.
At the core of the system are wearables which substitute different sensorial feedbacks that a remote operator would otherwise miss. The sensations are ranging from sound, haptics and even smell. The operator is able to teach the system a specific strategy of excavating in order to automate tasks if a mental rest is needed. Meanwhile, the wearables are keeping the operator in the loop about what is happing during the automated process by the different sensorial wearables on the body.
Details
University: Umea Institute of Design
Type: Speculative Design
Year: 2019
Duration: 10 weeks
Team: Inna Zrajaeva, Anna Puchalska
Partner: Komatsu Japan
Design question
"How might we add value for workers in semi-autonomous teleoperated mining systems by adding multi-sensorial feedback?"
Key features
tele-excavating materials
The excavator can be kilometres away from the workstation:
the feedback of the excavation is realized by several wearables and input-devices. It consists of haptic feedback, like vibrations, pressure and motions as well as temperature and smell.
smelling different materials in the pit
Different materials are represented through different smells.
sensing the environment by sound
When trucks are moving around in the mine they create different sounds depending on the area. If something is happening like falling rocks, it can be heard subtle or intense depending on how far away from the excavator it is happening.
feeling the machine through haptics
For example, when the excavator moves the acceleration can be felt by getting rhythmic vibrations on the back. Or wearables that are getting slightly warmer when the truck is getting closer to the excavator.
programing the excavation process
Some excation interventions are hard to determine for an AI. That is why the operator
programming excavation process
activating auto-mode
analyse the penetration depth, the excavation trajectory & the material characteristics
stay in the loop while leaving the workstation through wearables on the body
The interface
The main interface consists of a tabletop representation of the mine-pit. In which the topography, the different kinds of material and the positioning of the shovel/bucket as well the truck that is picking up the rocks can be seen. TThe front screen represents the first-person view as well as side-view of the excavation process.
The system consists of:
haptic & olfactory wearables
screen
table-top display
haptic device
finger pedals
Sensorial Wearables
At the core of the system are wearables that expand the senses of the operator. The ability to actually feel more and sense more is highly valuable especially when the operator is far away from the work location.
vibro-tactile
The user can feel the swinging of the excavator represented in the rhythmic vibration pattern.
vibration + pressure
The design of the back of the wearable system consists of vibrators along two lines along the back, translating the movement of the excavator (forward, backward, left, right). The tilting of the machine is represented through pressure feedback elements.
pseudo haptic weight
One arm is representing the state of the bucket while the other arm is representing the truck. When the actual bucket is filling up with materials the operator feels the weight on their arm via faux weight sensations.
feeling the machine through haptics
For example, when the excavator moves the acceleration can be felt by getting rhythmic vibrations on the back. Or wearables that are getting slightly warmer when the truck is getting closer to the excavator.
temperature
The proximity of the bucket to the material or truck is represented in tactile vibrations on the hand. The position of the truck is shared with the user by temperature feedback. When the truck is close to the excavator the wearable on the hand is getting warmer.
Discovered opportunities
(Semi-)automated systems often don't facilitate a sustainable human-machine collaboration. Instead, the human operator gets degraded to an over-seer role where he or she as to act on problems that the
AI-driven system cannot solve (black swan situation).
Teleoperation experiences lack production-efficiency due to an impoverished representation of sensorial experiences. This makes it also incredibly hard to concentrate on the operation and numerous brakes become a necessity for the operator.
Research trip to Gällivare & Kiruna
In our field trip that took place from the 4th to the 4th of October, we visited two different mines. The first one being the Aitik copper mine in Gällivare, a small city in the northern part of Sweden, the second one the iron ore mine in Kiruna, a small city north of the polar circle.
Human-centred ethnographic research
Our task was to do ethnographic research on workers and try to get the most information by talking to them, video and audio documenting and getting to know more about their work and life from their own unique perspective. We dived into the complexity of structures, responsibilities and tasks of miners, operators, truck drivers, supervisors and dispatchers.
“I work like a chess player - everyday is exciting.”
“Best part of my job? Lovely people!”
Stina, coordinator
Stina, a former machine operator and know coordinator showed us around and explained the social structures between the employer and the organization
Peter, machine operator
Peter agreed to a "follow-me-around" through his excavator where he explained to us the main functionalities and pain points
Expert interviews - possible futures & visualizing material
We’ve decided to conduct interviews with power experts researching on cognitive aspects of human-computer-interaction and human-in-the-loop computing, AI, HCI. We have visited specialists from Oryx Simulators and Komatsu Forest. Material gathered during the research was processed in various ways: task analysis, user journeys, stakeholder’s map, trends map, recordings of interviews
Mind castle for mapping findings from the interviews
Task analysis with multiple senses involved in the operation
Stakeholder’s map to discover complexity of relationships.
Quick weekly ideation sessions to keep us moving forward.
Testing & prototyping
The prototyping phase started right after the milestone presentation with Komatsu on the 28th of October. After making decisions for our future concept-pathway we have started to build first physical prototypes that we tested with the wizard-of-oz technique.
Sketches of concepts that were used for creating a future journey map.
Form-giving of the haptic device, wearables, tabletop
Testing the set up (device, screen, tabletop).
Lo fi prototyping of the feedback sensors.
Filming the concept video