Skip to content
eoPortal

Other Space Activities

Telerobotics

Last updated:Apr 21, 2022

Human Spaceflight

Telerobotics: Ground-based Rover's Touch Shared with Astronaut in Space

References

 

April 2022: If man's best friend is a dog, then in the future astronauts' closest companions might well be rovers. A technique allowing astronauts in orbit to control rovers exploring planetary surfaces has been developed by a research team from ESA, the German Aerospace Center DLR and European academia and industry, culminating in an Earth-based rover session commanded from the International Space Station. A paper published in the prestigious Science Robotics journal this week details their results. 1) 2)

Figure 1: Analog-1 rover. In November 2019, ESA astronaut Luca Parmitano made robotics history, reaching out from the International Space Station in orbit around Earth at 8 km/s, to control an Earth-based rover, equipped with an advanced gripper possessing the equivalent mobility and dexterity of a human hand. This gripper was able to pick up and collect rock samples from the mock-lunar environment. In the future a comparable system may well be used to explore alien environments, with astronauts controlling surface rovers from the safety and comfort of a surface habitat or an orbiting spacecraft. - The Analog-1 test project, which concluded with this two-hour space-to-ground test on 25 November, had multiple technical goals. High among them was to assess the use of ‘force-feedback' controls – like a high-end gaming joystick pushing back on their user, giving them a sense of touch – in space, to evaluate if this technology would enable high-precision robotic control in weightless conditions. - While selecting rocks Luca received advice from a team of geological experts based at the European Astronaut Centre simulating a real-life surface exploration survey. - The experiment benefitted from Luca's previous training through ESA's Pangaea programme, giving astronauts practical experience in geology to aid efficient discussion between the crew and the scientists (image credit: ESA–A. Koehler)
Figure 1: Analog-1 rover (Image credit: ESA–A. Koehler)

In November 2019, ESA astronaut Luca Parmitano made robotics history, reaching out from the International Space Station in orbit around Earth at 8 km/s, to control an Earth-based rover, equipped with an advanced gripper possessing the equivalent mobility and dexterity of a human hand. This gripper was able to pick up and collect rock samples from the mock-lunar environment. In the future a comparable system may well be used to explore alien environments, with astronauts controlling surface rovers from the safety and comfort of a surface habitat or an orbiting spacecraft. - The Analog-1 test project, which concluded with this two-hour space-to-ground test on 25 November, had multiple technical goals. High among them was to assess the use of ‘force-feedback' controls – like a high-end gaming joystick pushing back on their user, giving them a sense of touch – in space, to evaluate if this technology would enable high-precision robotic control in weightless conditions. - While selecting rocks Luca received advice from a team of geological experts based at the European Astronaut Centre simulating a real-life surface exploration survey. - The experiment benefitted from Luca's previous training through ESA's Pangaea programme, giving astronauts practical experience in geology to aid efficient discussion between the crew and the scientists.

"This is the first time that an astronaut in space managed to control a robotic system on the ground in such an immersive, intuitive manner," comments Aaron Pereira of DLR.

"Our 6 degree of freedom control interface incorporates force feedback so that the astronaut can experience just what the rover feels, even down to the weight and cohesion of the rocks it touches. What this does is help compensate for any limitations of bandwidth, poor lighting or signal delay to give a real sense of immersion – meaning the astronaut feels as though they are there at the scene."

Figure 2: Astronaut Luca Parmitano operated a rover with a 6 degrees of freedom arm from the International Space Station during the Analog-1 test campaign in November 2019 (image credit: ESA-NASA)
Figure 2: Astronaut Luca Parmitano operated a rover with a 6 degrees of freedom arm from the International Space Station during the Analog-1 test campaign in November 2019 (image credit: ESA-NASA)

Let the Robots Take the Strain

Robotics engineer Thomas Krueger, heading ESA's Human Robot Interaction Laboratory, adds: "Robots can be given limited autonomy in known, structured environments, but for systems carrying out exploratory tasks such as sample collection in unknown, unstructured environments some kind of ‘human-in-the-loop' oversight becomes essential. But direct control has not been feasible due to the inherent problem of signal delay – with transmission times constrained by the speed of light.

"So we have been working towards the concept of humans staying safely and comfortably in orbit around the Moon, Mars or other planetary bodies, but being close enough for direct oversight of rovers on the surface – combining the human strengths of flexibility and improvisation with a robust, dexterous robot on the spot to carry out their commands precisely."

A team from ESA's HRI Lab and DLR's Robotics and Mechatronics Center collaborated on a series of progressively more complex tests, first on Earth then extending into orbit.

"In the end we needed to perform feasibility experiments from space because past research shows that weightlessness can degrade human performance during force and motion tasks," adds Thomas. "This and other unique environmental factors meant that Earth-based simulations would not be sufficient."

Figure 3: Sample collection gripper on the Analog-1 Interact rover. ESA astronaut Luca Parmitano took command of a rover in the Netherlands on 18 November 2019 and expertly drove it over an obstacle course to a sampling site and collected a rock – all while circling our planet at 28,800 km/h in the International Space Station. - This test was the first in a series to prove the technology ESA has developed to operate rovers from afar. Called Analog-1 the test could hardly have gone any better. Given one hour of precious astronaut time, Luca ticked all the boxes for the exam in less than half an hour. -Robots can be fitted for special tasks and go places where no humans can go, but nothing beats our quick and adaptive thinking and the human touch. The Analog-1 rover is equipped with force feedback so astronauts can feel what the robot feels and adjust grip accordingly on a joystick that allows for six degrees of motion. - ESA's exploration strategy foresees astronauts controlling robots from orbit around the Moon or Mars or from inside a planetary base.- A week later all elements of the overarching Meteron project will be put to the test. Luca will drive the robot to three sites in the hangar in the Netherlands and decide in collaboration with a science team based at the European Astronaut Centre in Cologne, Germany, which rocks to pick up and keep for later analysis. - This experiment is as authentic as possible using the International Space Station as a stand-in for a lunar gateway and the hangar made to resemble a lunar landscape. Whereas the first "proficiency run" was used to test the systems and Luca had to follow a determined path, next week he will be more free to explore to meet the objectives set up by the science team at the European Astronaut Centre. - The same science software designed for guiding the ExoMars rover mission on the Red Planet will be used that allows the science team to indicate sites of interest as well as overlay dangerous areas that are beyond the limits of the rover's capabilities. - The Analog-1 experiment is proving the value of human-robotic cooperation in space and demonstrating the technology that will be used as the basis for many of ESA's exploration projects (image credit: ESA)
Figure 3: Sample collection gripper on the Analog-1 Interact rover. ESA astronaut Luca Parmitano took command of a rover in the Netherlands on 18 November 2019 and expertly drove it over an obstacle course to a sampling site and collected a rock – all while circling our planet at 28,800 km/h in the International Space Station (Image credit: ESA)

This test was the first in a series to prove the technology ESA has developed to operate rovers from afar. Called Analog-1 the test could hardly have gone any better. Given one hour of precious astronaut time, Luca ticked all the boxes for the exam in less than half an hour. -Robots can be fitted for special tasks and go places where no humans can go, but nothing beats our quick and adaptive thinking and the human touch. The Analog-1 rover is equipped with force feedback so astronauts can feel what the robot feels and adjust grip accordingly on a joystick that allows for six degrees of motion. - ESA's exploration strategy foresees astronauts controlling robots from orbit around the Moon or Mars or from inside a planetary base.- A week later all elements of the overarching Meteron project will be put to the test. Luca will drive the robot to three sites in the hangar in the Netherlands and decide in collaboration with a science team based at the European Astronaut Centre in Cologne, Germany, which rocks to pick up and keep for later analysis. - This experiment is as authentic as possible using the International Space Station as a stand-in for a lunar gateway and the hangar made to resemble a lunar landscape. Whereas the first "proficiency run" was used to test the systems and Luca had to follow a determined path, next week he will be more free to explore to meet the objectives set up by the science team at the European Astronaut Centre. - The same science software designed for guiding the ExoMars rover mission on the Red Planet will be used that allows the science team to indicate sites of interest as well as overlay dangerous areas that are beyond the limits of the rover's capabilities. - The Analog-1 experiment is proving the value of human-robotic cooperation in space and demonstrating the technology that will be used as the basis for many of ESA's exploration projects.

Their efforts culminated in the first part of the Analog-1 experiment in late 2019. Astronaut Luca Parmitano aboard the ISS operated the gripper-equipped ESA Interact rover in a mock lunar environment inside a hangar in Valkenburg, the Netherlands to survey rocks and collect samples. The two-hour space-to-ground test was a success, overcoming a two-way signal delay averaging more than 0.8 seconds and a data packet loss rate of 1% plus.

Figure 4: Analog-1 infographics. In a series of experiments, collectively called Analog-1, the International Space Station will be used as a stand-in for a spacecraft in orbit around another planetary body, such as the lunar Gateway. From here the next wave of exploration of our Solar System will see humans exploring ‘hand-in-hand' with robots. Robotic scouts guided from lunar orbit can safely investigate uncharted areas and prepare the land for human explorers to set up camp. - The same approach could also be used to explore Mars and other planets, transplanting human intelligence and agility to alien surfaces without the expense and potential hazard of landing. - The Interact rover pictured here will be commanded by ESA astronaut Luca Parmitano in space as part of ESA's Multi-Purpose End-To-End Robotic Operation Network, Meteron, project to develop the technology for human and robotic exploration. - Using a specially-developed ‘space internet' the signal from the Space Station will travel up to 10,000 km one way to the ground, but Luca will still be able to see and even feel what the robot does despite a considerable time delay. - If all goes well and Luca passes his driving test, he will do a full simulation to put Meteron to the test: can Luca command the robot to do a geological survey of interesting areas in the simulated lunar landscape? (image credit: ESA)
Figure 4: Analog-1 infographics. In a series of experiments, collectively called Analog-1, the International Space Station will be used as a stand-in for a spacecraft in orbit around another planetary body, such as the lunar Gateway. From here the next wave of exploration of our Solar System will see humans exploring ‘hand-in-hand' with robots. (Image credit: ESA)

Robotic scouts guided from lunar orbit can safely investigate uncharted areas and prepare the land for human explorers to set up camp. - The same approach could also be used to explore Mars and other planets, transplanting human intelligence and agility to alien surfaces without the expense and potential hazard of landing. - The Interact rover pictured here will be commanded by ESA astronaut Luca Parmitano in space as part of ESA's Multi-Purpose End-To-End Robotic Operation Network, Meteron, project to develop the technology for human and robotic exploration. - Using a specially-developed ‘space internet' the signal from the Space Station will travel up to 10,000 km one way to the ground, but Luca will still be able to see and even feel what the robot does despite a considerable time delay. - If all goes well and Luca passes his driving test, he will do a full simulation to put Meteron to the test: can Luca command the robot to do a geological survey of interesting areas in the simulated lunar landscape?

Solving the Time Factor

"Even though the ISS is in orbit just 400 km overhead, its signals are relayed to Earth via geostationary telecom satellites, then to Europe from Texas via a transatlantic cable," explains Aaron.

Figure 5: Analog-1 experiment hangar. ESA astronaut Luca Parmitano made robotics history, reaching out from the International Space Station in orbit around Earth at 8 km/s, to control an Earth-based rover, equipped with an advanced gripper possessing the equivalent mobility and dexterity of a human hand. - This gripper was able to pick up and collect rock samples from the mock-lunar environment. In the future a comparable system may well be used to explore alien environments, with astronauts controlling surface rovers from the safety and comfort of a surface habitat or an orbiting spacecraft. - The Analog-1 test project, which concluded with this two-hour space-to-ground test on 25 November, had multiple technical goals. High among them was to assess the use of ‘force-feedback' controls – like a high-end gaming joystick pushing back on their user, giving them a sense of touch – in space, to evaluate if this technology would enable high-precision robotic control in weightless conditions. - While selecting rocks Luca received advice from a team of geological experts based at the European Astronaut Centre simulating a real-life surface exploration survey. - The experiment benefitted from Luca's previous training through ESA's Pangaea programme, giving astronauts practical experience in geology to aid efficient discussion between the crew and the scientists (image credit: ESA- G. Porter)
Figure 5: Analog-1 experiment hangar. ESA astronaut Luca Parmitano made robotics history, reaching out from the International Space Station in orbit around Earth at 8 km/s, to control an Earth-based rover, equipped with an advanced gripper possessing the equivalent mobility and dexterity of a human hand. - This gripper was able to pick up and collect rock samples from the mock-lunar environment. (Image credit: ESA- G. Porter)

In the future a comparable system may well be used to explore alien environments, with astronauts controlling surface rovers from the safety and comfort of a surface habitat or an orbiting spacecraft. - The Analog-1 test project, which concluded with this two-hour space-to-ground test on 25 November, had multiple technical goals. High among them was to assess the use of ‘force-feedback' controls – like a high-end gaming joystick pushing back on their user, giving them a sense of touch – in space, to evaluate if this technology would enable high-precision robotic control in weightless conditions. - While selecting rocks Luca received advice from a team of geological experts based at the European Astronaut Centre simulating a real-life surface exploration survey. - The experiment benefitted from Luca's previous training through ESA's Pangaea programme, giving astronauts practical experience in geology to aid efficient discussion between the crew and the scientists.

"Our team at DLR had to design a control algorithm that could function on a stable basis despite this time lag. Because there is a delay in the force feedback received by the operator, they might continue to move the robot further even after it has hit a rock. This could lead to the robot going out of sync with its controller, potentially vibrating like crazy, perhaps even damaging itself.

"To prevent that happening, we use a concept called ‘passivity' – we look at the overall amount of energy an operator puts in, and on the remote side make sure the robot never gives out any more energy than that, and vice-versa. Like when you push a child on a swing, they are never going to go any higher than the first swing – and with friction and so on will swing gradually lower and lower.

"So for instance, when the robot arm is moving and suddenly hits a rock it would take extra energy to move which the astronaut did not command, so we reduce the command energy at once to slow down the arm. Then, after the 850 microsecond delay, when the astronaut feels the rock they can then choose to add the extra energy to push it.

"This ‘Time Domain Passivity Approach for High Delay' technique is very intuitive in practice and should be able to work well with higher delay times as well."

Figure 6: Rock sampling from space – Analog-1. Analog-1 has been the latest in a series of progressively more challenging human-robot test campaigns involving the ISS, collectively called Meteron (Multi-purpose End-to-End Robotic Operation Network). The first 1 degree of freedom force-feedback test took place back in 2015 with ESA's Haptics-1 experiment, progressing to DLR's 2 degrees of freedom Kontur-2 the following year – advancing now to a full 6 degrees of freedom movement (video credit: ESA)

Next Steps on Mount Etna

Thomas concludes: "The main limitation of the work done so far is that our indoor analog lunar environment lacks realism. So this summer will see the second part of Analog-1 take place on the volcanic slopes of Mount Etna in Italy as part of a larger international robotic test campaign called ARCHES.

"Luca Parmitano will once again be controlling the Interact rover, this time in 1G from the ground."

Figure 7: Mount Etna erupting. ESA astronaut Luca Parmitano captured this image of Mount Etna erupting from the International Space Station. Etna is an active stratovolcano on the east coast of Sicily, Italy. Luca was launched to the International Space Station for his second mission, Beyond, on 20 July 2019. He will spend six months living and working on the orbital outpost where he will support more than 50 European experiments and more than 200 international experiments in space (image credit: ESA/NASA-L. Parmitano)
Figure 7: Mount Etna erupting. ESA astronaut Luca Parmitano captured this image of Mount Etna erupting from the International Space Station. Etna is an active stratovolcano on the east coast of Sicily, Italy. Luca was launched to the International Space Station for his second mission, Beyond, on 20 July 2019. He will spend six months living and working on the orbital outpost where he will support more than 50 European experiments and more than 200 international experiments in space (image credit: ESA/NASA-L. Parmitano)

 


References

1) "Ground-based rover's touch shared with astronaut in space," ESA Enabling & Support, 21 April 2022, URL: https://www.esa.int/Enabling_Support/Space_Engineering_Technology/Ground-based_rover_s_touch_shared_with_astronaut_in_space

2) Michael Panzirsch ,Aaron Pereira,Harsimran Singh, Bernhard Weber, Edmundo Ferreira, Andrei Gherghescu,Lukas Hann,Emiel den Exter,Frank van der Hulst, Levin Gerdes, Leonardo Cencetti, Kjetil Wormnes, Jessica Grenouilleau, William Carey, Ribin Balachandran, Thomas Hulin,Christian Ott, Daniel Leidner, Alin Albu-Schäffer, Neal Y. Lii, and Thomas Krüger "Exploring planet geology through force-feedback telemanipulation from orbit," Science Robotics, Volume 7, Issue 65, Published: 20 April 2022, https://www.science.org/doi/10.1126/scirobotics.abl6307
 

References    Back to top