ISS: CIMON (Crew Interactive Mobile Companion)
ISS Utilization: CIMON (Crew Interactive Mobile Companion)
CIMON is a mobile and autonomous assistance system designed to aid astronauts with their everyday tasks on the ISS (International Space Station). This will be the first form of AI (Artificial Intelligence) on an ISS mission. Airbus, in cooperation with IBM, developed CIMON, an AI-based assistant for astronauts for the DLR Space Administration. CIMON will be tested on the ISS by astronaut Alexander Gerst during the Horizons mission of ESA (European Space Agency) between June and October 2018. 1)
CIMON is able to see, hear, understand, speak – and fly. It is roughly spherical, has a diameter of 32 cm with a mass of 5 kg. Its robotic predecessor was Professor Simon Wright's 'flying brain', with sensors, cameras and a speech processor in the 1978 cartoon series, 'Captain Future'. Almost 40 years later, CIMON, the astronaut's flight attendant and assistance system, could turn science fiction into 'science fact'. 2)
From summer 2018 onwards, CIMON will become the new 'crew member' on the ISS, to demonstrate cooperation between humans and intelligent machines in the form of a technology experiment. The interactive astronaut assistant was developed and built by Airbus in Friedrichshafen and Bremen on behalf of DLR (German Aerospace Center) and funded by the German Federal Ministry for Economic Affairs and Energy (Bundesministerium für Wirtschaft und Energie; BMWi). Watson AI (Artificial Intelligence) technology from the IBM Cloud provides voice-controlled artificial intelligence. The human aspects of the assistance system were co-developed and supervised by scientists at LMU (Ludwig-Maximilian University) Hospital in Munich (LMU Klinikum ). An approximately 50-strong DLR, Airbus, IBM and LMU project team has been working on implementing CIMON since August 2016. "CIMON is globally unique in this form," summarizes Christian Karrasch, CIMON Project Manager at the DLR Space Administration in Bonn. "We have implemented this experiment in a very short time. It is intended to show to what extent the astronauts' work can be supported in the European Columbus module on the ISS and relieve them, in particular, of routine tasks. Ideally, the astronauts could use their time better and more effectively. With CIMON, we are entering new territory and operating at the threshold of technological feasibility."
Microgravity in Parabolic Flight
CIMON will enter microgravity on 9 March 2018 during the 31st DLR parabolic flight campaign in Bordeaux. In particular, orientation, navigation and steering will be tested, to ensure that it is optimally prepared for use on the ISS – in permanent microgravity. In June, CIMON will then travel to the ISS on board the US Space-X CRS-15 space transport mission, where it will be greeted by the German ESA astronaut Alexander Gerst.
Following a functional test, the German astronaut will perform three experiments with his artificial colleague. On the agenda are experiments with crystals and a Rubik’s cube, and a medical experiment in which CIMON will be used as a flying camera.
CIMON allows the astronaut to keep both hands free, with no need to manually operate a computer, for example. Thanks to this fully voice-controlled access to documents and media, the astronaut can conveniently navigate through operating and repair instructions and procedures for experiments and equipment. CIMON will thus serve as a complex database of all the necessary information for working on the ISS, and can also be used as a mobile camera for documentation purposes.
When making its space debut, however, the intelligent artificial assistant will not have all the capabilities envisaged by its developers: "In the introduction period, we want to concentrate on group effects that develop in small teams over a long period of time and can occur during long-term missions to the Moon and Mars. Social interaction between humans and machines, and between astronauts and emotionally intelligent flight attendants could play an important role in the success of these missions," explains Till Eisenberg, CIMON project lead at Airbus Friedrichshafen. The engineers are also interested in processing Big Data and data mining.
"The aim of the CIMON project is to examine the current capabilities of artificial intelligence in a complex environment such as the International Space Station, in order to provide the best possible support to people in such conditions," says Matthias Biniok, IBM's lead Watson architect in Germany, explaining their interest in the project. For example, CIMON uses Watson AI to process text, speech and images, find specific information and knowledge, and interpret moods and feelings. Biniok adds: "These skills can be individually trained and enhanced within the context of their respective application. Artificial intelligence also makes particular use of artificial neural networks." 3)
IBM Watson services run on the IBM Cloud, which provides a further advantage for users, in general, and for use on the ISS in particular: sensitive, proprietary data can remain where it is created, such as in the protected area of your own server or database. You don’t need to upload it to an external cloud for it to be enriched with appropriate AI capabilities.
CIMON also has a scientific background; its advisors are Judith-Irina Buchheim and Alexander Choukèr from the Department of Anesthesiology at the Ludwig-Maximilian University Hospital in Munich. During a variety of research projects, Buchheim and Choukèr are investigating the effects of stress on the human immune system. "We not only examine patients in intensive care units, but also people exposed to extreme stress and workload as a result of their environment, such as polar explorers in the Antarctic and astronauts on the International Space Station," reports Buchheim, adding: "Our studies show that being subjected to microgravity for a certain period can significantly affect the functioning of an astronaut's immune system. Stress is a major factor here." For example, strenuous tasks that are performed with a colleague are generally less arduous when people work well together. "As a partner and assistant, CIMON could support astronauts with their high workload of experiments and maintenance and repair work, thereby reducing their exposure to stress." According to Buchheim, possible applications on Earth are the support of engineers, researchers and doctors, AI-based enquiries about medical symptoms and everyday assistance for elderly people living alone.
CIMON the Assistant
CIMON's structure was built entirely using a 3D printing process, and is made of metal and plastic. Its 'face' is a display unit intended to hover at the astronaut's eye level. It can present and explain information and instructions for scientific experiments and repairs. Its 'eyes' are two cameras, and it also has an additional camera for face recognition. Two side cameras are used for video documentation and could also be used for additional computer-generated functions (augmented reality). Ultrasonic sensors measure distances for collision detection. Seven microphones act as 'ears' for detecting where sounds are originating, and there is a directional microphone for good voice recognition. Its 'mouth' is a loudspeaker that can be used to talk or play music. The core speech comprehension element of the AI is the IBM Watson system.
CIMON is unable to learn independently; a human must actively train it. The AI for autonomous navigation is provided by Airbus and used for motion planning and object recognition. CIMON can freely move and rotate in all spatial directions using 14 internal fans. It can therefore turn towards the astronaut when it is spoken to, nod and shake its head and follow the astronaut – autonomously or on command. In microgravity on the ISS, it can be used for two hours. The dimensions of CIMON's face are modelled on the proportions of a human face. Gestures and facial expressions are also possible, as is a female, male or neutral appearance and voice. CIMON's product design was implemented in collaboration with Reichert Design, a company from Stetten at the Bodensee (Lake Constance).
The Watson AI was trained using voice samples and photos of Alexander Gerst, and procedures and plans of the Columbus module of the International Space Station were loaded into the database. Alexander Gerst also had a say in the selection of CIMON’s screen face and computer voice so that he, too, could ‘make friends’ with his electronic colleague.
Once the functional testing of the system has been completed, Gerst will work in Space with CIMON a total of three times: They will experiment with crystals, work together to solve the Rubik’s cube and perform a complex medical experiment using CIMON as an ‘intelligent’ flying camera.
In its first Space mission, CIMON will only be equipped with a selected range of capabilities. In the medium term, aerospace researchers also plan to use the CIMON project to examine group effects that can develop over a long period of time in small teams and that may arise during long-term missions to the Moon or Mars. Social interaction between people and machines, between astronauts and assistance systems equipped with emotional intelligence, could play an important role in the success of long-term missions. Airbus’ developers are convinced that, here on Earth, developments of the assistance system could also find future use in hospitals and social care.
The SpaceX CRS-15 (Commercial Resupply Services-15) Dragon logistics mission was launched on 29 June 2018 on a Falcon-9 vehicle. The launch site was Cape Canaveral SCL-40. Dragon carried more than 2672 kg of supplies and payloads, including critical materials to directly support more than 250 science and research investigations that will occur onboard the orbiting laboratory. 4) 5) 6)
Orbit: Near-circular orbit of the ISS, altitude of ~400 km, inclination = 51.6º, period of ~ 93 minutes.
1) ECOSTRESS of NASA/JPL. measures the temperature of plants and uses that information to better understand how much water plants need and how they respond to stress.
2) Mobile Companion, an ESA (European Space Agency) investigation, also known as CIMON (Crew Interactive Mobile Companion), explores the use of AI as a way to mitigate crew stress and workload during long-term spaceflight.
3) Rodent Research-7 takes a look at how the microgravity environment of space affects the community of microoganisms in the gastrointestinal tract, or microbiota.
4) Angiex Cancer Therapy examines whether microgravity-cultured endothelial cells represent a valid in vitro model to test effects of vascular-targeted agents on normal blood vessels.
5) Chemical Gardens are structures that grow during the interaction of metal salt solutions with silicates, carbonates or other selected anions. Their growth characteristics and attractive final shapes form from a complex interplay between reaction-diffusion processes and self-organization.
These investigations join hundreds of others currently happening aboard the orbiting laboratory.
• Project Biarri Squad 1-3 technology demonstration: The Biarri project is a four nation defence related project involving Australia, the US, the UK and Canada to build three 3U CubeSats for precision formation flying experiments and a risk mitigation satellite.
• Bhutan-1, a 1U CubeSat: Bhutan, along with Malaysia and the Philippines, is currently participating in the second joint global multi-nations BIRDS Project called BIRDS-2, initiated by KIT (Kyutech Institute of Technology), Japan, which is represented by four engineers from the telecom and space division of the information and communications ministry (MoIC).
• MAYA-1, the first Filipino 1U CubeSat implemented by the PHL-Microsat Program of the University of the Philippines Diliman, in collaboration with the DOST-ASTI (Department of Science and Technology - Advanced Science and Technology Institute) and the KIT (Kyushu Institute of Technology), Japan. The development of the Maya-1 falls under BIRDS-2 (Birds Satellite Project; note: the BIRDS-2 CubeSats are Bhutan-1, MAYA-1 and UiTMSAT-1), a cross-border interdisciplinary satellite project that accommodates non-space faring countries.
• UiTMSAT-1, a 1U CubeSat of Malaysia. JAXA arranged for the testing and the launch of the BIRDS-2 CubeSats to the ISS.
• August 28, 2019: CIMON, the mobile astronaut assistant, equipped with artificial intelligence (AI), returned to Earth on 27 August 2019. The SpaceX CRS-18 Dragon spacecraft carrying CIMON was undocked from the International Space Station (ISS) at 16:59 CEST; the capsule splashed down in the Pacific Ocean approximately 480 km southwest of Los Angeles and was recovered at 22:21 CEST. 8)
- "We expect CIMON to return to Germany at the end of October," reports Christian Karrasch, CIMON Project Manager at the German Aerospace Center (Deutsches Zentrum für Luft- und Raumfahrt; DLR). He looks back on the past few months: "CIMON is a technology demonstration that has completely met our expectations. During its initial operation in space – a 90-minute mission with the German ESA astronaut Alexander Gerst on the ISS in November 2018 – it showed that it functions well in microgravity conditions and can interact successfully with astronauts. We are very proud to have been the first to use AI on the Space Station and have been working for several months on an improved successor model. With CIMON, we were able to lay the foundations for human assistance systems in space to support astronauts in their tasks and perhaps, in the future, to take over some of their work."
- The new CIMON – like its predecessor – will be built by Airbus in Friedrichshafen and Bremen on behalf of the DLR with funds from the German Federal Ministry of Economic Affairs and Energy (BMWi). Airbus in Friedrichshafen has assembled and tested the new CIMON hardware. Airbus in Bremen is working on improving the software for flight and attitude control, while IBM is implementing new functions for the AI. Till Eisenberg, CIMON Project Manager at Airbus says: "Overall, there are several upgrades such as better microphones, a more robust computer, improved flight and attitude control, and new software features for conversation, such as speech recognition, call history and intent analysis."
- Matthias Biniok, Project Manager at IBM, adds: “With CIMON we have a unique application case in an extreme working environment. We have seen that we can use AI – in our case IBM Watson – to support the work of astronauts. The further development of CIMON will primarily address improved contextual language comprehension and linguistic emotional analysis.”
- Ethical questions concerning the future use of CIMON will be examined and evaluated by physicians at LMU (Ludwig-Maximilian University) in Munich. Personal rights are impacted through the interaction between human and machine, because images and audio of the astronaut are recorded, processed and interpreted by CIMON. High technical standards in the field of data security are necessary, and trust in such a system is important when working in a team consisting of humans and machines. Specifically, what is CIMON allowed to do, know and say? "The new CIMON has a built-in switch that enables the data streams from all cameras and microphones to be interrupted from the ISS. The astronaut has control over CIMON at all times, which was especially important for us," stresses LMU researcher Judith Buchheim.
- The DLR Space Administration, which manages Germany's contributions to the European Space Agency (ESA), is working with ESA to deliver the new CIMON to the ISS in December 2019 and to obtain crew time with astronauts.
- CIMON arrived on the ISS as a technology demonstration on 2 July 2018. On 15 November 2018, the robotic assistant with the smart 'face' undertook its globally acclaimed mission. For 90 minutes, it 'worked' successfully with Alexander Gerst. CIMON proved its basic functionality – its flight characteristics in microgravity – with autonomous navigation through several rotations and movements in all directions. It also searched for and recognized Gerst's face, made eye contact with him, and spoke to him. Additionally, CIMON displayed the instructions for an experiment on its screen, played music, and acquired video and images of the astronaut.
- In 2018, the CIMON project won the US 'Popular Science Award' in the category 'Best of What's New in 2018' for space. Airbus was also awarded the 'German Innovation Award 2019' in the category 'Large Enterprises'.
• March 31, 2019: CIMON, the technology demonstrator developed and built by Airbus on behalf of the German Aerospace Center (Deutsches Zentrum für Luft- und Raumfahrt; DLR), has won the prestigious German Innovation Prize (Deutsche Innovationspreis) in the large companies category 2019. The award was presented in Munich on Friday evening. 9)
- Every year, Accenture, EnBW and the business magazine WirtschaftsWoche, which joined forces to launch the German Innovation Award, recognize outstanding, forward-looking innovations from German companies, which are transforming business and markets with their innovative strength.
- CIMON (Crew Interactive MObile companioN), is the first free-flying astronaut assistant to be tested on board the International Space Station (ISS), and is a technology experiment designed to demonstrate how humans and intelligent machines can work together.
- CIMON can see, hear, understand, speak and fly. It is spherical with a diameter of 32 cm and weighs 5 kg. It is modelled after a character from the 1980s anime series ‘Captain Future’, Professor Simon Wright, which was known as ‘the flying brain’ and featured sensors, cameras and a speech processor.
- On 15 November 2018, CIMON successfully completed its first 90-minute space mission with German ESA astronaut Alexander Gerst.
- CIMON can support astronauts, for example by displaying and explaining procedures or instructions for scientific experiments and repairs. In particular, CIMON could be used in the future to complete routine tasks, such as documenting experiments, searching for items and taking inventory. It can also be used as a mobile camera. CIMON technology could also be useful here on Earth – assisting in industrial processes, in operating theatres or in the social sector.
- The interactive astronaut assistant was developed and built by Airbus in Friedrichshafen and Bremen on behalf of the German Aerospace Center (DLR) Space Administration and funded by the German Federal Ministry for Economic Affairs and Energy (Bundesministerium für Wirtschaft und Energie; BMWi). Watson AI technology from the IBM Cloud provides voice-controlled artificial intelligence. The scientific aspects of the assistance system were co-developed and supervised by a team at Ludwig-Maximilian University Hospital in Munich (Klinikum der Ludwig-Maximilians-Universität München, LMU).
- Currently, the CIMON team is analyzing the results of its first space mission, programming improvements and new tasks, and preparing the technology demonstrator for new work missions with new – human – colleagues.
• On 15 November 2018 at 11:40 CET (Central European Time), the mission team in the BIOTESC (Biotechnology Space Support Center) at Lucerne University of Applied Sciences and Arts (Switzerland) watched with baited breath. After two-and-a-half years of highly intensive preparations, as well as countless testing and training sessions with CIMON (Crew Interactive Mobile CompanioN) on Earth, you could hear a pin drop – there was an atmosphere of total concentration and thrilled anticipation. After a software upload to the International Space Station, a software update for CIMON himself, an audio check and a test of the navigation camera, Alexander Gerst took a good look at his new robotic housemate and put him straight into operation. The world premiere lasted 90 minutes – the first 'rendezvous' between the German ESA astronaut and his autonomous mobile robot assistant. 10)
- Once Alexander Gerst had taken his manmade helper out of its box in the Columbus module of the ISS, the German astronaut woke him up with the words "Wake up, CIMON!" The answer came promptly: "What can I do for you?” After this initial banter, Gerst allowed CIMON to float around freely – initially by remote control from Earth. The guidance, navigation and control system was thus activated.
- Then came some autonomous navigation with multiple turns and movements in all directions. Once complete, CIMON was able to locate Gerst's face and make eye contact. To demonstrate its capabilities as an assistant, CIMON used its 'face' – a display at the center of the sphere – to show the instructions for a student experiment on crystallization and also played a song. It then recorded a video and photo of Alexander Gerst using its integrated cameras. Afterwards, Alexander Gerst brought CIMON back to its place in the Columbus module. "The voice communication worked perfectly and I am very relieved that the cooperation between CIMON and Alex ran so smoothly," says Gwendolyne Pascua, the BIOTESC project manager who spoke directly with Alexander Gerst during the commissioning phase to guide him through the experiment.
- "It is an incredible feeling and an absolute delight to witness how CIMON is seeing, hearing, understanding and speaking. For us, this first real deployment in space is part of history and is hopefully just the beginning of its usage on the ISS," says Dr Christian Karrasch, CIMON project manager from the DLR Space Administration. "Interaction with artificial intelligence fascinates me. As a system, CIMON is unparalleled elsewhere in the world and was designed specifically for deployment on the ISS. We are entering uncharted territory here and broadening technological horizons in Germany."
- "CIMON represents the embodiment of our vision," adds Till Eisenberg, project manager for CIMON at Airbus. "It is a huge step for human spaceflight and one that we are taking here as a team. In CIMON, we have laid the foundation for social assistance systems that can work even under extreme conditions."
- CIMON used the Wi-Fi on the International Space Station for data transmission and established an Internet connection to the IBM Cloud via satellite link and ground stations. "When CIMON is asked a question or when it is addressed, the Watson AI first converts the audio signal into text that can be understood or interpreted by the AI," says Matthias Biniok, IBM project manager, describing the processes taking place in CIMON's 'brain'. "IBM Watson is thus able to grasp the underlying intention, as well as the context of the words. The result is a pinpoint response, which is then converted back into language and beamed up to the ISS. This process enables a natural, dynamic spoken dialog."
- Bernd Rattenbacher, team leader at the BIOTESC Biotechnology Space Support Center of the Lucerne University of Applied Sciences and Arts, says: "The data connection to the Earth runs via satellite to NASA and to the Columbus Control Center at the DLR site in Oberpfaffenhofen. The signal travels from there to us, the CIMON ground station at BIOTESC in Lucerne, the Swiss User Support and Operations Center, which is connected to the IBM Cloud in Frankfurt by Internet. The runtime for the signal alone via the satellites is 0.4 second in one direction. A large number of firewalls and VPN tunnels are enabled to ensure data security."
- CIMON also has a scientific background. The consultants are Judith-Irina Buchheim and Professor Alexander Choukèr from the Department of Anesthesiology at LMU (Ludwig Maximilian University) Munich. "As an AI partner and companion, CIMON could support astronauts in their high workload of experiments as well as maintenance and repair work, thus reducing their exposure to stress," Buchheim says.
• July 6, 2018: After 3 days of orbiting th Earth with ground teams in Hawthorne and Houston watching its every move, Dragon was captured by NASA astronaut Ricky Arnold. The Dragon spacecraft was attached to the Node 2 Harmony module on 2 July 2018. 11)
- CIMON will be the first AI-based assistant used on station. Its primary design is to help with routine tasks. However, thanks to its neural AI network and ability to learn, CIMON may very well offer solutions to onboard problems should they occur.
- In order to serve as a test case for the new technology, the first mission of CIMON will only utilize a selected range of capabilities. If all goes well, AI-based assistants like CIMON will be a critical part of long-duration human space flight missions in the future. Until then, only time will tell what CIMON says.
Developed and built in Germany, CIMON is a technology experiment to support astronauts and increase the efficiency of their work. CIMON is able to show and explain information, instructions for scientific experiments and repairs. The voice-controlled access to documents and media is an advantage, as the astronauts can keep both hands free. It can also be used as a mobile camera to save astronaut crew time. CIMON could perform routine tasks, in particular, such as the documentation of experiments, the search for objects and for taking inventory. CIMON is also able to see, hear, understand and speak. Its eyes are actually two cameras that it uses for facial recognition, as well as five other cameras for orientation and video documentation. Ultrasound sensors measure distances to recognize potential collisions. Its ears consist of eight microphones to identify directions, and an additional directional microphone to improve voice comprehension. Its mouth is a loudspeaker used to speak or to play music. At the heart of the AI for language understanding is the IBM Watson AI technology from the IBM Cloud. CIMON was not equipped with self-learning capabilities and requires active human instruction. The AI used for autonomous navigation was contributed by Airbus and is designed for movement planning and object recognition. Twelve internal fans allow CIMON to move and rotate freely in all directions. This means it can turn toward the astronaut when addressed. It can also nod or shake its head and follow through space either autonomously or on command.
1) ”Airbus is developing the CIMON astronaut assistance system for the DLR Space Administration,” Airbus Press Release, 26 February 2018, URL: http://www.airbus.com/newsroom
2) ”CIMON - the intelligent astronaut assistant,” DLR, 2 March 2018, URL: http://www.dlr.de/dlr/en/desktopdefault.aspx
3) Matthias Biniok, ”Watson Hops On Board Human Space Flight,” 26 Feb. 2018, URL: https://www.ibm.com/blogs/think/2018/02/watson-space/
4) ”Dragon Resupply Mission (CRS-15),” SpaceX News, 29 june 2018, URL: http://www.spacex.com/news/2018/06/29/dragon-resupply-mission-crs-15
6) US Commercial ELV launch manifest, 6 March 2018, URL: http://www.sworld.com.au/steven/space/uscom-man.txt
7) ”ECOSTRESS Among Science Payloads on Next Space Station Mission,” NASA/JPL, 12 June 2018, URL: https://www.jpl.nasa.gov/news/news.php?feature=7157&utm_source=iContact&
8) ”CIMON back on Earth after 14 months on the ISS,” DLR, 28 August 2019, URL: https://www.dlr.de/dlr/en/desktopdefault.aspx
9) ”Astronaut assistant CIMON* wins prestigious German Innovation Prize,” Airbus Press Release, 1 April 2019, URL: https://www.airbus.com/newsroom/press-releases/en/2019/04
10) ”World premiere – Rendezvous between CIMON and Alexander Gerst on the International Space Station,” DLR, 16 November 2018, URL: https://www.dlr.de/dlr/en/desktopdefault.aspx
11) Mark Garcia, ”Space to Ground: Meet CIMON: 07/06/2018,” NASA, 6 July 2018, URL: https://www.nasa.gov/mediacast/space-to-ground-meet-cimon-07062018
The information compiled and edited in this article was provided by Herbert J. Kramer from his documentation of: ”Observation of the Earth and Its Environment: Survey of Missions and Sensors” (Springer Verlag) as well as many other sources after the publication of the 4th edition in 2002. - Comments and corrections to this article are always welcome for further updates (firstname.lastname@example.org).