PhiSat-1 & -2 Nanosatellite Mission

PhiSat-1 & -2 Nanosatellite Mission

Overview

Φ-Sat-1 Mission

PhiSat-1 & -2 (Φ-Sat-1 & -2) are the first European satellite to demonstrate how onboard artificial intelligence can improve the efficiency of sending Earth observation data back to Earth. This revolutionary artificial intelligence technology will fly on one of the two CubeSats that make up the FSSCat (Federated Satellite System) mission – a Copernicus Masters winning idea. ¹⁾

As the overall 2017 Copernicus Masters winner, FSSCat, was proposed by Spain’s UPC (Universitat Politècnica de Catalunya) and developed by a consortium of European companies and institutes.

FSSCat is an innovative mission concept consisting of two federated 6U CubeSats in support of the Copernicus Land and Marine Environment services. They carry a dual microwave payload (a GNSS-Reflectometer and a L-band radiometer with interference detection/mitigation), and a multispectral optical payload to measure soil moisture, ice extent, and ice thickness, and to detect melting ponds over ice. It also includes a radio/optical inter-satellite link and an Iridium intersatellite link to test some of the techniques and technologies for upcoming satellite federations. FSSCat will be the precursor of a constellation of federated small satellites for Earth observation achieving high temporal resolution and moderate spatial resolution in a cost-effective manner. ²⁾

«The Federated Satellite System 6U tandem mission for sea ice and soil moisture monitoring captured the interest of the challenge experts immediately. Not only because the mission concept shows a high degree of well thought through technical novelties, but also because it will provide data that is complementary to the Sentinel fleet. This is especially true for the soil moisture monitoring component, which is not part of the current Sentinel portfolio. The FSSCat mission development is good to go and due to its disruptive approach, we are confident that it will be seen as a breakthrough in procuring future small missions at ESA.»

The two CubeSats will collect data, which will be made available through the Copernicus Land and Marine Environment services, using state-of-the-art dual microwave and hyperspectral optical instruments. They also carry a set of intersatellite communication technology experiments.

During Φ-week (9-13 September 2019 at ESA/ESRIN), ESA’s Director of Earth Observation Programs, Josef Aschbacher, said, “We see that there is huge interest in Φ-Sat and thanks to our partners, it is ready to be launched. We live in exciting times, the pace at which digital technology is developing coupled with the wealth of satellite information being delivered and, indeed, the growing demand for such data, means there are many opportunities to make a step change for the future of Earth observation. And, with Φ-Sat – Europe’s first artificial intelligence in space – we are going to do just this.”

The hyperspectral camera on one of the CubeSats will collect an enormous number of images of Earth, some of which will not be suitable for use because of cloud cover. To avoid downlinking these less than perfect images back to Earth, the Φ-Sat artificial intelligence chip will filter them out so that only usable data are returned.

Marco Esposito, from cosine Remote Sensing, the company that led the development of the artificial intelligence algorithm, explained, “While compact, the instrument – which covers the visible and near infrared with hyperspectral capability, enhanced with bands in the thermal infrared – is very powerful and will acquire terabytes of data that can be used to monitor vegetation changes and to assess water quality, for example.

“However, generating this amount of data actually poses a problem, as the data have to be handled efficiently so that they can reach the users in a timely manner. With Φ-Sat we have effectively given the instrument its own brain, which processes the data onboard to detect clouds in the images. This not only ensures better quality data, but makes the delivery much more efficient.”

ESA’s Massimiliano Pastena, noted, “Indeed, this will be the first satellite to demonstrate the use of artificial intelligence in orbit and we are very much looking forward to it being launched in the coming months.”

Φ-Sat-1 is the first scientific and technology initiative of the Earth Observation directorate of the European Space Agency (ESA) in the NewSpace field. ³⁾ Φ-Sat-1 is the result of an enhancement of the FSSCAT mission using AI (Artificial Intelligence) and improved imaging capabilities of the hyperspectral camera HyperScout-1 VNIR imager. cosine Remote Sensing is leading the development and execution of the demonstration, including the highly integrated spectral imaging in the VNIR and TIR, as well as the integration of state-of-art AI accelerators including the first AI algorithms for cloud screening. The HyperScout-2 spectral camera is the successor of HyperScout-1, a miniaturized reflective optical instrument equipped with enhanced processing capabilities, in orbit since February 2018 as part ESA GOMX-4B technology demonstration mission.

Φ-Sat-1 will demonstrate the capabilities of small instruments for scientific applications, and will validate in-orbit innovative technologies. ⁴⁾ The technology demonstration includes the miniaturization and highly integration of visible, near infrared and thermal spectral channels, as well as state-of-art processors and machine learning algorithms. The hardware co-registration of the VNIR and TIR channels will enable a variety of applications for terrain classification and change detection. Φ-Sat-1 will demonstrate an AI-based inference engine for cloud detection and will be launched as enhancement of the FSSCat mission, ⁵⁾ during the VEGA SSMS (Small Satellite Mission Service) Proof Of Concept flight.

Φ-Sat-2 Mission

The selected Φsat-2 concept will provide a combination of on-board processing capabilities (including AI) and a medium to high resolution Visible to Near Infra-Red (VIS/NIR) multispectral instrument able to acquire 8 bands (7 multispectral bands + one panchromatic band).
These resources will be made available to a series of dedicated applications (hereafter called Apps) that will run onboard the spacecraft that have been partially pre-selected during the initial phases of the mission design (4 out of 6 to be tested in the operative phase). The remaining 2 Apps will be selected via a global competition open to the AI and EO communities.

Φ-Sat-2 Mission Timeline

The Φsat-2 spacecraft is designed for a 14-month lifetime from launch with potential extension up to 2 more years. Figure 1 shows the nominal top-level timeline of the mission, and Table 1 provides a description of the activities undertaken during each phase.

Observation Capacity and Strategy

After the system has been successfully commissioned and confirmed to be operating nominally, the activities are predominantly driven by scheduling imaging opportunities. Upon image acquisition requests, the objective is to provide multispectral imagery to the AI Apps, images will be acquired once the spacecraft reaches the selected Region(s) Of Interest (ROI) both during App fine tuning phase but also App operation phase. Simulations show that the revisit period depends highly on the location. The selected orbit on average offers global access to any area of the world every 15 days and allows imaging of any area in quasi-nadir conditions, i.e., with an off-nadir (roll) angle lower than 10 degrees. With a 5 off-nadir (roll) angle the average revisit period increases to 23 days. The mission will be providing global coverage (excluding north poles).

Based on the results obtained from simulations, it is considered that the proposed approach will provide enough images to fine-tune compression, vessel detection and cloud detection applications within 2 months. In addition to the above AOIs, another fine-tuning phase simulation is foreseen for Sat2map focusing on specific local areas in Southeast Asia prone to extensive flood events.

 Spacecraft

The FSSCat mission will demonstrate for the first time worldwide a reliable optical intersatellite link (O-ISL) between two 6U CubeSats flying in LEO (Low Earth Orbit). We have developed a technology demonstration with a full duplex O-ISL terminal of 1.5U in size (15cm x 10cm x 10cm), 85mm optical aperture, suitable for LEO to LEO operations on coarse pointing CubeSats (pointing accuracy < 0.5º).

The terminal adopts a novel adaptive variable divergence laser mechanism with a hybrid payload/platform seek-and-track pointing algorithm, and an amplitude modulated signal to ensure reliable optical communications at 1 Gbit/s at a nominal intersatellite distance of up to 2,000 km in LEO. In addition of establishing an intersatellite link, the terminal provides the ability for simultaneous imaging through the optical aperture and the improvement of the attitude determination capability of the hosting satellite through the mixing of optical information in the overall platform attitude feedback control loop. ¹²⁾

Φsat-2

The spacecraft is a 6U CubeSat based on the standard Open Cosmos OpenSat 6U platform.  Figure (TBD) shows the Φsat-2 spacecraft in Deployed (A) and Stowed (B) configuration.

Main Subsystems

Beyond the mechanical and thermal subsystems, the platform is composed of the following main subsystems:

Electric Power Subsystem

The power subsystem is composed by:
-Solar Panels, that consists of a double deployable solar panel, a single deployable solar panel and six body mounted panels, with a total of 89 cells.
-Two Electrical Power System Assembly, which provides both the interface to the solar arrays, battery charging, and power distribution thanks to 3.3V and 5V power out lines, as well as batteries and battery cell management, with the nominal energy storage of ~46 Wh.

On-Board Data Handling

The On-Board Data Handling consists of a microprocessor that acts as a central data processing unit, acting as the central authority for system-wide data handling operations. This includes inter-subsystem communications, data storage and management of automated operations. The On-Board Data Handling also physically contains the GNSS receiver which is functionally part of the Attitude and Orbit Determination and Control Subsystem.

Communications

Communications subsystem is subdivided in:
-Telemetry, Tracking and Command Communication based on S-band transceivers. It acts as an interface between the space and ground segments, enabling telecommand and software update functionality from the ground, and telemetry transmission from the satellite.
-High Data Rate Communication, based on X-band transmitter and a patch antenna array, enabling high-speed payload data transmission from the satellite.

Attitude and Orbit Determination and Control Subsystem

The Attitude and Orbit Determination and Control Subsystem is a subsystem highly tailored to the mission requirements and configuration. Specifically, it is composed of:
A dedicated computer, that includes 3-Axis MEMS rate sensors,
A set of sensors, which are magnetometers, Sun and Earth horizon sensors and one star tracker,
 A set of actuators, which are primarily 4 jitter-stabilised reaction wheels and magnetorquers, required for detumbling, sun-spin mode and desaturation of the reaction wheels
A GNSS receiver and antenna. 19)

On-Board Application Environment

Nanosat MO Framework

The NanoSat MO Framework (NMF) is a software framework for CubeSats based on the Consultative Committee for Space Data Systems (CCSDS) Mission Operations services. It facilitates not only the monitoring and control of the nanosatellite software applications, but also the interaction with the nanosatellite platform. This is achieved by using the latest CCSDS standards for monitoring and control, and by exposing services for common peripherals that are available in nanosatellite platforms, such as, GPS, Camera, ADCS, and others. Furthermore, it is capable of managing the software on-board by exposing a set of services for software management.
In simple terms, the NanoSat MO Framework introduces the concept of apps in space that can be installed, and then simply started and stopped from ground. Apps can retrieve data from the nanosatellite platform through a set of well-defined platform services. Additionally, the framework includes CCSDS standardised services for monitoring and control of apps. An NMF App can be easily developed, distributed, and deployed on a spacecraft. Just like Android and iOS, ESA created a software framework for flight and ground software of CubeSats which allows flight software to become Apps that run in space.
The main objective of the NanoSat MO Framework is to facilitate the development of software for small satellites and to simplify its orchestration. For example, new software can be easily deployed in a satellite just by starting and stopping Apps.

The core functionalities of the framework are:
● Monitoring and control of AI applications
● Easy access to platform peripherals via services
● Simple on-board software management: Deploy and run AI applications
● Updating AI applications and AI models
● Deleting AI applications and AI models
In the Φsat-2 Mission, the NMF will provide the possibility to integrate models developed by the AI experts, wrapping the AI models and enabling them to:
● Deploy a model on the CogniSat AI processor
● Retrieve image from the camera
● Apply the model in prediction on the acquired image
The NanoSat MO Framework allows the AI App developers to make their App using well defined interfaces and APIs that allow the App to reach the on-board devices via a set of services. The framework by itself will also prevent conflict when different Apps will compete to access a resource (e.g., imager) which in any case will be scheduled via the Payload Orchestrator Center.
NanoSat MO Framework and AI applications will operate in a dedicated memory space allocated specifically for them, which will not intersect with the rest of the software running on the OBC. In case of NanoSat MO Framework and any of the AI apps malfunction, OBC flight software will not be affected.

Towards the Incubation Phase

During the Incubation phase, the provided AI models (in python) may be further checked by the Φsat-2 team including:
• Run on generic GPU
• Tested in targeted hardware Intel® Movidius™ Myriad™ 2 on Neural Compute Stick with OpenVINO (2020.3)
• Tested with FlatSat in real configuration.

Application Lifecycle

The main App activities during routine operations are:
● Define an Area of Interest (AOI). The corresponding image will be acquired depending upon duty cycle and various constraints.
● Depending on the readiness of the App (from low to high):
1. Dataset consolidation, on ground fine tuning (retraining)
Download raw Image and payload data to the ground segment: activity to be performed as often as necessary during the visibilities over the ground stations to allow the maximum use of on-board storage capacity and provide data to the users and developers.
On ground, fine tune the parameters of AI Apps to improve their performance (if required).
Upload newly developed AI applications and AI models and test performance. The AI Apps and AI models can be updated in orbit after fine tuning is done with real images on the ground.
2. App validation
Run App inference onboard and download Image, payload data, and App outputs.
On ground, validation of the AI apps.
3. App operation
Run App inference onboard, and download App outputs.
On ground, dissemination of the Apps outputs.
The NanoSat MO Framework will support software updates. When a new software update is created by the developers on ground, the new or updated libraries will be packaged into an NMF Package. The NMF Package containing the software updates will then be transferred to the spacecraft and installed.
For all the above steps, the selected winners of the OrbitalAI Challenge, a competition where participants submit their ideas for new apps and tools for the PhiSat-2 onboard processing, will be fully supported by the Φsat-2 team.

The participants will be in charge of:
• Providing general requirements for image acquisition such as AOIs
• Eventual fine tuning of the AI models.
• Scientific validation of the AI apps.

In-Orbit Test Bed: A Platform for Artificial Intelligence Experiments

Applying Artificial Intelligence processing algorithms on the data acquired by HyperScout-2 directly onboard and in real time may represent a big leap forward in inferring and delivering Earth Observation derived information at a very high temporal resolution and with a very short timeliness. Alternatively, an interesting use case is the screening of the data before being downloaded. This is the case for the first demonstration that will be performed in orbit as part of the Φ-Sat-1 mission, where AI is leveraged to select the data to be downloaded based on the cloud coverage of the acquired scene. The application has been selected by ESA as the presence of clouds is one of the fundamental problems of optical remote sensing and it becomes particularly relevant considering the large data volume generated by Earth Observation missions based on hyperspectral imaging.

However, it should be noted that the HyperScout-2 processing chain has been designed in order to accept third party AI based software that can be run on the either on the VPU, or CPU/GPU, for testing purposes before being employed in operational scenario. The processing chain is depicted in Figure 7.

The first step prepares the data and corrects it for known aberrations. After the raw data is corrected, the spectral cube can be computed in order to produce spectral bands. The second pre-processing step is mandatory if the VPU is used. In this case the spectral cube is prepared and loaded into the VPU, ready to be analyzed by the AI algorithm. The inference step is use-case based, and as discussed can be performed on the VPU if high time efficiency is required.

Launch

PhiSat-1 

The Φ-Sat-1 CubeSat mission (6U), as enhancement of the FSSCat mission, was launched as a passenger payload at 01:51 UTC, 03:51 CEST on 3 September, 2020 (22:51 local time on 2 September in Kourou).The launch vehicle was the VEGA SSMS (Small Spacecraft Mission Service) Proof Of Concept flight. ⁸⁾ Sun-synchronous orbit; target orbit for the 7 microsatellites: altitude of 515 km, inclination of 97.45º; target orbit for the 46 nanosatellites: altitude of 530 km, inclination = 97.51º. The nominal mission duration (from liftoff to separation of the 53 satellites) is: 1 hour, 44 minutes and 56 seconds.

PhiSat-2

Sun-synchronous orbit, altitude of 500 km, inclination of 97.404º and 11:00pm LTDN.

Mission Status

• October 2023: Phase II ‘Incubation & production release’ starts. During the second phase, the selected teams will work on their solution together with the Φsat-2 consortium led by Open Cosmos and the Φ-lab team. The solution will be tested and verified on conditions with Φsat-2 configuration and integrated within the NanoMO framework.
In the end, 2 teams will have their solutions launched into space.

• September 20, 2023: Deadline for Phase I of the ‘The open competition’. During Phase I, participants submitted their best solution for new apps and tools for PhiSat-2 onboard processing, in the form of a Jupyter Notebook together with a PowerPoint presentation, and a pitch video of their solution. 10 teams will be selected for the next stage. 20)

• May 12, 2022: Could we capitalise on the Earth-based digital revolution to make our satellites smarter? ¹³⁾

- Over the last decade, rapid advances in computing techniques, combined with exponential growth in the amount of data being collected, have led to a true revolution of AI feeding on big data on Earth.

- ESA launched the first artificially intelligent Earth observation satellite in September 2020. Named φ-sat (pronounced phi-sat), the satellite carried an AI-accelerator chip that automatically discards cloudy images and send only useful data down to Earth.

- Then, in April 2021 ESA Discovery invited business and academic communities to propose innovative ideas to fly onboard ESA's OPS-SAT mission, which is open to the public to experiment with. AI had already been used on OPS-SAT to rapidly process images of Earth in flight and to fully automate the mission; some of the new ideas also involve a variety of AI tools for novel applications.

- The latest step came in October 2021, when the Discovery element of ESA's Basic Activities – working together with F-lab and ESA's new commercialisation department – launched a call for ideas on how we can make space systems more powerful using novel computing techniques and AI. The most interesting ideas have now been selected for funding.

- "Following the announcement of the call during Φ-week by ESA's Director General, industry and academia could propose projects addressing innovative technologies and applications," explains Gabriele Meoni, an ESA Φ-lab research fellow focusing on AI. "We received a really interesting suite of ideas, addressing a variety of space domains, showing that there is big interest from European stakeholders on cognitive computing in space."

- Applying modern computing techniques – for example, processing data directly on board satellites – could transform space activities and the space economy in the coming decade. The ability to make satellites smarter offers significant benefits for future space missions and business models.

- “We intentionally made the scope of the campaign very broad to explore the full potential of this new computing paradigm,” continues Gabriele.

- Leopold Summerer, head of ESA's Advanced Concepts and Studies Office adds: "The call for ideas was driven by the vision of ESA Agenda 2025, which calls for Europe to step up its role in space, support commercialisation and help create new space markets."

- "We think that in-space cognitive computing offers substantial commercial potential for first movers, and had planned to fund up to ten teams to develop their ideas in more detail," continues Leopold. "However, European industry and academia proposed so many good ideas that we decided to increase the number of funded projects to 12. We expect these to help Europe take a step further towards fully benefitting from the still-emerging field of cognitive computing in space."

- "The selected ideas involve new technologies developed outside the space sector – including blockchain, edge computing, and neuromorphic computing (inspired by the workings of the human brain)," says Gabriele. "The ideas address amazing applications across the space domain including early detection of methane gas and natural disasters, autonomous rovers on the Moon, space surveillance and tracking – all because information could be processed more efficiently."

- The 12 projects are being funded by ESA's Discovery element; each will run for six months with a budget of 100,000 EUR. All are financed using the ESA Initial Support for Innovation (EISI) scheme, which focuses on knowledge and skill development rather than achieving specific results.

- "What is the potential of new computing techniques in space? What are the new applications, or new science to be carried out? What would their business value be? These studies will help ESA further explore how cognitive computing could reshape the future of space missions," concludes Gabriele.

• October 14, 2021: How can a digital replica of Earth help us understand our planet’s past, present and future? As part of the fourth edition of Φ-week taking place this week, a group of European scientists have put forward their ideas on the practical implementation of Digital Twins and the potential application areas for a Digital Twin Earth in the real world. ¹⁴⁾

- In the coming decades, population growth and human activities are expected to amplify the current pressures on critical resources such as fresh water and food, intensify the stress on land and marine ecosystems, as well as increase environmental pollution and its impacts on health and biodiversity.

- Responding to these challenges, ESA came together at the 2020 edition of Φ-week to discuss how Earth observation can contribute to the creation of a digital twin of Earth – a dynamic, digital replica of our planet which accurately mimics Earth’s behavior.

- Constantly fed with Earth observation data, combined with in situ measurements and artificial intelligence, Digital Twin Earth will help visualize and forecast natural and human activity on the planet. The model will be able to monitor the health of the planet, perform simulations of Earth’s interconnected system with human behavior, and help support European environmental policies (see Figure 17).

- In September 2020, ESA launched several Digital Twin Earth Precursor Activities to explore some of the main scientific and technical challenges in building a digital twin of Earth. These activities included: Forest, Hydrology, Antarctica, Food Systems, Ocean and Climate Hot Spots.

- Each activity addressed a different scientific, technical and operational challenge regarding Digital Twin Earth including the role of artificial intelligence and consistent data, stakeholder engagement scientific credibility and role of sectorial models and Information and Communication Technology (ITC) infrastructure.

- At this year’s Φ-week, experts from the community came forward with the results of the activities over the last year.

Digital Twin Antarctica

- Antarctica is a major reservoir of freshwater in the word, with a huge potential to contribute to sea level rise in the future. Current ice sheet models present major differences and deviations among models, as well as strong variability in unstable areas.

- Therefore, a digital twin of Antarctica is necessary. Noel Gourmelen, from the University of Edinburgh commented, “By harnessing satellite observations, numerical simulations, and Artificial Intelligence, we have built a twin of the Antarctic ice sheet system, its hydrology, surrounding ocean, atmosphere, and biosphere. We have used the Antarctic twin to track the whereabouts of melt water on and under the ice sheet, and to explore how fringing ice shelves melt under various hydrology scenarios.”

Digital Twin Food Systems

- The Food Systems digital twin simulates agricultural activities and interactions within ecosystems on a daily basis. Different models can be run separately for each simulation unit, depending on crop, water and irrigation management system.

- Chandra Taposea, from CGI IT UK Lt, said, “Digital Twin Earths and the scope we are trying to achieve is vital in helping us reach the next step in sense-making and decision-making, and be able to help both individual users and large-scale policy makers. Our Food Systems Digital Twin has managed to integrate models from different domains, looking at how extreme precipitation would affect global crop models, but not without its trials and tribulations.”

Digital Twin Hydrology

- Luca Brocca, from the National Research Council, Italy, explains what the Hydrology Digital Twin entails, “In the ESA Digital Twin Earth Hydrology project, we have developed a 4D reconstruction of dynamic hydrology at unprecedented resolution through the integration of Earth observation and an advanced modelling system. The DTE Hydrology Prototype has been used for water resources management and for identifying locations and times of risk for landslides and flooding in the Po River Basin, in northern Italy.”

Digital Twin Climate Impacts

- The Climate Impacts Digital Twin will enable decision makers, without expert technical knowledge, to generate and visualize, in real-time, decision-relevant information related to regionalised impacts of climate change.

- Robert Parker, from the University of Leicester, said, “Our Climate Impact Explorer Digital Twin, initially focused on African drought, utilizes an innovative combination of Earth observation, environmental modelling and Machine Learning to bring enhanced decision support capabilities directly to our stakeholders.

- “By emulating these complex models and deploying them as fast and simple cloud-based tools, our prototype helps democratize access to these expert systems, giving stakeholders the capability to explore potential climate-driven drought responses.”

Digital Twin Forest

- Matti Motus, Principal Scientist at VTT, explains how the Forest Digital Twin works: “This digital twin will be a specialized Digital Twin of Earth, providing a reconstruction of the forest system at levels of detail not possible with generic land surface models. Satellite-based Earth observation, especially the high-quality Copernicus Sentinel data, allows us to get unique and uniform information for all forests of the globe.

- “Translating this into understanding on forest structure and to drive models of forest functioning requires local measurements, which are far more scattered and heterogeneous. In the precursor project, we have learned how to overcome these obstacles and provide growth and carbon balance predictions for different forests in Europe. We know now that we have the basic tools and the computing power to build a fully functioning digital twin of forests. It has been a very exciting, yet demanding journey, especially considering that it was fully implemented during the Covid-related restrictions.”

• October 13, 2021: With a boom in the global market for Earth observation information and data products, participants at this year’s Φ-week conference have been digging deep into the ‘market perspective’. This important topic includes how to gain a better understanding of what governments, industry, the public and other users of Earth observation products and services need and expect to create value for society and the economy. ¹⁵⁾

- According to the latest report from the Geospatial Industry Outlook and Readiness Index, GeoBuiz, the Earth observation industry as a whole is estimated to have been worth almost US $58 billion in 2019, rising to almost US $76 billion in 2020. Moreover, it is thought that this steep rise merely marks the beginning of bigger things to come.

- In addition, with respect to Europe industry, the 2021 survey published by the European Association of Remote Sensing Companies, EARSC, indicates that in 2020 there was a 24% growth in the number of companies associated with the Earth observation industry, as well as a 24% increase in revenue and a 17% rise in employees, compared to 2019.

- Essentially, Earth observation satellite data made available through ESA and through Europe’s Copernicus program are free and open to all. While this traditional approach of providing Earth observation data works extremely well for the scientific community and established users of data-based services, and support business creation in Europe, the rapidly growing commercial market for Earth observation products opens a myriad of opportunities – for the Earth observation industry, and for the protection of our planet.

- During the session dedicated to the market perspective at Φ-week, Steve Bochinger from Euroconsult delivered an extremely positive view of the future Earth observation market with emerging new verticals, albeit not completely free from potential complications.

- ESA’s Geraldine Naja showed how ESA, with its newly established Directorate of Commercialization, Industry and Procurement, will take a new approach to supporting European industry, taking benefits from worldwide commercial and New Space opportunities.

- During his talk, Marc Nolla Harvey, Director of Strategic Initiatives at SAP, said simply, “Earth observation benefits companies by allowing them to make more profit, be more efficient and be more sustainable.”

- And, Raffaele Mauro, General Partner at Primo Space Fund, a venture investor organization focused on the space business, explained the why private investors such as venture capitalists increasingly see the New Space economy as a very interesting investment target, which explains the current boom in private investment.

- ESG (Environmental, Social and Governance) criteria are also increasingly important to companies, their investors and other stakeholders. With growing concern about companies’ ethical and sustainability practices, Earth observation has a critical role to play.

- While ESG certification picks up pace, the growth in Europe’s market for Earth observation is also closely linked the European Commissions’ Green Deal – which aims to transform the EU into a modern, resource-efficient and competitive economy, ensuring no net emissions of greenhouse gases by 2050. Businesses and industry in many sectors such as agriculture, energy, insurance, security, are clearly using Earth observation to help in their efforts to curb greenhouse emissions and adopt more sustainable and profitable practices.

- This growth in the Earth observation market comes at a time when there are an increasing number of commercial companies developing small satellites relatively quickly and selling the resulting data products directly to their customers alongside space agencies such as ESA, which tend to take the ‘slow road’ developing satellite missions, which either prove new observing techniques, respond to gaps in scientific knowledge or deliver systematic data over many years.

- Simonetta Cheli, ESA’s Head of the Earth Observation Strategy, Program & Coordination Office, said, “The need for new ways for ESA to support the commercial Earth observation industry to develop innovative satellite missions, products and services remains more important than ever, particularly with respect to growing market demands and to monitoring and managing the impacts of climate change. We are adopting the concept of New Space to explore different ways of bringing benefits to the commercial world, and ultimately help building a stronger and leading European EO economy.”

- Giuseppe Borghi, Head of ESA’s of Φ-lab Division, added, “Sessions at this year’s Φ-week are certainly sparking some interesting and lively discussions. We are hearing a lot about the needs of different sectors such as the agriculture, energy, re/insurance, security and private investors and how ESA can support the growth of a flourishing ecosystem. Commercialization is one of the priorities in ESA’s Agenda 2025 and we, as ESA, have to work together with the market, the Earth observation industry and private investors to ensure that everyone can benefit from the wealth of information we get from space.

- “We can do this in a number of ways. We are working hard to tap into disruptive technologies and business models, including quantum computing, machine learning and, in general, artificial intelligence, the internet of things and cloud computing in space. We are making transformative innovation funding available through challenges as announced by the ESA Director General during the Φ-week opening session. A key instrument is also our commercialization co-funding program InCubed, backed by support from our Member States and the partnerships we build with leading research centers, industry and private investors.”

- While Φ-week continues and ‘the market perspective’ becomes clearer, there are already other moves towards blending traditional Earth observation with business initiatives. For example, ESA has just signed a contract with ICEYE, a commercial provider of satellite radar imagery, to include their constellation of small satellites in the fleet of missions contributing to Europe’s Copernicus environmental monitoring program – a perfect example of New Space.

- This week has also seen the signing of a contract with the German Research Center for Artificial Intelligence to support a new development initiative focused on artificial intelligence for Earth Observation.

• October 11, 2021: Kicking off with a bold flourish, Φ-week 2021 promises to bring space even closer to the forefront of addressing society’s biggest challenges, namely issues associated with the climate crisis, while boosting the economy through transformative New Space, artificial intelligence, and quantum and cognitive computing. ¹⁶⁾

- ESA’s Φ-week is now in its fourth edition, and each year this forward-looking event is increasingly relevant to society, business and the economy.

- In his opening address, Josef Aschbacher, ESA’s Director General said, “Europe faces unprecedented societal, economic, and security challenges. Space has enormous untapped potential to play in tackling pressing current and future crises, while simultaneously providing new impulse for the European space sector. We need to develop smart, automated services and applications to protect lives of people and to increase the protection of assets and natural resources.”

- Dr Aschbacher went on to explain how his priorities include transforming ESA and boosting green and digital commercialization, as outlined in Agenda 2025.

- He also announced a brand-new challenge, making €1 million available for funding studies centered on artificial intelligence and cognitive computing in space.

- Launched in 2020, Φ-sat-1 – the first artificial intelligence to be carried on a European Earth observation mission – demonstrated the ability to run machine learning algorithms in space on an ultra-low power, off-the-shelf artificial intelligence accelerator chip.

- Φ-sat-1 enables the pre-filtering of Earth observation data so that only images with usable information are downlinked to the ground, thereby significantly reducing aggregated downlink costs.

- The next artificial intelligence mission, Φ-sat-2, is currently under development and will demonstrate new ways of collecting, distributing and analyzing Earth observation data.

- Dr Aschbacher continued, “Taking the potential of this new paradigm forward, ESA is planning a challenge to explore how the use of computing and artificial intelligence in space could help transform the way we develop space missions and applications. This will not be exclusive to the field of Earth observation, but also open to ideas linked to telecoms, exploration, robotics and more. We plan to select 10 activities to the value of €100 000 each.”

- The opening ceremony also saw talks from Toni Tolker-Nielsen, ESA Acting Director of Earth Observation Programs; Ekaterini Kavvada, Director of Innovation and Outreach at the European Commission’s DG DEFIS; Kris Peeters, Vice President of the European Investment Bank; Mark Boggett, co-founder and CEO of Seraphim Capital; and Rafal Modrzewski CEO and co-founder of ICEYE – all discussing their visions of New Space and the economy.

- As the speakers highlighted, this year’s Φ-week is set to focus on how the New Space economy is developing in Europe alongside fierce competition worldwide. New technology is key, and the week will dig deeper into the latest computing breakthroughs and trends such as quantum computing and cognitive computing.

- Toni Tolker-Nielsen said, “This week we are naturally building on the successes of the previous Φ-weeks, but importantly taking stock of the fast-pace of change all around us. The Earth observation market is booming worldwide, driven largely by demands linked to Europe’s Green Deal and by the New Space economy and opportunities with private investors.

- “ESA is in the perfect position to align with the commercial world, but more importantly help lead the way. We are already busy helping industry be more competitive though our co-funding InCubed program – which has boomed since the program’s inception. We will of course be taking this much further and exploring many new opportunities though out the week and beyond.”

• October 20, 2020: As ubiquitous as artificial intelligence has become in modern life — from boosting our understanding of the cosmos to surfacing entertaining videos on your phone — AI hasn’t yet found its way into orbit. ¹⁷⁾

- That is until Sept. 2, when an experimental satellite about the size of a cereal box was ejected from a rocket’s dispenser along with 45 other similarly small satellites. The satellite, named PhiSat-1, is now soaring at over 17,000 mph (27,500 km/h) in sun-synchronous orbit about 329 miles (530 km) overhead.

- PhiSat-1 contains a new hyperspectral-thermal camera and onboard AI processing thanks to an Intel® Movidius™ Myriad™ 2 Vision Processing Unit (VPU) — the same chip inside many smart cameras and even a $99 selfie drone here on Earth. PhiSat-1 is actually one of a pair of satellites on a mission to monitor polar ice and soil moisture, while also testing intersatellite communication systems in order to create a future network of federated satellites.

- The first problem the Myriad 2 is helping to solve? How to handle the large amount of data generated by high-fidelity cameras like the one on PhiSat-1. “The capability that sensors have to produce data increases by a factor of 100 every generation, while our capabilities to download data are increasing, but only by a factor of three, four, five per generation,” says Gianluca Furano, data systems and onboard computing lead at the European Space Agency, which led the collaborative effort behind PhiSat-1.

- At the same time, about two-thirds of our planet’s surface is covered in clouds at any given time. That means a whole lot of useless images of clouds are typically captured, saved, sent over precious down-link bandwidth to Earth, saved again, reviewed by a scientist (or an algorithm) on a computer hours or days later — only to be deleted.

- “And artificial intelligence at the edge came to rescue us, the cavalry in the Western movie,” says Furano. The idea the team rallied around was to use onboard processing to identify and discard cloudy images — thus saving about 30% of bandwidth.

- “Space is the ultimate edge,” says Aubrey Dunne, chief technology officer of Ubotica. The Irish startup built and tested PhiSat-1’s AI technology, working in close partnership with cosine, maker of the camera, in addition to the University of Pisa and Sinergise to develop the complete solution. “The Myriad was absolutely designed from the ground up to have an impressive compute capability but in a very low power envelope, and that really suits space applications.”

- The Myriad 2, however, was not intended for orbit. Spacecraft computers typically use very specialized “radiation-hardened” chips that can be “up to two decades behind state-of-the-art commercial technology,” explains Dunne. And AI has not been on the menu.

- Dunne and the Ubotica team performed “radiation characterization,” putting the Myriad chip through a series of tests to figure out how to handle any resulting errors or wear-and-tear.

- ESA “had never tested a chip of this complexity for radiation,” says Furano. “We were doubtful we could test it properly ... we had to write the handbook on how to perform a comprehensive test and characterization for this chip from scratch.”

- The first test, 36 straight hours of radiation-beam blasting at CERN in late 2018, “was a very high pressure situation,” Dunne says. But that test and two follow-ups “luckily turned out well for us.” The Myriad 2 passed in off-the-shelf form, no modifications needed.

- This low-power, high-performance computer vision chip was ready to venture beyond Earth’s atmosphere. But then came another challenge.

- Typically, AI algorithms are built, or “trained,” using large quantities of data to “learn” — in this case, what’s a cloud and not a cloud. But given the camera was so new, “we didn’t have any data,” says Furano. “We had to train our application on synthetic data extracted from existing missions.”

- All this system and software integration and testing, with involvement of a half-dozen different organizations across Europe, took four months to complete. “We were very proud to be able to be so quick and so efficiently flexible, to put everything on board in such a short time,” says Max Pastena, PhiSat-1 officer at ESA. As far as spacecraft development goes, the timeline “is a miracle,” adds Furano.

- “Intel has given us background support on the Myriad device when we’ve needed it, to enable PhiSat-1’s AI using our CVAI Technology,”

says Dunne. “That’s very much appreciated.”

- Unfortunately, a series of unrelated events — delays with the rocket, the coronavirus pandemic and unfriendly summer winds — meant the teams had to wait more than a year to find out if PhiSat-1 would function in orbit as planned.

- The Sept. 2 launch from French Guiana — a first-of-its-kind satellite ride-share run by Arianespace — went fast and flawlessly. For the initial verification, the satellite saved all images and recorded its AI cloud detection decision for each, so the team on the ground could verify that its implanted brain was behaving as expected.

- After a three-week deep breath, Pastena was able to proclaim: “We have just entered the history of space.”

- ESA announced the joint team was “happy to reveal the first-ever hardware-accelerated AI inference of Earth observation images on an in-orbit satellite.”

- By only sending useful pixels, the satellite will now “improve bandwidth utilization and significantly reduce aggregated downlink costs” — not to mention saving scientists’ time on the ground.

- Looking forward, the usages for low-cost, AI-enhanced teensy satellites are innumerable — particularly when you add the ability to run multiple applications.

- “Rather than having dedicated hardware in a satellite that does one thing, it’s possible to switch networks in and out,” says Jonathan Byrne, head of the Intel Movidius technology office. Dunne calls this “satellite-as-a-service.”

- Consider: When flying over areas prone to wildfire, a satellite can spot fires and notify local responders in minutes rather than hours. Over oceans, which are typically ignored, a satellite can spot rogue ships or environmental accidents. Over forests and farms, a satellite can track soil moisture and the growth of crops. Over ice, it can track thickness and melting ponds to help monitor climate change.

- Many of these possibilities will soon be tested: ESA and Ubotica are working together on PhiSat-2, which will carry another Myriad 2 into orbit. PhiSat-2 will be “capable of running AI apps that can be developed, easily installed, validated and operated on the spacecraft during their flight using a simple user interface.”

- For Intel, the potential impact is unquestionable. As Pastena puts it, we can eventually understand “the pulse of our planet.”

• September 28, 2020: ESA’s 2020 -week event kicked off this morning with a series of stimulating speeches on Digital Twin Earth, updates on Φ-Sat-1, which was successfully launched into orbit earlier this month, and an exciting new initiative involving quantum computing. ¹⁸⁾

Digital Twin Earth

- The third edition of the Φ-week event, which is entirely virtual, focuses on how Earth observation can contribute to the concept of Digital Twin Earth – a dynamic, digital replica of our planet which accurately mimics Earth’s behavior. Constantly fed with Earth observation data, combined with in situ measurements and artificial intelligence, the Digital Twin Earth provides an accurate representation of the past, present and future changes of our world.

- Today’s session opened with inspiring statements from ESA’s Director General, Jan Wörner, ESA’s Director of Earth Observation Programs, Josef Aschbacher, ECMWF’s Director General, Florence Rabier, European Commission’s Deputy Director General for Defence Industry and Space, Pierre Delsaux, as well as Director General of DG CONNECT at the European Commission, Roberto Viola.

- Pierre Delsaux commented, “As our EU Commission President repeated recently during her State of the Union speech, it’s clear we need to address climate change. The Copernicus program offers us some of the best instruments, satellites, to give us a complete picture of our planet's health. But space is not only a monitoring tool, it is also about applied solutions for our economy to make it more green and more digital.”

- Roberto Viola said, “Φ-week is the week for disruptive technology and it is communities like this that our European programs were designed to support.”

- Florence Rabier added, “Machine learning and artificial intelligence could improve the realism and efficiency of the Digital Twin Earth – especially for extreme weather events and numerical forecast models.”

- Jan Wörner concluded, “Φ-week is the perfect example of the New Space approach focusing on disruptive innovation, artificial intelligence, agility and flexibility.”

- During the week, experts will come together to discuss the role of artificial intelligence for the Digital Twin Earth concept, its practical implementation, the infrastructure requirements needed to build the Digital Twin Earth, and present ideas on how industries and the science community can contribute.

Φ-sat-1

- Earlier this month, on 3 September, the first artificial intelligence (AI) technology carried onboard a European Earth observation mission, Φ-Sat-1, was launched from Europe’s spaceport in French Guiana. An enhancement of the Federated Satellite Systems mission (FSSCat), the pioneering artificial intelligence technology is the first experiment to improve the efficiency of sending vast quantities of data back to Earth.

- Today, ESA, along with cosine remote sensing, are happy to reveal the first ever hardware-accelerated AI inference of Earth observation images on an in-orbit satellite – performed by a Deep Convolutional Neural Network, developed by the University of Pisa.

- Φ-Sat-1 has successfully enabled the pre-filtering of Earth observation data so that only relevant part of the image with usable information are downlinked to the ground, thereby improving bandwidth utilization and significantly reducing aggregated downlink costs.

- Initial data downlinked from the satellite has shown that the AI-powered automatic cloud detection algorithm has correctly sorted hyperspectral Earth observation imagery from the satellite’s sensor into cloudy and non-cloudy pixels.

- Massimiliano Pastena, Φ-sat-1 Technical Officer at ESA, commented, “We have just entered the history of space.”

- Today’s successful application of the Ubotica Artificial Intelligence technology, which is powered by the Intel Movidius Myriad 2 Vision Processing Unit, has demonstrated real on-board data processing autonomy.

- Aubrey Dunne, Co-Founder and Vice President of Engineering at Ubotica Technologies, said, “We are very excited to be a key part of what is to our knowledge the first ever demonstration of AI applied to Earth Observation data on a flying satellite. This is a watershed moment both for onboard processing of satellite data, and for the future of AI inference in orbital applications.”

- As the overall 2017 Copernicus Masters winner, FSSCat, was proposed by Spain’s Universitat Politècnica de Catalunya and developed by a consortium of European companies and institutes including Tyvak International.

Quantum computing

- Also mentioned in his opening speech this morning, Josef Aschbacher made a special announcement regarding an exciting new ESA initiative, the “EOP AI-enhanced Quantum Initiative for EO - QC4EO” in collaboration with the European Organization for Nuclear Research (CERN).

- Quantum computing has the potential to improve performance, decrease computational costs and solve previously intractable problems in Earth observation by exploiting quantum phenomena such as superposition, entanglement and tunnelling.

- The initiative involves creating a quantum capability which will have the ability to solve demanding Earth observation problems by using artificial intelligence to support programs such as Digital Twin Earth and Copernicus. The initiative will be developed at the Φ-lab – an ESA laboratory at ESA/ESRIN (ESA’s Center for Earth Observation) in Frascati, Italy, which embraces transformational innovation in Earth observation.

- ESA and CERN enjoy a long-standing collaboration, centered on technological matters and fundamental physics. This collaboration will be extended to link to the CERN Quantum Technology Initiative, which was announced in June 2020 by the CERN Director General, Fabiola Gianotti.

- Through this partnership, ESA and CERN will create new synergies, building on their common experience in big data, data mining and pattern recognition.

- Giuseppe Borghi, Head of the Φ-lab, said, “Quantum computing together with AI are perhaps the most promising breakthrough to come along in computer technology. In the coming years, we will see more Earth or space science disciplines employing current or future quantum computing techniques to solve geoscience problems.”

- Josef Aschbacher added, “ESA will exploit the broad range of specialized expertise available at ESA and we will place ourselves in a unique position and take a leading role in the development of quantum technologies in the Earth observation domain.”

- Alberto Di Meglio, Coordinator of the CERN Quantum Technology Initiative, said, “Quantum technologies are a rapidly growing field of research and their applications have the potential to revolutionize the way we do science. Preparing for that paradigm change, by building knowledge and tools, is essential. This new collaboration on quantum technologies bears great promise.”

Sensor Complement

HyperScout-2

The HyperScout product series consists of very compact spectral imagers, in the order of 1-2 kg. The instruments are rather inexpensive, which are not meant to replace or compete with larger institutional satellites like Sentinel or Landsat, but more to complement large observation spacecrafts, being exploited for early detection of anomalies, as well as other applications. HyperScout is the one of the first hyperspectral imagers with such a compact envelope, therefore enabling applications which rely on the potential of very high temporal resolution (i.e. order of hours) in order to detect anomalies as early as possible and to refer to other space or ground assets for more detailed investigations.

A spectral imager with such a small envelope becomes an attractive solution for deployment in constellations, and therefore offers to the user the possibility of achieving high revisit times over regions of interest. The HyperScout concept is a combination of state of the art optics packed in an extremely compact format, in combination with very powerful processing abilities. It was first designed to be installed on CubeSat and small satellites, however it can be easily incorporated into larger platforms given the small engineering budgets and the compact volume.

HyperScout-2 is based on the building blocks of HyperScout-1 that was successfully demonstrated in flight by cosine Remote Sensing and ESA onboard the ESA GOMX-4B satellite and it is operational since February 2018. ⁶⁾ The HyperScout-2 specifications are reported in Table 1. The optical front-end is based on a monolithically and reflective TMA configuration, based on freeform optical design and manufactured using diamond turning machining technique. The dual channel implementation envisages a beam splitter at the end of the front end optics, diverting the shorter wavelengths towards the VNIR sensor, and allowing the longer wavelengths to impinge on the TIR sensor.

This configuration was envisaged during other ESA activities related to navigation and planetary science, with goal of designing a multispectral camera for relative navigation. ⁷⁾ A very compact relay has been conceived in order to enhance the numerical aperture of the TIR channel. The VNIR Focal Plane Array (FPA) is a CMOS sensor with a linearly variable hyperspectral filtering element that separates the different wavelengths from 400 to 1000 nm, while the TIR FPA is based on a microbolometer and a set of four spectral filters from 8 to 14 µm. Both FPAs are 2D sensors operated in push broom mode. The TIR sensor is actively thermally controlled via a thermoelectric cooler (TEC) to limit the dependency of the sensor readout on the sensor temperature fluctuations.

HyperScout-2 has the following subsystems:

• Telescope assembly;

• VNIR Focal Plane Array (VNIR FPA);

• TIR Focal Plane Array (TIR FPA);

• Back-End Electronics unit (BEE);

• On-Board Data Handling (OBDH);

• Mass Memory Units (MMUs): operated in hot and cold spare configuration;

• Instrument Control Unit (ICU): main point of contact from the S/C bus for data interface and main instrument control;

• Eyes of Things board (EOT): data processing board for on-board artificial intelligence algorithms.

The architectural diagram of HyperScout-2 is depicted in Figure 2, while the subsystems locations in the payload is indicated in Figure 3. The back-end electronics (BEE) is a compact and versatile FPGA based single board image acquisition system. The On-Board Data Handling (OBDH) is physically installed on the BEE. The Instrument Control Unit (ICU) controls and monitors the operational status of the payload subsystems. The Vision Processing Unit (VPU) consists of a Myriad 2 processor located on the Eyes of Things (EoT) board. The ICU can independently power the BEE, OBDH, the EoT board and each MMU. The FPAs are independently powered via the BEE. All subsystems have independent latch-up and overcurrent protection.

The majority of the HyperScout subsystems can be operated independently from each other, offering an additional degree of freedom to optimize the power budget by powering up only the components that play a role within the specific operating mode. The ICU is always powered to serve as the contact point for communication with the S/C bus and to monitor the status of the payload throughout the orbit. During acquisition the FPAs, BEE and MMU are mainly used. In this phase for example, the OBDH will consume minimal power transferring the image data to disk. During processing both FPAs and BEE are powered off completely. The OBDH will normally consume the most power in the processing mode. The dependency of the subsystems per operating mode is reported in Figure 4.

The frequency of the observation mode can vary according to the mission planning. The acquired data is cached in the instrument MMUs for subsequent processing or transmission to ground, as requested by the S/C bus. The processing operating mode can be run using different processing subsystems. For this scope HyperScout-2 offers a powerful computational set of processors, including a FPGA, CPU, GPU and a dedicated board for AI inference represented by an Eyes Of Things (EOT) board consisting of the Intel® Movidius™ Myriad™ vision processing unit (VPU) designed for accelerating machine vision tasks. The Intel's Movidius™ Myriad™ 2 VPU (Ref. 5) (second generation VPU from Movidius™, an Intel® company) is specialized for AI, vision and imaging applications where both performance and low power consumption are important.

HyperScout-2 is therefore capable of performing a vast range of processing operations, that can be continuously updated during flight, even with completely new applications.

HyperScout-2 calibration and applications

In-flight Calibration approach: HyperScout-2 has been characterized in the laboratory before integration into the spacecraft. For what concerns in-flight calibration, typically on-board stimuli are used to calibrate the full optical chain for the long waves infrared channel, such as blackbodies. For a miniaturized instrument like HyperScout this is currently not an option. A lean system approach is adopted that relies on vicarious calibration in combination with a thermally stable instrument and detector.

For the absolute calibration of the HyperScout VNIR spectral channel, instrumented test sites of RadCalNet are considered. Test sites consist of large, homogeneous, cloud-free areas used as radiance or reflectance reference targets. The method relies on the comparison of satellite date with a simultaneous ground truth measurements. Two methods are distinguished:

• Reflectance based: the reflectance of the earth surface is measured on-ground, together with extinction depth and other meteorological parameters. Subsequently a radiative transfer model is used to convert these values to a TOA (Top of Atmosphere) radiance, which can be compared to the sensor data. These methods allow absolute radiometric calibration with a accuracy <5%.

• Radiance based: the radiance of the earth scene is measured at an altitude much above aerosol scattering. After correction for residual scattering and absorption, the radiance can be compared directly to the sensor for absolute radiometric calibration within 2.8% accuracy.

Pseudo invariant characterization sites (PICS), are sites with a high temporal or spatial stability. Desert sites are the most promising candidates as they are characterized by large, homogeneous areas and constant atmospheric conditions (no clouds). An extensive catalogue with site information is available on the websites of the USGS and CEOS. The most common application is to monitor the relative radiometric response of the sensor, without the need of an on-ground reference measurement.

Data will be crossed calibrated on the long term using Sentinel-2A/-2B, acquired over the same region of interest (ROI). For the purposes of cross-calibration of HyperScout, the Sentinel-2 satellites are considered thanks to the good match between the spectral bands of the Sentinel Multispectral Instrument (MSI) and HyperScout and the in-orbit HyperScout data collection opportunity provided by the FSSCat spacecraft with limited costs for cosine.

For the TIR channel in-flight calibration, a spacecraft maneuver with the instrument pointed at deep space is used to measure the instrument thermal background signal. As there is no black body calibrator on-board the payload for in-flight calibration, two options are considered: 1) intercalibration or cross-calibration, 2) vicarious calibration. Intercalibration calibrates a satellite instrument by relating its measurements to those of a well calibrated reference satellite instrument. This technique relates measurements in similar spectral bands. Well-calibrated reference satellite instruments that match each TIR band of HyperScout-2 are for example Sentinel-3, MODIS or Landsat-8. Vicarious calibration uses natural or artificial sites on the surface of the earth to calibrate satellite instruments. Within this technique, well-calibrated ground-based or airborne radiometers take measurements of a spectrally and spatially homogeneous test sites at the time of the satellite instrument overpass.

Two methods can be used to perform vicarious calibration in the thermal infrared:

• Radiance-based method;

• Temperature-based method.

The radiance-based method requires that the spectral response of the well-calibrated reference radiometer matches the spectral response of the satellite instrument. In practice, this condition is most likely not going to be fulfilled. Consequently, this method is also not considered to be a suitable option for inflight monitoring and calibration. In contrast to the radiance-based method, the temperature-based method in principle does not impose restrictions on the spectral response of the instrument. Hence, this method is considered a suitable option for in-flight monitoring and calibration. The following calibration sites could be envisaged for the in-flight calibration of HyperScout:

• Lake Tahoe on the California/Nevada border

- ideal thermal calibration target due to thin atmosphere above the lake thanks to the high altitude

- does not freeze in winter as it is very deep

- annual temperature ranges from about 4 to 20°C

- excluding instrument noise, uncertainty in the predicted at-sensor spectral radiance for TIR bands and skin surface temperature of 300 K is about 0.43 K - expressed in equivalent apparent temperature

- freely available data

• Salton Sea in Southern California

- less ideal target than Lake Tahoe due to thicker atmosphere (lower altitude)

- annual temperature variation ranges from about 4 to 35°C (extends temperature range compared to lake Tahoe)

- excluding instrument noise, uncertainty in the predicted at-sensor spectral radiance for TIR bands and skin surface temperature of 300 K is about 0.43 K - expressed in equivalent apparent temperature

- freely available data

• NOAA Ocean and Great Lakes

- larger surface area than Lake Tahoe and Salton Sea

- temperature variation ranges from about 3 to 30°C depending on the season and location

- excluding instrument noise, uncertainty in the predicted at-sensor spectral radiance for TIR bands and skin surface temperature of 300 K is about 0.48 K - expressed in equivalent apparent temperature

- freely available data.

cosine is also leading a Dutch consortium for the development and commercialization of a miniaturized spectroradiometric calibration stimuli, expected to be integrated and tested in orbit in late 2020 / early 2021. This miniaturized calibration source will be considered for future flights of HyperScout related products as well as third parties compact spectrometers.

HyperScout-2 data products

A number of data products have been selected for HyperScout-1 and this is being extended for the HyperScout-2 mission adding the potential applications enabled by the coexistence of reflectance and thermal information of the target. The target applications are: agriculture, thermal inertia and soil moisture, urban heat island, fire hazard and monitoring, and water quality.

MultiScape100

MultiScape100 instrument from Simera Innovate GmbH (CH) is a push-broom imager. The imager provides continuous line-scan imaging in 8 spectral bands in the visible and nearinfrared (VNIR) spectral range. A push-broom instrument obtains the images by scanning along the ground track as the spacecraft is orbiting the Earth (see Figure (TBD) below). The scan is done for each spectral band separately.

In Table (TBD) below information on each spectral band including its line number on the detector plane is provided.

​​​​​

The ground sampling distance (GSD) for each band from a 500 km orbit is 4.75 m.
The modulation transfer function (MTF) for separate spectral bands is expected to be between 3.9% and 7.2% at Nyquist Frequency. The SNR is expected to be between 54 and 129 for the multispectral bands. The expected SNR for the PAN band is 256.
The images of each spectral band are retrieved by the on-board computer and stored in its internal memory. The images are then radiometrically corrected, the spectral bands are coregistered, and pixels are geo-located. The data is then ready to be used by AI applications.
By default, the roll angle will be set to 0. The spacecraft is capable of capturing images with at least 15 degrees roll angle while keeping pitch at 0 degrees. No pitch manoeuvres are planned to be used during imaging.
Unless specified by the end user, the spacecraft will only capture images when the Solar Zenith Angle is between 0 and 90 degrees, i.e., when the Earth’s surface is illuminated by the sun.

Payload Chain - CogniSat AI Processor

The Ubotica CogniSAT-XE1TM CubeSat Board (Figure (TBD)) brings the power of Edge CV and AI compute acceleration to a PC/104 form-factor for SmallSat and CubeSat missions. It is built around the Intel® Movidius™ Myriad™ 2 Computer Vision (CV) and Artificial Intelligence (AI) COTS VPU whose 12 vector cores provide high-performance parallel and hardware accelerated compute within a low power envelope.
CogniSat™ combines the power efficient compute of the Myriad 2 VPU with a wide range of interfaces and peripherals, providing broad flexibility for integration into satellite platforms.
Either Gigabit Ethernet or USB2.0/3.0 can be used as the primary control and data interface to the board, enabling data rates sufficient to handle many CV and AI applications at nearstreaming throughput.
Common NN frameworks (e.g., TensorFlow, PyTorch, Caffe) can be used for NN model development and training, with the model subsequently imported into Intel®’s OpenVINO™ toolkit for targeting to the Myriad device. CogniSat™ leverages the broad range of pre-qualified models and layers available within OpenVINO™.
Custom Computer Vision pipelines can easily be deployed and executed on CogniSat™ using the CVAI Toolkit™ software toolkit. The software supports application-specific CV and ISP pipelines that utilise a combination of the power-efficient Myriad 2 streaming hardware blocks and software filters implemented on the vector engines. Deployment to the hardware platform involves the transfer of only a single configuration file, and runtime updates enable the updating of pipelines without requiring application re-compile or system reboot.

Pre-Selected A.I Applications

Four AI applications are already pre-selected for Φsat-2 mission: Sat2Map App, Cloud Detection App, Vessel Detection App, Compression App.
Sat2Map App: The objective of this application is to demonstrate how the exploitation of AI on board will enable the generation and delivery of “near real time” to the ground.
Sat2Map App - Transforms a satellite image to a street map using Artificial Intelligence. The APP will rely on Cycle-Consistent Adversarial Networks (CycleGAN) deep learning technique to do the transformation from the satellite image to the street map.
Cloud Detection App: Cloud detection feature is essential in EO satellites focused on on-board data processing. Although the concept is well-established in on-ground processing pipelines and even demonstrated aboard satellites for bandwidth reduction, the Φsat-2 is taking the next step and proposes a novel application-agnostic deep learning-powered cloud detection service.
The App will demonstrate the APP chaining on board showing how the output of different applications can be exploited by others directly on board. The other App can then use cloud mask as input (including as well semi-transparent cloud, shadow)
Vessel Detection App: The Autonomous Vessel Awareness experiment (AVA) is an application which aims to demonstrate in Φsat-2 the capability to autonomously develop awareness about vessels in the maritime domain using Artificial Intelligence (AI).
The AVA app will exploit latest deep neural network techniques to perform not only vessel detection but also vessels classification.
Compression App: The Deep Compression application aims at demonstrating that AI processing on-board can reduce the amount of data to be stored and sent to the ground with a limited information loss enabling other APPs to reconstruct the image on the ground exploiting directly the compressed image (e.g., for object detection). The app exploits a Deep Autoencoder to perform image compression.

References

1) ”First Earth observation satellite with AI ready for launch,” ESA, 12 September 2019, URL: http://www.esa.int/Our_Activities/Observing_the_Earth/First_Earth_observation_satellite_with_AI_ready_for_launch

2) ”Smallsats win big prize at Copernicus Masters,” ESA, 8 November 2017, URL: http://www.esa.int/Our_Activities/Observing_the_Earth/Copernicus/Smallsats_win_big_prize_at_Copernicus_Masters

3) Marco Esposito, Bernardo Carnicero Domınguez, Massimiliano Pastena, Nathan Vercruyssen, Simon Silvio Conticello, Chris van Dijk, Pierluigi Foglia Manzillo, Rick Koeleman,”Highly integration of hyperspectral, thermal and artificial intelligence for the ESA PhiSat-1 mission,” Proceedings of the 70^th IAC (International Astronautical Congress), Washington DC, USA, 21-25 October 2019, URL: https://iafastro.directory/iac/proceedings/IAC-19/IAC-19/B4/4/manuscripts/IAC-19,B4,4,4,x54611.pdf

4) Massimiliano Pastena, Bernardo Carnicero Domínguez, Pierre Philippe Mathieu, Aman d a Re ga n,Marco Esposito, Simon Conticello, Chris Van Dijk, Nathan Vercruyssen, Pierluigi Foglia Manzillo, Rick Koelemann, John Hefele, ” Earth Observation Directorate NewSpace Initiatives,” Proceedings of the 70^th IAC (International Astronautical Congress), Washington DC, USA, 21-25 October 2019, paper: SSC19-V-05, URL: https://digitalcommons.usu.edu/cgi/viewcontent.cgi?article=4391&context=smallsat

5) A Camps, A. Golkar, A. Gitierrez, J. Ruit-de-Azua, J. Muñoz‐Martin, L. Fernandez-Capon, C. Diez, A. Aguilella, S. Briatore, N. Garzaniti, F. Nichele, R. Mozzillo, M. Vanotti, M. Esposito, B. Carnicero, G. Filippazzo, and A. Regan, “FSSCat: A Cubesat-based Tandem Mission for Earth Observation of the Polar Regions” Living Planet Symposium ‘19, Milan, May 2019

6) Marco Esposito, Alessandro Zuccaro Marchi, “In-Orbit Demonstration of the first hyperspectral imager for nanosatellites” ICSO ( International Conference on Space Optics) 2018, Chania, Greece

7) Pierluigi Foglia Manzillo, Ljubisa Babic, Marco Esposito, Cristian Corneliu Chitu, Marco Canetri, Paolo Corradi, Jesus Gil Fernandez, Michael Kueppers. ”TIRI: A Multi-Purpose Thermal Infrared Payload for Planetary Exploration” Proceedings of the 69th IAC (International Astronautical Congress) Bremen, Germany, 1-5 October 2018, paper: IAC-18,A3,4B,4, URL: https://iafastro.directory/iac/proceedings/IAC-18/IAC-18/A3/4B/manuscripts/IAC-18,A3,4B,4,x47339.pdf

8) ”Vega return to flight proves new rideshare service,” ESA Enabling & Support, 3 September 2020, URL: https://www.esa.int/Enabling_Support/Space_Transportation/Vega/Vega_return_to_flight_proves_new_rideshare_service

9) ”With Vega, Arianespace successfully performs the first European mission to launch multiple small satellites,” Arianespace Press Release, 3 September 2020, URL: https://www.arianespace.com/press-release/with-vega-arianespace-successfully-performs-the-first-european-mission-to-launch-multiple-small-satellites/

10) ”Vega flight VV16,” Wikipedia, URL: https://en.wikipedia.org/wiki/Vega_flight_VV16

11) Mike Safyan, ”Planet’s First Launch of 2020: 26 SuperDoves on a Vega,” Planet, 13 February 2020, URL: https://www.planet.com/pulse/planets-first-launch-of-2020-26-superdoves-on-a-vega/

12) ”Overview of the FSSCAT Optical Intersatellite Link (O-ISL) - Technology Demonstration,” EPFL, 2019, URL: https://memento.epfl.ch/event/overview-of-the-fsscat-optical-intersatellite-link/

13) ”Smarter satellites: ESA Discovery accelerates AI in space,” ESA Enabling & Support, 12 May 2022, URL: https://www.esa.int/Enabling_Support/Preparing_for_the_Future/Discovery_and_Preparation/Smarter_satellites_ESA_Discovery_accelerates_AI_in_space

14) ”Working towards a Digital Twin of Earth,” ESA Applications, 14 October 2021, URL: https://www.esa.int/Applications/Observing_the_Earth/Working_towards_a_Digital_Twin_of_Earth

15) ”Putting the Earth observation into ‘the market perspective’,” ESA Applications, 13 October 2021, URL: https://www.esa.int/Applications/Observing_the_Earth/Putting_the_Earth_observation_into_the_market_perspective

16) ”Spotlight on climate and the New Space economy,” ESA Applications, 11 October 2021, URL: https://www.esa.int/Applications/Observing_the_Earth/Spotlight_on_climate_and_the_New_Space_economy

17) ”Intel Powers First Satellite with AI on Board,” Intel, 20 October 2020, URL: https://newsroom.intel.de/news/intel-powers-first-satellite-with-ai-on-board/#gs.jvbakw

18) ”Digital Twin Earth, quantum computing and AI take centre stage at ESA’s Φ-week,” ESA Applications, 28 September 2020, URL: https://www.esa.int/Applications/Observing_the_Earth/Digital_Twin_Earth_quantum_computing_and_AI_take_centre_stage_at_ESA_s_Ph-week 

19) ESA, “ΦSAT-2 Mission Overview for the #ORBITALAI CHALLENGE”, ESA UNCLASSIFIED – Releasable to the Public, February 15, 2023, URL: https://ai4eo.eu/wp-content/uploads/2023/02/Phisat-2_Mission_Overview_Web.pdf  

20) ESA, “OrbitalAI Φsat-2 Challenge”, AI4EO, URL: https://platform.ai4eo.eu/orbitalai-phisat-2