You Can’t Fix What You Can’t See: The Realities of AI and Satellite Data

Earth observation (EO), the monitoring of the Earth from space using satellites, has undergone fundamental changes in the last decade. We have seen the convergence of two exciting trends in remote sensing and processing algorithms that now herald a new era of space renaissance.

The implementation of ambitious government initiatives such as the European Union’s Copernicus Programme, and an explosion in commercial satellite sensing constellations like Planet’s, has been matched by incredible breakthroughs in algorithm performance. This is due to advancements in accelerated computing, open source software, and broadly accessible training data.

From a time of relative data paucity, we have entered an era where data exploitation is often compared to taking a drink from a fire hydrant. This change in the availability of what is considered quite high spatial resolution (5-10 m pixels) image data has left many potential users reeling from the opportunities of these new-found riches, and the EO service industry IS scrabbling for the best ways to deal with the data deluge.

At the forefront of the strategies to convert vast amounts of data into information has been artificial intelligence (AI), a system able to perform human tasks such as speech recognition, visual perception and decision-making. In the case of vast amounts of image data, AI has the potential to help humans extract insights from huge volumes of unstructured spatiotemporal data and facilitate the process of information discovery. In short, no number of humans would be able to analyze the millions of images being sent down to Earth daily—but, potentially—an AI-capable machine could support this.

An obviously powerful tool, AI attracts a great deal of suspicion and criticism and has rather negative connotations for some, such as George Orwell’s Big Brother in the novel 1984, the HAL9000 in 2001: A Space Odyssey, or SkyNet in Terminator. Others see unregulated and unaccountable forms of surveillance supported by AI as a threat to privacy and an intrusion on our lives. A more rational fear is around the uncritical use of AI by computer engineers who have received little training in the issues being addressed and may be unaware of existing solutions which may be relevant. AI could also be an amplifier of the power imbalances between data, information suppliers and users, and the general public. Therefore, can we trust an AI solution to be appropriate and unbiased?

Dove satellite flying over Earth © 2019, Planet Labs Inc. All Rights Reserved.

AI is a “black box,” and even to those deploying it, its rules may remain unknowable. Such black boxes reduce transparency in decision-making. To effectively use AI we must understand the limitations of the inputs to help build trust in the outputs.

One key fact often missed from the discussions of the combined exploitation of AI and EO data is that the observing capability of EO is constrained by the underlying physics of imaging systems. Although tremendous progress has been made in the last 100 years in sensor design (like multi-spectral imaging beyond the capabilities of the human eye), the underlying physics of optics hasn’t changed and we are still unable to get closer than 500km from an orbiting platform. The design and operation of an imaging system must therefore fit within a well-known specification envelope agreed for the system when built.

In terms of optical systems, the formula says that the larger the telescope, the more light (photons) you can collect, and the more detail on the ground you’ll be able to see (e.g. higher spatial resolution). However, with a large telescope capturing high spatial resolution, you end up with an imaging capability that has a narrow field of view; it’s like looking down at the Earth through a soda straw. Conversely, lower spatial resolution satellites have a wider field of view and can capture more of the ground when they image. It is therefore not surprising that there are no sub-meter spatial resolution systems producing images with footprints of hundreds of kilometers.

Even with the above constraints, the daily data volumes produced by systems such as the Planet constellations and the Copernicus high performance imagers, are still considerable and in the order of tens of Tb per day. These volumes are multiplied significantly when considering multi-temporal analyses over extents up to global scales. Thus, for operational applications the amount of raw or pre-processed data available for analysis can be daunting.

Vessel detection in Port of Wonsan, North Korea © 2019, Planet Labs Inc. All Rights Reserved.

Some form of data rationalization is vital and AI does have a key strength of being able to identify patterns in dense time series of spatially detailed data. AI has been shown to work well when identifying clearly specified and distinct features in the landscape or at sea.  It has been possible to map the road networks and physical structures of refugee camps in the Middle East, deforestation in the tropics, and the types and number of ships in harbours (Figure 3). However, once we get to less clearly defined problems and subtle changes in landscapes which are not well understood at the scale of EO data, then the AI will still be challenged by these requirements.

Fortunately, the new constellations of EO sensors offer the ability to capture daily images. The higher density of spatiotemporal observations provides a more solid statistical basis for monitoring and modeling a broad range of phenomena like the phenology of vegetation and in building confidence when assessing persistent change. However, it’s up to us to first understand the limitations of our knowledge, the uncertainties in the system, and the constraints of the input data and their physics when pressing on with an AI-based solution.

In summary, we obviously need some form of AI to be able to drink effectively from the image data “fire hydrant” that is now being made available to service developers. However, our expectations of what EO can deliver should be based on what is actually observable in the images (or temporal data cube), not what we wish to see based on preconceived notions we may have when designing and training our AI models. When asked to deliver impossible results, chief engineer Montgomery Scott of the USS Enterprise said, “I canna’ change the laws of physics, Captain!” This still holds today when exploiting EO.

More recently, the phrase, “You can’t fix what you can’t see,” was said by Planet CEO Will Marshall to highlight the need to increase the frequency, scale and, possibly, modality of the observations employed to monitor our planet. While it has gained traction to promote the need for EO-based applications, it can also be taken as a warning about being asked to deliver impossible results given the constraints of the input data. Therefore, as we consider how EO is exploited to address environmental problems, we should heed both of these phrases to make sure that we can see what we hope to address before we enlist the services of powerful black boxes in the hope of seeing it.

Dove satellite flying over our world © 2019, Planet Labs Inc. All Rights Reserved.

This piece was co-authored by Giovanni Marchisio, vice president of analytics in product engineering at Planet, and Geoff Smith, director and consultant at Specto Natura Ltd.