Snapshot CMOS: The Future of Hyperspectral Imaging.
Lead Research Organisation:
The Open University
Department Name: Physical Sciences
Abstract
Whilst Charge-Coupled Devices (CCDs) have been used for Hyperspectral missions for many years with great success, new developments in Complementary Metal Oxide Semiconductor (CMOS) image sensor technology offer the chance to significantly improve detectors for, and to provide higher resolution datasets in Earth observation. e2v technologies are the supplier of CCD imagers to the European Space Agency's (ESA) Sentinel missions. Future Sentinels, launched from 2013, are to carry a range of technologies from radar to Hyperspectral imaging instruments for land, ocean and atmospheric monitoring, covering a wide range of NERC environmental science themes. However, as CCD image sensors have fundamental limitations in this application which prohibit further improvements in performance, to move forwards and provide significant advances in this field one must consider the use of the newer and potentially superior technology and establish programmes for investment and development.
CMOS Image Sensor (CIS) technology has many potential advantages over CCD-based systems. Firstly, each Hyperspectral image consists of many spectral lines which vary largely in intensity. A CCD must transfer all faint spectral lines through the part of the imager that has been illuminated by more intense lines, leading to cross-talk and reducing the quality of the dataset. CIS sensors do not require charge to be transferred and can therefore completely remove this cross-talk. Secondly, a CCD can only operate at one frame-rate and one sensitivity at any given time and a compromise must be made in the sensitivity and dynamic range; the difference in the brightness in an image between snow, vegetation and water can vary dramatically yet the CCD can only be optimised for one spectral band. Advanced CIS pixels offer the potential to be read and reset in any order at any time, allowing the sensitivity to be set on a line-by-line basis; high-intensity bands can be read out more frequently, dramatically increasing the dynamic range of the detector.
Current CMOS image sensors are generally based around 4 or 5 transistors per pixel (4T or 5T), with 5 transistors allowing the application of a global reset or double sampling, with Correlated Double Sampling (CDS) applied off-pixel. However, a global snapshot shutter is required to ensure that all pixels are integrating over exactly the same time period and therefore the same region of the Earth's surface, removing smearing that can be present when using CCDs due to the transfer of charge. Lowest noise performance can only be achieved through the use of CDS, which must be included in the pixel to allow variable readout-rates from one spectral band to the next, therefore optimising the sensitivity across all spectral bands. However, these properties cannot be achieved simultaneously using current 5T CIS technology. In order to achieve both a global snapshot shutter and in-pixel CDS, one must develop a CIS pixel containing many more transistors.
Through innovations in CIS at e2v, a new 10T pixel design has been implemented in a small-area test array. This technology is as yet unproven (currently at TRL 2) and requires thorough characterisation to determine not only the more general performance of the pixel, but the specific applicability to the field of Hyperspectral imaging. Through in-depth characterisation and optimisation of the pixel, backed up by Silvaco ATLAS simulations of the pixel performance, we aim to implement a proof-of-concept study of this new development in CIS technology for the field of Hyperspectral imaging. The programme would proceed through TRL 3 with testing of analytical and critical function, moving into testing for TRL 4 through component and breadboard validation. Only through in-depth characterisation, optimisation and simulation can the device be fully analysed and optimised, leading to the consequent developments for the design and production into a full-scale device.
CMOS Image Sensor (CIS) technology has many potential advantages over CCD-based systems. Firstly, each Hyperspectral image consists of many spectral lines which vary largely in intensity. A CCD must transfer all faint spectral lines through the part of the imager that has been illuminated by more intense lines, leading to cross-talk and reducing the quality of the dataset. CIS sensors do not require charge to be transferred and can therefore completely remove this cross-talk. Secondly, a CCD can only operate at one frame-rate and one sensitivity at any given time and a compromise must be made in the sensitivity and dynamic range; the difference in the brightness in an image between snow, vegetation and water can vary dramatically yet the CCD can only be optimised for one spectral band. Advanced CIS pixels offer the potential to be read and reset in any order at any time, allowing the sensitivity to be set on a line-by-line basis; high-intensity bands can be read out more frequently, dramatically increasing the dynamic range of the detector.
Current CMOS image sensors are generally based around 4 or 5 transistors per pixel (4T or 5T), with 5 transistors allowing the application of a global reset or double sampling, with Correlated Double Sampling (CDS) applied off-pixel. However, a global snapshot shutter is required to ensure that all pixels are integrating over exactly the same time period and therefore the same region of the Earth's surface, removing smearing that can be present when using CCDs due to the transfer of charge. Lowest noise performance can only be achieved through the use of CDS, which must be included in the pixel to allow variable readout-rates from one spectral band to the next, therefore optimising the sensitivity across all spectral bands. However, these properties cannot be achieved simultaneously using current 5T CIS technology. In order to achieve both a global snapshot shutter and in-pixel CDS, one must develop a CIS pixel containing many more transistors.
Through innovations in CIS at e2v, a new 10T pixel design has been implemented in a small-area test array. This technology is as yet unproven (currently at TRL 2) and requires thorough characterisation to determine not only the more general performance of the pixel, but the specific applicability to the field of Hyperspectral imaging. Through in-depth characterisation and optimisation of the pixel, backed up by Silvaco ATLAS simulations of the pixel performance, we aim to implement a proof-of-concept study of this new development in CIS technology for the field of Hyperspectral imaging. The programme would proceed through TRL 3 with testing of analytical and critical function, moving into testing for TRL 4 through component and breadboard validation. Only through in-depth characterisation, optimisation and simulation can the device be fully analysed and optimised, leading to the consequent developments for the design and production into a full-scale device.
Planned Impact
Imaging in Earth observation has many uses across the full range of NERC scientific areas of interest, including the climate system, biodiversity, the sustainable use of the Earth's natural resources, Earth system science, natural hazards and their effects and pollution. By improving the quality of data sets from future missions through advancing the technology of the detectors, it is possible to impact a wide range of people.
Through the characterisation and optimisation of the 10T CMOS pixel, it will be possible for a business case to be made at e2v for a full global snapshot shutter imaging device with in-pixel CDS (see Case for Support). Whilst e2v currently provide many CCDs for Hyperspectral imaging, these new developments would enable them to maintain their market position. This benefits both e2v as a company and the UK economy through job creation and market growth. With appropriate funding, this growth and development could be achieved over the next few years.
Staff working on the project will develop their research and professional skills. The universal skills of problem solving and time management are relevant across a wide range of employment sectors. More specifically, the characterisation and optimisation of silicon-based detectors has relevance across a large number of areas in high-technology industries.
Beyond e2v, the scope of positive impact is much wider. Academics with interests in any of the above NERC science themes will be able to benefit from the higher resolution datasets provided by the improved detectors when placed in orbit in future missions. More details on the Academic beneficiaries can be found in the separate section of the appropriate document.
Hyperspectral imaging is a standard technique in Earth observation, with many missions making use of such imagers as outlined in the Case for Support. These missions range from land and ocean monitoring to the investigation of atmospheric phenomena. With such a wide scoping range of applications, the benefits brought about by the improved quality of datasets from such missions will encompass many of NERC's scientific themes. With this in mind, one can consider that these improvements will impact not only the academics who collect and make use of the data from such missions, but they will also affect the wider public who benefit from the research carried out by said academics.
Taking one such example, one can consider how the impact of detector developments in Hyperspectral imaging might impact the wider public at large. The eruption of the volcano Eyjafjallajökull in 2010 caused widespread disruption world-wide, paralysing Europe through travel chaos. ESA's small Proba-1 Earth-watcher provided imagery of the source, revealing the breach in the glacier made by the volcanic eruption. The part of the steam and ash plume that rose from the icecap was tracked by ESA's Envisat as it spread across Europe. Such data provided vital information to aid in the management of the situation, from the import and export of goods via airfreight through to the disrupted holiday plans of Joe Public; results from research in these areas are essential to guide policy makers across all scales, from local to international levels.
Hyperspectral imaging has many wide ranging applications, all of relevance to NERC's science themes, with each area covering a wide range of beneficiaries in both academia and the general population. Recent studies have shown that a geostationary Hyperspectral imager can provide the most significant benefits in short term regional numerical weather prediction models. Such improvements benefit both the academics who are involved in the studies and the wider general industry and public who rely on accurate weather prediction in their day to day work and lives. Hyperspectral imaging also has applications on many other scales, from deriving the leaf chlorophyll content of green-leaf desert plants to the monitoring of global vegetation.
Through the characterisation and optimisation of the 10T CMOS pixel, it will be possible for a business case to be made at e2v for a full global snapshot shutter imaging device with in-pixel CDS (see Case for Support). Whilst e2v currently provide many CCDs for Hyperspectral imaging, these new developments would enable them to maintain their market position. This benefits both e2v as a company and the UK economy through job creation and market growth. With appropriate funding, this growth and development could be achieved over the next few years.
Staff working on the project will develop their research and professional skills. The universal skills of problem solving and time management are relevant across a wide range of employment sectors. More specifically, the characterisation and optimisation of silicon-based detectors has relevance across a large number of areas in high-technology industries.
Beyond e2v, the scope of positive impact is much wider. Academics with interests in any of the above NERC science themes will be able to benefit from the higher resolution datasets provided by the improved detectors when placed in orbit in future missions. More details on the Academic beneficiaries can be found in the separate section of the appropriate document.
Hyperspectral imaging is a standard technique in Earth observation, with many missions making use of such imagers as outlined in the Case for Support. These missions range from land and ocean monitoring to the investigation of atmospheric phenomena. With such a wide scoping range of applications, the benefits brought about by the improved quality of datasets from such missions will encompass many of NERC's scientific themes. With this in mind, one can consider that these improvements will impact not only the academics who collect and make use of the data from such missions, but they will also affect the wider public who benefit from the research carried out by said academics.
Taking one such example, one can consider how the impact of detector developments in Hyperspectral imaging might impact the wider public at large. The eruption of the volcano Eyjafjallajökull in 2010 caused widespread disruption world-wide, paralysing Europe through travel chaos. ESA's small Proba-1 Earth-watcher provided imagery of the source, revealing the breach in the glacier made by the volcanic eruption. The part of the steam and ash plume that rose from the icecap was tracked by ESA's Envisat as it spread across Europe. Such data provided vital information to aid in the management of the situation, from the import and export of goods via airfreight through to the disrupted holiday plans of Joe Public; results from research in these areas are essential to guide policy makers across all scales, from local to international levels.
Hyperspectral imaging has many wide ranging applications, all of relevance to NERC's science themes, with each area covering a wide range of beneficiaries in both academia and the general population. Recent studies have shown that a geostationary Hyperspectral imager can provide the most significant benefits in short term regional numerical weather prediction models. Such improvements benefit both the academics who are involved in the studies and the wider general industry and public who rely on accurate weather prediction in their day to day work and lives. Hyperspectral imaging also has applications on many other scales, from deriving the leaf chlorophyll content of green-leaf desert plants to the monitoring of global vegetation.
| Description | The developments made over the course of this grant represent a large step towards the use of optimised snapshot CMOS sensors for Hyperspectral imaging. Through our collaborative links with e2v technologies and through efficient and effective knowledge exchange, we have made great strides and have been able to propose and outline the developments required to produce a new generation of sensors for Earth observation and Hyperspectral imaging. The pixel architecture of the sensor offers several advantages over previous designs. The design, without going into the complex in-pixel architecture, offers increased sensitivity, greatly reduced power consumption and suppressed common mode noise (full technical details will be in the publications currently in press). This 10T architecture, when compared to reported 8T and 9T dseigns, offers increased device sensitivity under identical conditions. The finer details are again available in the related journal publications (in press). The readout noise of the sensor, however, was found to be approximately double that of a more standard 4T equivalent sensor. Our simulations show that there is good agreement between the measured readout noise and what one might expect. Methods for readout noise reduction in future sensors are under consideration and will be followed up in the next phase of the detector development (subject to funding). The fill factor of the device is 25%. This could be improved dramatically through the use of microlenses or through, more preferentially, backside illumination. This change in sensor design is one of the main areas proposed for future study (subject to funding). One final issue noted with the prototype sensor design is a large current draw which would make the device difficult to scale to the sizes required for Hyperspectral imaging applications. An alternative pixel design has been developed theoretically towards the end of the grant period that would allow for elimination of the standing current. This is the second of many improvements and optimisations that we have proposed to e2v and discussed at length with the aim towards continuing our research development in this area. Our main aim moving forwards it to take the results from this study, both positive and negative, and design, manufacture, test and optimise a new, larger format sensor. This next step towards an idealised Hyperspectral imager for Earth observation is vital to realise the large improvements that this could bring to the field and this will form the basis of a future application for funding from NERC, through the technology proof of concept scheme or otherwise. We believe that there is a great opportunity for the continued drive towards UK-produced sensors in Hyperspectral imaging and Earth observation, developed through efficient knowledge exchange between industry and academia. The studies towards the recommended developments listed above have helped to direct the development of a new generation of sensors, currently under test having been designed and manufactured over recent years, and showing great promise for the next generation of CMOS sensors for EO. |
| Exploitation Route | Throughout the grant, we have been concentrating on reaching two main audiences: device manufacturers e2v technologies and the academics that drive the requirements for Hyperspectral Imaging. Through efficient and effective knowledge exchange, we have made our findings known to e2v technologies, including through the Centre Operational Group (COG), containing many high ranked members of the company and the university, through senior e2v engineers that have been seconded to work at the university several days each week, through presentations at collaborative meetings and through less formal communications by telephone and email. This exchange of knowledge and ideas has continued since the start of the grant and now beyond the grant end date. Further collaboration has driven the submission of two research papers on the sensor and the ongoing development of the next stages in taking an optimised CMOS sensor for hyperspectral imaging towards the market place. Through the outputs of this and the proposed future studies, e2v will aim to maintain their market share in Hyperspectral imaging as the market moves forwards, benefitting the UK economy and increasing job creation in the UK. The results of the study have been presented in America at the international SPIE conference on Optics and Photonics, generating interest in the community for further developments in this area. We are currently pursuing opportunities with academics both at the Open University and other institutions towards taking the next steps required to drive the next developments in the detector for optimised performance in Hyperspectral imaging. Hyperspectral imaging through Earth observation covers many of NERC's core science themes. On this note, policy makers and the general public will benefit from the continued developments following this research. The science themes involved are relevant to the everyday life and quality of living of the wider public, such as the understanding and control of pollution or the impact of natural disasters. Further publications and, subject to further funding, future ongoing studies and developments in the detector design and production will continue to ensure the findings can be taken forward for the maximum possible impact. Some findings have led into new designs that are under manufacture and test (not publishable). The most wide-ranging and exciting findings in this research so far are those which will help us to develop the next generation of sensors for Hyperspectral imaging and Earth observation (as evidenced by the latest new CMOS developments within the CEI). The research performed under this grant has enabled us to delve deeply into the current technology through the prototype device. Through experiment and simulation we are now able to propose designs that will dramatically improve the device performance. With future funding we hope to be able to make these designs a reality, and through an extended programme of testing and development, optimise a new range of sensors for the benefit of these vital research areas and NERC science. |
| Sectors | Aerospace, Defence and Marine,Agriculture, Food and Drink,Digital/Communication/Information Technologies (including Software),Education,Electronics,Energy,Environment,Healthcare,Leisure Activities, including Sports, Recreation and Tourism,Manufacturing, including Industrial Biotechology,Culture, Heritage, Museums and Collections,Other |
| Description | The standard devices used for Earth observation via Hyperspectral imaging are Charge Coupled Devices (CCDs). However, CCDs have limitations for these applications that cannot be overcome through improvements to CCD technology itself. These limitations can only be overcome, and hence the scientific data achievable dramatically improved, by moving to a new technology, i.e. CMOS imaging sensors (CIS). CIS have their own issues which are yet to be solved and need to be solved before they can be used for this application. e2v technologies have produced prototype sensors which have the potential to be the future of detector systems for Earth observation and hyperspectral imaging. We have succesfully tested, characterised and optimised the operation of these test devices towards use in future hyperspectral imaging missions. Through our research performed under this grant, detailed in our publications and technical notes, we have been able to feed highly important information back to e2v technologies that will both build upon the successes of the prototype sensor AND enable the development of a sensor that can overcome the shortfalls that our research has uncovered. Our use of experiments coupled with in-depth simulations has allowed us to propose a new sensor design, idealised and optimised towards effective and efficient Hyperspectral imaging and Earth observation. This knowledge exchange will prove crucial in moving forwards with the development of sensors aiming to secure the benefits detailed here on Research Fish and in the grant documentation. We aim to propose further developments through additional grant funding applications in the near future to proceed with the development of such sensors, taking our research outcomes and new designs into the marketplace. These developments will only be possible through further research grant income to support the knowledge exchange, research, design, development and optimisation required. Over the last year, the outcomes from this research have helped to direct our further research into new device designs. Prototype devices have been designed and produced with testing currently underway. The sensors have the potential to revolutionise various areas of the field if successful (although non-disclosure agreements are currently in place). |
| First Year Of Impact | 2014 |
| Sector | Digital/Communication/Information Technologies (including Software),Electronics,Other |
| Impact Types | Economic |
| Description | SPIE conference(s) |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Other audiences |
| Results and Impact | Talks at an international conference, sparking questions and discussion and possible future projects/funding. |
| Year(s) Of Engagement Activity | 2008,2009,2010,2011,2012,2013,2014,2015,2016,2017,2018 |
| Description | e2v Knowledge Exchange meetings |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | Local |
| Primary Audience | Industry/Business |
| Results and Impact | Knowledge Exchange across the wider network of Space Instrumentation at the Open University and e2v technologies. |
| Year(s) Of Engagement Activity | 2006,2007,2008,2009,2010,2011,2012,2013,2014,2015,2016,2017,2018 |