Ultimate Microsocopy: Wavelength-Limited Resolution Without High Quality Lenses

Lead Research Organisation: University of Sheffield
Department Name: Electronic and Electrical Engineering


At the beginning of the 20th century, scientists discovered how to measure the size and spacing of atoms using a technique called diffraction, which led to a revolution in the understanding of chemistry, biology and solid-state physics. X-rays and electrons behave like waves, but with a wavelength which is much smaller than the spacing between the atoms of a solid. These waves scatter and interfere with one another, producing strong beams coming out of the object at particular angles. By measuring these angles, and knowing the wavelength of the waves, the separation of atoms could be calculated. It was using this method that Watson and Crick determined the structure of DNA in the 1950s. However, diffraction is only useful if the object is a regular lattice structure. In order to look at more complicated atomic structures, scientists have relied on electron or X-ray microscopes. In a standard microscope, a lens is used to produce a magnified image, but the method still relies on the waves that make up the radiation (light, electrons or X-rays) interfering with one another to build up the image. With light, this is experimentally easy, but with very-short wavelength radiation (a fraction of an atomic diameter), the tiniest error in the lens or the experimental apparatus makes the waves interfere incorrectly, ruining the image. For this reason, a typical electron or X-ray microscope image is about one hundred times more blurred than the theoretical limit defined by the wavelength.In this project, we aim to unify the strengths of the above apparently very different techniques to get the best-ever pictures of individual atoms in any structure (which is not necessarily crystalline). Our approach is to use a conventional (relatively bad) X-ray or electron lens to form a patch of moderately-focussed illumination (like burning a hole in a piece of paper with the sun's rays through a magnifying glass). In fact, we do not need a lens at all! Just a moveable aperture put in front of the object of interest will suffice. We then record the intensity of the diffraction pattern which emerges from the other side of the object on a good-quality high-resolution detector, for several positions of the illuminating beam. This data does not look anything like the object, but we have worked out a way of calculating a very good image of the object by a process called 'phase-retrieval'. To make an image of an object we have to know what's called the relative phase (the different arrival times) of the waves that get scattered from it. In diffraction, this information is lost, although some of it is preserved (badly) by a lens. Our data is a complex mixture of diffraction and image data, but the key innovation in this project is that we can use a computer to calculate the phase of the very high resolution data which could never be seen by the lens alone. Other workers in the United States have demonstrated very limited versions of this new approach, but we have a much more sophisticated computational method which eliminates essentially all earlier restrictions.The new method, which has received patent protection, could be implemented on existing electron or X-ray microscopes, greatly enhancing their imaging capability. It is even possible to contemplate a solid-state optical microscope, built into a single chip with no optical elements at all. All the weakness and difficulties and costs of lenses would be replaced by a combination of good quality detectors and computers. Our ultimate aim is to be able to image in 3D directly (using X-rays or electrons) any molecular structure, although this will require a great deal of research. The work put forward in this proposal will build the Basic Technology foundations of this new approach to the ultimate microscope.


10 25 50
publication icon
Clark JN (2014) Continuous scanning mode for ptychography. in Optics letters

publication icon
Godden TM (2014) Ptychographic microscope for three-dimensional imaging. in Optics express

publication icon
Batey DJ (2014) Information multiplexing in ptychography. in Ultramicroscopy

publication icon
Clark JN (2014) Dynamic imaging using ptychography. in Physical review letters

publication icon
Clark JN (2015) Imaging transient melting of a nanocrystal using an X-ray laser. in Proceedings of the National Academy of Sciences of the United States of America

publication icon
Robinson I (2015) My life and the world of crystals in Physica Scripta

publication icon
Huang X (2015) Fly-scan ptychography. in Scientific reports

publication icon
D'Alfonso A (2016) Dose-dependent high-resolution electron ptychography in Journal of Applied Physics

publication icon
Claus D (2019) Diffraction-limited superresolution ptychography in the Rayleigh-Sommerfeld regime. in Journal of the Optical Society of America. A, Optics, image science, and vision

Description Until our research, microscopes that look through objects (transmission microscopes) have used lenses to magnify the object. We have developed a way of making transmission microscopes without using lenses. Instead, we pass radiation (light, X-rays or high speed electrons) through an object and just record how the radiation has been affected by the object using a detector positioned a long way beyond it. What arrives on the detector (called a 'diffraction pattern') doesn't resemble the object at all. However, we have developed a way of processing this intensity pattern in a computer to make an artificial image. This roundabout method turns out to have many advantages over a conventional lens-based microscope. First, the picture we obtain has much more information in it about the object than a normal image: for example, we can see transparent objects, like biological cells, much more clearly and quantitatively. Second, it is very hard to make good lenses for X-rays or electrons. However, these 'high-energy' radiations are the only ones that can see really small objects like individual atoms. In fact, X-ray and electron lenses are so bad that their resolution (the smallest thing they can see) is about 20-100 bigger than the theoretical limit. In other words, our method opens up the opportunity to see the smallest atomic-sized details ever when looking through an object.

Until recently, using just a detector and computer to calculate an image like this was regarded as extremely difficult and could only be done for small objects. The technique we have developed during this project uses two or more (we often use up to a thousand) diffraction patterns which can make images of arbitrarily big objects. The fundamental algorithm which had acted as our original proof-of-principle had quite a few limitations. When we began to use it on real data, all sorts of experimental errors damaged the resulting image. The project consisted of four main parts: developing the method at (1) light, (2) X-ray and (3) electron wavelengths, and (4) developing the computational methods to overcome experimental problems that we encountered at these various wavelengths.

A big hurdle that we overcame was extending the algorithm so that we didn't need to know the shape of the radiation patch illuminating the object: previously a great weakness of the method. We also developed a way of working out exactly where the object had been illuminated: a key parameter which we discovered was almost impossible to control accurately in the electron microscope, and to a lesser extent at X-ray wavelengths. We also developed methods of greatly improving the potential resolution of the technique and a way of handling thick objects which scatter strongly (the original algorithm could only work on thin objects).

We also demonstrated that we could overcome the lens-based resolution limit in electron microscopy by a factor five: a revolutionary result. We also applied the new computational methods to various objects in light and X-ray imaging, for example using it to differentiate between cancerous and healthy cells.
Exploitation Route Many of the activities listed under 'Narrative Impact' are relevant to commercial research laboratories which require very high performance microscopy facilities. Particular areas of activity will include drug discovery (eg using our technology for high-throughput screening); development of high-performance catalysts for use in the oil or automobile industry; the semiconductor industry, say in the context of surface metrology (using light) or 3D structural discovering (using transmission electron microscopy).

A spin-off company formed by the University of Sheffield (Phase Focus Ltd) has acquired and is prosecuting five out of the six patents generated by the project. (The fifth patent is not relevant to Phase Focus - a licensee is being sought). It has so far raised £5.7M private equity funding and is currently engaged in a further funding round. It has 15 employees and is planned to expand quickly.
The company has developed a product which is now being sold into a niche but profitable market - the contact lens industry. It turns out that the very accurate phase image produced by the technique is uniquely capable of measuring with extreme accuracy soft contact lens profiles in their native solution (which of course are transparent in normal transmission) over a very large field of view.

The company has also developed a 'demonstrator' general biological phase imaging microscope, one of which is installed and is working very successfully in a leading UK research laboratories. This will be marketed shortly.

The company is working in collaboration with a third party to develop an 'add-on' device which will improve the resolution of a standard transmission electron microscope (TEM) by a significant factor, and will facilitate sensitive phase imaging (say for whole cell tomography, magnetic field imaging or inner potential mapping in semiconductors).

Although dedicated microscopy (light, X-ray and electron) devices will be sold mostly into the academic research environment, Phase Focus is exploring direct sales (or licensing agreements) in several industrial markets (most being confidential at the time of writing): the contact lens industry is the first such market being exploited.
Transmission microscopy is a universal enabling technology that allows for the qualitative and quantitative imaging of the microscopic structure all types of objects - biological, mineral and man-made. Our research has greatly enhanced the power of microscopy at all scale sizes from 10s of microns down to the sub-nanometre. It has defined a new route to much higher resolution at atomic-scale wavelengths; it has provided a new image signal (absolute phase contrast) which has previously been only approximately accessible by other contrast techniques (holography, differential interference microscopy and Zernike contrast), which is uniquely capable of delivering quantitative images without distortion; our latest developments have shown it can deliver 3D imaging while at the same time removing multiple scattering effects, thus opening up a new route to imaging thick objects.

Imaging using the high-resolution and sensitive phase image in electron microscopy will have applications in: all areas of research currently undertaken using (the very difficult) technique of electron holography - for example, the measurement of microscopic magnetic fields in materials used for magnetic storage media, the measurement of the inner potential (eg the doping profile of semiconductors); studies into porous materials; catalysis particles; whole cell tomography; metallurgy; soft matter materials (eg characterisation of polymer films used for inexpensive solar cells); ceramics such as hard and tribological coating technology; structure of sol-gels; and all areas of semiconductor metrology.

Similar applications will be found at X-ray wavelengths, with the added advantage of imaging thick objects (relative to the electron microscope). The phase sensitivity of the technique is particularly useful in the field of imaging and tomography of thick biological objects such as bone.

As well as improving 2D imaging of all types of biological structure, the novel 3D technique alone at optical wavelengths should have applications in: 3D developmental biology, 3D drug delivery and diffusion, cancer research (specifically early 3D growth of tumorous cell clusters, their blood supply and the effect chemotherapy), 3D chemical signally in realistic live structures, tissue engineering (especially development of effective 3D artificial scaffold materials), 3D dynamics of nano-particle drug delivery mechanisms, 3D chromosome topology during division, live 3D brain cell topology and growth, the study of artificially induced nerve repair.

In view of the above, the final benefits will accrue to a very wide range of societal outputs: improved health via new drugs developed using cell assays; less expensive or more efficient manufactured objects (ranging from cars to aircraft to domestic appliances) via more controlled and/or better engineered and/or less expensive materials, including in the field of low-carbon energy sources.
Sectors Electronics,Healthcare,Pharmaceuticals and Medical Biotechnology,Other

URL http://www.pi-phi.org
Description A principal route to market has been via a University of Sheffield spin-out company (Phase Focus Ltd). 5 patent families arising directly from this work have been sold to Phase Focus (two of these are not yet published). The main company product is a visible-light microscope specifically for imaging live cells in culture over hours or days for application in the Pharma and diagnostic sectors. It also produces the most accurate contact lens metrology device in the world (employing directly methods developed in this project), used by most of the leading contact lens manufacturers. It aims also has plans to sell add-on systems to improve performance of electron microscopes. More generally, our research is being employed at many synchrotrons around the world. The inverse 'lensless' algorithm methods developed in the program are used for very high resolution X-ray imaging and tomography of a very wide variety of structures in materials science and biology. It is being used to image the next generation of semiconductor technology (17nm linewidth) using XUV wavelengths. A Bragg-reflection version of our technology is being used to measure strain in semiconductors - critical for band-gap engineering. Discoveries using our generic enabling imaging technology will have very wide opportunities for societal and economic impact.
First Year Of Impact 2012
Sector Electronics,Healthcare,Manufacturing, including Industrial Biotechology
Impact Types Societal,Economic

Description Linac Coherent Light Source, Chair of Science Advisory Committee
Geographic Reach Multiple continents/international 
Policy Influence Type Participation in a advisory committee
Impact Facility improvement and development for benefit of users.
URL https://lcls.slac.stanford.edu/
Description London Centre for Nanotechnology
Amount £1,436,518 (GBP)
Funding ID EP/I022562/1 
Organisation Engineering and Physical Sciences Research Council (EPSRC) 
Sector Public
Country United Kingdom
Start 09/2011 
End 08/2016
Description London Centre for Nanotechnology
Amount £1,436,518 (GBP)
Funding ID EP/I022562/1 
Organisation Engineering and Physical Sciences Research Council (EPSRC) 
Sector Public
Country United Kingdom
Start 09/2011 
End 08/2016
Description Diamond 
Organisation Diamond Light Source
Country United Kingdom 
Sector Private 
PI Contribution Executed ptychography for the first time on I13. Commissioning users of I13. Set up computer routines for collecting ptychographic data.
Collaborator Contribution Provision of Beamline at a 3rd generation synchrotron light source.
Impact Two papers: Reciprocal-space up-sampling from real-space over sampling in x-ray ptychography Sampling in x-ray ptychography
Start Year 2011
Description Phase Focus 
Organisation Phase Focus
Country United Kingdom 
Sector Private 
PI Contribution Provided expertise on certain types of inversion methods for process lensless imaging data.
Collaborator Contribution Software for interfacing with equipment in our laboratory, data obtained from equipment we do not have access to.
Impact Papers: An annealing algorithm to correct positioning errors in ptychography Ptychographic transmission microscopy in three dimensions using a multi-slice approach Ptychographic electron microscopy using high-angle dark-field scattering for sub-nanometre resolution imaging Superresolution imaging via ptychography Ptychographic microscope for three-dimensional imaging
Start Year 2007
Title Calibration of a probe in ptychography 
Description A method of calibrating the flux of incident radiation in ptychography, which is a lensless imaging method. 
IP Reference WO2012001397 
Protection Patent application published
Year Protection Granted 2011
Licensed Yes
Impact Upon sale to Phase Focus Ltd, this patent is being used in a number of commercial products associated with contact lens metrology and live cell imaging.
Title Improvements in providing image data 
Description A method of reconstructing diffraction data which lies outside the size of the detector. This is for use in ptychography, a type of lensless imaging technique. 
IP Reference WO2012073043 
Protection Patent application published
Year Protection Granted 2010
Licensed Yes
Impact This patent was sold to Phase Focus Ltd, which develops products relating to various forms of phase-retrieval microscopy
Title Provision of Image Data 
Description A method of recovering an arbitrary illumination function ptychography, which is a lensless imaging method 
IP Reference EP2410353 
Protection Patent application published
Year Protection Granted 2009
Licensed Yes
Impact Upon sale to Phase Focus Ltd, this method is being used in all its products, including those used for contact lens metrology and live cell imaging.
Title Duos Camera System 
Description One of the most significant data-acquisition obstacles in all diffractive imaging is the high dynamic range present in the data. In the particular, optical geometry utilised under this project, a central, bright disc of illumination is surrounded by relatively low-intensity dark-field diffracted data. Multiple-exposure methods increase the total experimental time in a potentially unacceptable manner and, without abeam-stop, the central disc can exhibit blooming on conventional slow-readout CCDs, resulting in serious loss of data. CMOS Active Pixel Sensors (APSs) provide a high-speed imaging capability with much reduced blooming artefacts when compared to CCDs and have the potential for multiple Region of-Interest (ROI) readout. Such sensors may solve some of the fundamental problems encountered with conventional detectors used for diffractive imaging. The project supported the building of a custom detector, which may be used in both X-ray and Electron configurations, to acquire data in the diffraction plane. The X-ray detector has been successfully commissioned on the new beamline I16 and now may be used for routine experiments. The EM detector has been installed on the new R005 Electron Microscope at Sheffield and was commissioned. During the commissioning some modifications were found to required to enhance the systems flexibility, these have been made and the camera is now ready for reinstallation and continuing use. 
Type Of Technology Physical Model/Kit 
Year Produced 2011 
Impact The detector was used within the project itself, but being specific to the particular electron diffraction problem it resolved, it has not gone on to create further impact. 
Company Name Phase Focus Ltd 
Description Development, manufacture and sale of ptychographical microscopes, including a contact lens profilometer, a general biological imaging light microscope with very high contrast phase imaging capability, and a cell life cycle monitoring system. Other products include a generic computer system for adding on to electron, EUV or X-ray microscopes. A contact lens metrology service is also offered. 
Year Established 2006 
Impact The contact lens industry employs the Phase Focus metrology microscope to compare various contact lens designs. This has revealed that many lenses do not comply with oxygen permeability standards. The Phase Focus Livecyte system, which can track many hundreds of cells reproducing and moving in vitro has, for example, demonstrated that certain cancer drug treatments are not as efficacious as previously thought, and can even exacerbate cell motility.
Website http://www.phasefocus.com
Description Bragg Centenary Celebration (Cambridge Philosophical Society) 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Public/other audiences
Results and Impact Talk given at the Bragg Cenenary Celebration by the PI on the subject matter of this programme.

11 January 2013.

A celebration of the first report of Bragg's Law, upon which all solid state physics and much of molecular protein structural discovery is based. About 150 attendees, open to the general public. The PI gave one of 8 talks at the meeting.

No explicit impacts other than the education of the public
Year(s) Of Engagement Activity 2013
Description Chair of Royal Society Conference "Real and reciprocal space X-ray imaging", 2013 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact A double event, supported as part of the Royal Society scientific meetings, was organized in February 2013 in London and at Chicheley Hall in Buckinghamshire by Dr A. Olivo and Prof. I. Robinson. The theme that joined the two events was the use of X-ray phase in novel imaging approaches, as opposed to conventional methods based on X-ray attenuation. The event in London, led by Olivo, addressed the main roadblocks that X-ray phase contrast imaging (XPCI) is encountering in terms of commercial translation, for clinical and industrial applications. The main driver behind this is the development of new approaches that enable XPCI, traditionally a synchrotron method, to be performed with conventional laboratory sources, thus opening the way to its deployment in clinics and industrial settings. The satellite meeting at Chicheley Hall, led by Robinson, focused on the new scientific developments that have recently emerged at specialized facilities such as third-generation synchrotrons and free-electron lasers, which enable the direct measurement of the phase shift induced by a sample from intensity measurements, typically in the far field. The two events were therefore highly complementary, in terms of covering both the more applied/translational and the blue-sky aspects of the use of phase in X-ray research. optics, image processing

Since Roentgen's discovery well over a century ago, X-ray imaging has been based on attenuation. While this is an effective and reliable approach, it is known to have limitations whenever objects with similar attenuation characteristics have to be distinguished, as this leads to poor image contrast. One possible solution is to generate image contrast by exploiting phase changes in the X-ray wavefront rather than attenuation: this approach is called X-ray phase contrast imaging (XPCI). Its main advantage is that the term responsible for phase effects (the unit decrement of the real part of the refractive index) is much larger than the imaginary part, responsible for attenuation. In many practical cases, this difference can be up to 1000-fold. Hence, albeit that it can be difficult to detect, phase is intrinsically a much stronger effect and can therefore lead to much higher image contrast if appropriately exploited.

The first X-ray phase contrast image dates back to 1965 and was acquired by Bonse & Hart [1] with a crystal interferometer, which has been bearing their names since. We were fortunate enough to have Prof. Hart attending our conference. Following a significant attempt by Ando & Hosoya [2] in 1972, the idea of using a crystal-based X-ray interferometer was picked up again in the 1990s, in particular by Momose's group. XPCI exploded in the mid-1990s, and, to the best of our knowledge, the first of a long series of papers published from 1995 onwards is Momose's [3], published on 1 January 1995, on results already presented in the previous year. Momose and his group continued to explore medical and biological applications of XPCI implemented with the Bonse-Hart interferometer, as reported, for example, in their 1996 Nature Medicine paper [4].

Almost simultaneously, researchers started to explore a different approach to the use of crystals in XPCI-namely the use of a perfect crystal as an 'angular analyser'. It was in fact observed that the faint distortions in the wavefront caused by phase changes translate directly into slight changes in the X-ray direction (X-ray refraction), which could therefore be picked up by a system with sufficient angular sensitivity like a perfect crystal. The fact that the refraction angle is proportional to the gradient of the phase shift gave rise to the name 'differential' XPCI, which is common to other methods listed below, such as edge-illumination and Talbot interferometry. The most famous paper pioneering this approach is the 1995 Nature letter by Davis et al. [5]. It has to be said, however, that a few known examples exist that pre-date that deservedly famous paper, where effectively the same method had been used to address specific problems, and the results were published in journals with relatively limited diffusion [6,7]. Another paper on the crystal method appeared later in the same year [8] and the method continued to be explored later, leading to some of the first simplified phase retrieval algorithms [9] and extraction of the first quantitative X-ray 'dark-field' images [10-12] (qualitative dark-field images had already been presented in [5]).

Still in 1995, a much simpler implementation, based on free-space propagation, was proposed by Snigirev et al. [13]. Through this approach, phase contrast images are acquired without the need for any additional optical element, simply by increasing the distance between sample and detector. If the source possesses enough coherence, this is sufficient to detect phase-induced interference fringes at the boundaries of the imaged objects. In the following year, Wilkins et al. [14] demonstrated that this approach works also with polychromatic radiation, something that was impossible to observe before as the use of a crystal automatically renders the beam monochromatic. Interestingly, similar images had been obtained before, but the substantially improved image contrast had not been interpreted on the basis of phase effects (e.g. [15]). Owing to its simplicity of implementation, the free-space propagation technique is still widely used, and it is the only one to date to have reached the in vivo stage for human patients [16].

A method that combines the angular selectivity principle exploited with crystal 'analysers' and the simplicity of free-space propagation, called edge-illumination XPCI, was developed in the late 1990s [17]. As the name suggests, it performs a fine selection on X-ray direction by illuminating only the physical edge of the detector pixels, and by doing so it eliminates the need for a perfect crystal, opening the way to polychromatic and divergent X-ray beams. This was later exploited to adapt the method for use with conventional X-ray sources [18], which also led to the first quantitative phase retrieval performed with incoherent illumination of the sample [19].

In the early 2000s, a further approach was introduced, based on the adaptation to X-rays of Talbot and Talbot-Lau interferometers, well-known methods among the optics community. This adaptation was made possible by novel nanofabrication techniques, which enabled the development of gratings with a pitch sufficiently small to produce Talbot 'self-images' [20] at X-ray wavelengths when coherently illuminated. The approach was first demonstrated in the 'Talbot' set-up by David et al. [21] and by Momose's group shortly afterwards [22]; a few years later, Pfeiffer et al. [23] demonstrated that the Talbot-Lau arrangement could also be implemented with X-rays, opening the way to the use of laboratory sources. In a further study, it was shown that the same set-up could also be used to obtain quantitative dark-field images [24], along the lines of what was previously done with analyser crystals [10-12]. To extract phase and dark-field signals, a technique called phase stepping is employed, in which one of the two (Talbot) or three (Talbot-Lau) gratings is scanned with respect to the others, and individual images are acquired at each scanning position. Later on, Wen et al. [25] developed a similar approach, which, however, employed a single grating, and, instead of using a phase-stepping method to isolate the different signals, did so by applying a Fourier-based analysis method to a single frame.

Quite remarkably, most of the leading figures behind the above developments agreed to participate in the conference, which meant that most of the above topics were covered in the various talks. Another interesting aspect that emerged is that the field is continuing to evolve, with new examples of implementation already being presented at this same meeting (e.g. Wen et al. [26] realized gratings with nanometric pitch which led to a hybrid between Talbot and Bonse-Hart interferometry), new implementation schemes (e.g. [27]), use of materials as common as paper to replace gratings or analysers [28] and much more. This extreme liveliness indicates a prosperous future for the area, both in terms of new methods and new applications, and definitely looks encouraging in terms of ultimately reaching the 'translation into mainstream application' goal which inspired this meeting.
Year(s) Of Engagement Activity 2013
URL http://rsta.royalsocietypublishing.org/content/372/2010/20130359
Description Invited talks at conferences: 20 per year 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Postgraduate students
Results and Impact Invitation accepted to travel to international conference, give invited talk and discuss with participants at length.
Year(s) Of Engagement Activity 2006,2007,2008,2009,2010,2011,2012,2013,2014,2015,2016,2017
URL http://www.ucl.ac.uk/~ucapikr/talk16.htm