Colour space homography

For commercial 'imaging' products like the digital camera in your phone, engineers seek simple and elegant solutions to hard problems. You will have noticed that newer cameras take better pictures than older ones. This is in part due to ever advancing hardware: there are faster processors and more pixels. Yet, there has been some important problem solving insights that make it possible to solve problems that were hitherto intractable. A good example of this is the problem of colour casts (one of the foci of this proposal). When you take a picture with a camera the colours that are recorded are dependent on the colour of objects in the scene and (perhaps surprisingly) on the colour of the light. Physically, your white T-shirt is yellowish and bluish when viewed in direct sunlight or when you are in the shadows (because sunlight and shadows are respectively yellowish and bluish). We do not see the colour casts as our vision system factors out the colour of the light and, likewise, digital cameras process images to (much of the time) remove colour casts. But, if you compare the outputs of cameras today with those 15 years ago, the modern era cameras are much better at removing colour casts. Why? because of clever insights that helped engineers build systems which correctly infer the colour of the prevailing light.

This research project begins with a surprising new observation which we believe will help us improve still further imaging products such as colour cameras. We make, for the first time, a deep link between the colours recorded in image - typically three R,G and B numbers that measure the redness, greenness and blueness of a pixel - and the 3D locations of points in the world and how these points correspond to pixel locations in an image. Thinking about geometry, we are all aware that train tracks appear to converge in the distance even although we know they stay the same distance apart (intrinsically, we understand how the 3D world maps to 2D pictures). In this research we begin by showing that the relationship between the colours - same colourful object viewed under different viewing conditions - is exactly the same as the relationship that links different viewpoints - say of the front of a building - in geometry. Mathematically, this relationship is called an homography.

Importantly, now that we have linked the colours you see in the camera to how the 3D world maps to images we can use this observation - we can exploit lessons learnt in the geometric domain - to help us solve some colour imaging problems. First, by exploiting colour space homography, camera manufacturers (and enthusiasts) will be able to more accurately calibrate their cameras and this will further optimize image colour fidelity. Homographies also play an important role the colour cast removal problem and in this proposal we will develop a homography-based algorithm that will advance the state of the art. Outside of photography, we are interested in problems such as colour based medical diagnosis e.g. we will be aiming to improve the automatic identification of skin lesions. We propose that homographies are also the key to solving high level vision tasks including identifying, manipulating and removing shadows from images.

More novelly, we have found that answering the question "how do we find a coloured filter which, when place in front of a camera makes the camera measure light in a way that is similar to the human visual system' also involves solving for a homography. Homographies have a link to problems such as image fusion - fusing 100s of images into a single colour summary - which process images in the so-called derivative domain. Here images are transformed into an edge representation and then these edges are manipulated before an output is reintegrated. Homographies hold the promise of making the reintegration process faster and less prone to introducing artifacts (a well known problem of existing techniques).

Conventional colour correction proceeds by taking an image of a reference target and then mapping the image colours to reference counterparts. We then apply the same correction to pictures of unknown objects. Unfortunately, this calibration method assumes light intensity is the same across the chart and this is rarely the case. Assuming uniformity of light intensity (when it is not present) can result in twice the correction error. Our research proposal starts with the discovery that finding the correct colour transform, in an intensity invariant way, involves solving for a homography. Moreover, relating colours across viewpoints with a homography is in direct analogy to geometric vision where a homography is the correct mathematical tool for relating planar point sets across images. Further, this proposal casts the problems of filter design, illuminant estimation, image retrieval and derivative domain processing in a homographic framework.

This project benefits from the close collaboration from two industry partners. Apple Inc have a particular interest in photographic applications and are interested in the potential of our research to help them attain improved colour fidelity and better illuminant estimation. They are also keenly interested in understanding which parts of images are lit by different lights. Spectral Edge Ltd, a UEA spin out in image fusion, has developed technology for fusing different images in the derivative domain and so are interested in how homographies help in derivative domain processing. One aspect of Spectral Edge's work has already been approximated as an optical process (light passing through a specially designed optical filter). So, there is a synergy with our filter design homography research.

Of course, through our extensive dissemination plans we also plan to publish all our algorithms and data. So, there is an opportunity for wider industry to make use of our research. Detailed in our scientific case is an application to improving the largely colour based automated diagnoses of skin lesions. More widely, colour space homography has the potential to impact of big image data (search) and photo processing amongst other applications.

The research team has a track record of working with industry on colour standards including ISO 17321 (camera characterization). The colour homography work may have application in photography ISO/TC42 and machine vision European Machine Vision Association EMVA 288. The former technical committee will be approached via the Society of Imaging Science and Technology (through Finlayson who is a fellow). Indirectly, through the British Machine Vision Association (Fisher is a fellow) we have links to the EMVA and plan to communicate our work to that organization.

The project provides unrivaled training opportunities for the Research Co-investigator Dr Bastiaan Boom and also the PhD students to be trained along side this grant. Dr Boom will gain expertise in the management side of research and also in supervising research students (both activity supported by targeted courses from UEA's Centre for Staff Education and Development). Dr Boom and the students will have the chance to work with industrial partners at the partner sites. The whole team will have the opportunity to see their research transferred and prototyped in an industrial context.

Regarding outreach to a wider community, this research project begins coincidently at the same time as the EPSRC Network on Biological and Machine Visiion begins its work. Not only is the topic of this network intimately related to the proposed research, one of the founding aims of the network is public engagement. Finlayson is the leader of the 'Appearance: Shadow removal and lightness constancy' theme (which relates strongly to our research proposal). Finlayson will work with the network in delivering workshops and will liaise with interested societies including RPS, IET and IS&T to ensure the widest participation