Basic Image Features

This is a project about studying how the brain understands the image that the eye sees. We think that the brain analyzes each bit of the image separately, saying to itself 'that bit's an eye', 'that's an edge', 'that's a bit of shading' and so on. Then it stitches all these bits together so that it sees the whole image as one. We call the first stage (the looking-at-little-bits-of-the-image part) local analysis, and the second stage (the stitching-the-bits-together part) multi-local analysis. Both are important in understanding how vision works, but this project is only about the local part. To help explain what we will do in the project I'm going to describe a couple of activities that you might be asked to do in an art class.Imagine that you were given the task of making mosaic versions of ordinary colour photos; but, to make it a bit harder, you have to choose a limited palette of chip colours before you see the photographs. What colours would it be best to choose? Well, you could always do a reasonable job as long as you had some chips of each of the colours: black, white, grey, red, orange, yellow, green, blue, purple, pink and brown. These 11 colours are called the Basic Colours as everyone in the world, whatever language they speak, agrees that they are the main ones. Now imagine a different, odder task. Again we have to make versions of photos, but there are two differences from the mosaic task. First, the photos are black & white, not colour. Second, rather than making our versions out of little featureless mosaic chips, this time we are going to use something more like jigsaw pieces. These 'jigsaw pieces' don't have the lugs that ordinary pieces have (so we can always fit them together), but they do have a little patch of image detail on them like a regular jigsaw piece. If you had enough different 'jigsaw pieces' you could probably make versions of any ordinary photo. But what if, just like in the mosaic task, you had to choose some limited set of types with which to make versions of any photo? What kinds of 'jigsaw piece' would you need? Well you'd certainly need a line, and an edge; probably a corner and a T junction; maybe two dots close together, or perhaps a little bit of shading like that distinctive pattern you get on the folds of pushed-up shirt sleeves. Remember the mosaic task, and how all that you really needed were the 11 Basic Colours? Well, when we wonder what types of jigsaw piece we need, we are trying to work out the Basic Image Features. No-one knows what they are (though a few like 'edge' and 'corner' can be guessed), nor how many there are (I guess between thirty and one hundred).So what did the mosaic and jigsaw tasks have to do with how the brain understands what the eye sees? The idea that we will test in this project is that when the brain does local analysis of the image, it uses Basic Image Features to label each little patch. These labels are then analyzed (and probably improved) by the bit of the brain that does the global analysis, but we'll leave that for another project.In the project we will find out what the Basic Image Features are, and will program computers so that they can label images with them. We've got four different ways of trying to find the Basic Image Features, and we will try all of them. If they all give the same answer, we'll know that we are on the right track; if not, it may mean our idea is wrong.We came up with the four methods that we will use by thinking about the similarity between the jigsaw (pattern) and mosaic (colour) tasks. We think the similarities between colour and pattern run very deep, but they're not easy to explain as they are to do with how similar the maths of the two things are. Since they are so similar, and since a lot more is understood about colour than pattern, we can raid colour science for ideas to use in pattern science, and that's where our 4 methods come from.