Personalized Exploration of Imagery Database
Lead Research Organisation:
Lancaster University
Department Name: Computing & Communications
Abstract
"I want to see jackets which are stylish, but not too fancy. Say, 70% stylish."
This project aims to develop new techniques which can significantly improve data browsing experience in online shopping, dating, media recommendations, and many other applications.
Two very common ways to explore large collections of imagery items, for instance, in online shopping, are to browse a hierarchy of items and to search with textual keywords. The returned results are browsed in lists, typically ordered by popularity. However, popularity is defined across all users as one homogeneous peoples, and users cannot sort by their own subjective criteria, e.g., by their own personal `style' for clothes; What is `stylish' to one person will be passe to another. Furthermore, there is no way to place items on a continuous scale, where the criteria amount for each item is known, e.g., how stylish a particular piece of clothing is to a user.
Our goal is to develop new techniques which enable users to organize and explore imagery data based on their own subjective criteria at a high semantic level. This is a challenging problem: Many criteria are hard to quantify and a user may not even be able to articulate the criteria.
We face this challenge by observing that even though users may not be able to specify their criteria quantitatively, or even fully describe them, they are still able to communicate their own notions by providing examples, e.g., "this shoe is cooler than that one". Our goal is to build an algorithm that arranges a large corpus of visual data according to these examples. Once built, the arranged data can be browsed with an interface that exploits the learned criteria to navigate the continuous scale.
The key contributions of the proposed research will include 1) exploring different modes of user interaction and elaborate on reflecting the resulting knowledge to 2) a new algorithm that, by breaking the limitations of existing approaches, effectively and efficiently learns from user-provided examples and thereby makes personalized data exploration realistic.
This project aims to develop new techniques which can significantly improve data browsing experience in online shopping, dating, media recommendations, and many other applications.
Two very common ways to explore large collections of imagery items, for instance, in online shopping, are to browse a hierarchy of items and to search with textual keywords. The returned results are browsed in lists, typically ordered by popularity. However, popularity is defined across all users as one homogeneous peoples, and users cannot sort by their own subjective criteria, e.g., by their own personal `style' for clothes; What is `stylish' to one person will be passe to another. Furthermore, there is no way to place items on a continuous scale, where the criteria amount for each item is known, e.g., how stylish a particular piece of clothing is to a user.
Our goal is to develop new techniques which enable users to organize and explore imagery data based on their own subjective criteria at a high semantic level. This is a challenging problem: Many criteria are hard to quantify and a user may not even be able to articulate the criteria.
We face this challenge by observing that even though users may not be able to specify their criteria quantitatively, or even fully describe them, they are still able to communicate their own notions by providing examples, e.g., "this shoe is cooler than that one". Our goal is to build an algorithm that arranges a large corpus of visual data according to these examples. Once built, the arranged data can be browsed with an interface that exploits the learned criteria to navigate the continuous scale.
The key contributions of the proposed research will include 1) exploring different modes of user interaction and elaborate on reflecting the resulting knowledge to 2) a new algorithm that, by breaking the limitations of existing approaches, effectively and efficiently learns from user-provided examples and thereby makes personalized data exploration realistic.
Planned Impact
This project aims to develop new techniques which can significantly improve data browsing experience by enabling users to organize data collections based on their own subjective, semantic-level criteria. If successful, these techniques can be directly used in many applications that use/require data exploration. In particular, online shopping will be the biggest beneficiary of this research. The UK is one of the largest and ever growing markets in online shopping: As of November 2013, online shopping increased 10% over the year 2012 and revenues reached a monthly record of £10.1 billion. Specific application scenarios include
1) Finding the perfect chair for a user's room from thousands of possibilities across different styles, by ranking a small subset of chairs by preference.
2) Finding a tasty wine (in terms of personal preference) by trying a small number of different wines: Even novice users could easily establish their shopping portfolio, without having to gain knowledge of domain-specific keywords such as `Tannin' and `Tartaric Acid'.
This research will therefore, contribute to qualitative and quantitative growth of the online shopping market in the UK by attracting users with a significantly improved experience.
Online shopping is only an example of many data browsing applications. Additional application examples are
- Online dating (£170 million market in the UK): Attractiveness is personal-- ranking a small subset of people would help to tailor the personal matches you received by a personal appearance attractiveness scale.
- Media recommendation (e.g., Netflix, iTunes, Kindle): Ranking a few films, albums, or books in an online library to quickly organize the entire collection by your preference.
Our strategy for realizing such a browsing system is to make advances in machine learning, computer vision, and HCI. In particular, one of key technical contributions of this project will be an improved algorithm for semi-supervised learning. Since semi-supervised learning is nowadays extensively used in diverse areas including data mining, social networks analysis, robotics, and genetics, in the long-term, this project will impact on a much broader range of economic and academic activities which may benefit from these techniques.
Furthermore, our techniques will have a societal impact by helping people save time: if successful, users would no longer have to spend hours hunting for just the right item. Users could sort by their particular criteria, and have a good chance of finding it within a small amount of time. Collectively, this saves people a lot of time, and makes the shopping experience or more generally, the data browsing experience, much more pleasant.
1) Finding the perfect chair for a user's room from thousands of possibilities across different styles, by ranking a small subset of chairs by preference.
2) Finding a tasty wine (in terms of personal preference) by trying a small number of different wines: Even novice users could easily establish their shopping portfolio, without having to gain knowledge of domain-specific keywords such as `Tannin' and `Tartaric Acid'.
This research will therefore, contribute to qualitative and quantitative growth of the online shopping market in the UK by attracting users with a significantly improved experience.
Online shopping is only an example of many data browsing applications. Additional application examples are
- Online dating (£170 million market in the UK): Attractiveness is personal-- ranking a small subset of people would help to tailor the personal matches you received by a personal appearance attractiveness scale.
- Media recommendation (e.g., Netflix, iTunes, Kindle): Ranking a few films, albums, or books in an online library to quickly organize the entire collection by your preference.
Our strategy for realizing such a browsing system is to make advances in machine learning, computer vision, and HCI. In particular, one of key technical contributions of this project will be an improved algorithm for semi-supervised learning. Since semi-supervised learning is nowadays extensively used in diverse areas including data mining, social networks analysis, robotics, and genetics, in the long-term, this project will impact on a much broader range of economic and academic activities which may benefit from these techniques.
Furthermore, our techniques will have a societal impact by helping people save time: if successful, users would no longer have to spend hours hunting for just the right item. Users could sort by their particular criteria, and have a good chance of finding it within a small amount of time. Collectively, this saves people a lot of time, and makes the shopping experience or more generally, the data browsing experience, much more pleasant.
People |
ORCID iD |
| Kwang In Kim (Principal Investigator) |
Publications
Dev K.
(2016)
Improving style similarity metrics of 3D shapes
Kim K
(2015)
Local high-order regularization on data manifolds
Rhodin H
(2015)
Generalizing wave gestures from sparse examples for real-time character control
in ACM Transactions on Graphics
| Description | Please see the Key Findings of EPSRC Grant: EP/M00533X/2. |
| Exploitation Route | Please see the Key Findings of EPSRC Grant: EP/M00533X/2. |
| Sectors | Creative Economy,Digital/Communication/Information Technologies (including Software) |
| URL | https://people.mpi-inf.mpg.de/~kkim/hreg/index.html |
| Description | The progress of this project has been reported to the scientific community at various international conferences including IEEE Conference on Computer Vision and Pattern Recognition (CVPR), International Conference on Computer Vision (ICCV), European Conference on Computer Vision (ECCV), and SIGGRAPH Asia. The software packages and datasets associated with this project have been made publicly available at ------- [1] http://mloss.org/software/view/644/ [2] https://people.mpi-inf.mpg.de/~kkim/hreg/index.html [3] https://people.mpi-inf.mpg.de/~kkim/relreg/index.html [4] https://people.mpi-inf.mpg.de/~kkim/diff/index.html ------- Please see the Key Findings of EPSRC Grant: EP/M00533X/2. |
| Sector | Creative Economy,Digital/Communication/Information Technologies (including Software),Other |
| Title | Context-guided diffusion algorithm on graphs |
| Description | Existing approaches for diffusion on graphs, e.g., for label propagation, are mainly focused on isotropic diffusion, which is induced by the commonly-used graph Laplacian regularizer. This algorithm facilitates anisotropic diffusion on graphs and the corresponding label propagation. The algorithm is obtained by constructing positive definite diffusivity operators on the vector bundles of Riemannian manifolds, and discretising them to diffusivity operators on graphs. The algorithm can be used in semi-supervised learning, spectral clustering, and spectral embedding of graph structured data. |
| Type Of Material | Computer model/algorithm |
| Year Produced | 2016 |
| Provided To Others? | Yes |
| Impact | Our new data analysis framework has been presented at an academic conference: International Conference on Computer Vision 2015. |
| URL | https://people.mpi-inf.mpg.de/~kkim/diff/index.html |
| Title | Explicit relationship-guided regularization algorithm |
| Description | In many machine learning tasks, the structure of the target space of a function holds rich information about the relationships between evaluations of functions on different data points. Existing approaches attempt to exploit this relationship information implicitly by enforcing smoothness on function evaluations only. Our framework explicitly regularizes the relationships between function evaluations: This framework has been instantiated as a new semi-supervised learning algorithm and a dimensionality reduction algorithm that facilitate the analysis of large-scale datasets. |
| Type Of Material | Computer model/algorithm |
| Year Produced | 2015 |
| Provided To Others? | Yes |
| Impact | Our new data analysis framework has been presented at an academic conference: IEEE International Conference on Computer Vision and Pattern Recognition 2015. |
| URL | https://people.mpi-inf.mpg.de/~kkim/relreg/index.html |
| Title | Joint diffusion of graph functions and Laplacians |
| Description | This algorithm is an instantiation of a discrete regularizer on a graph's diffusivity operator. The algorithm is grounded in the theory that regularizing the diffusivity operator corresponds to regularizing the metric on Riemannian manifolds, which further corresponds to regularizing the anisotropic Laplace-Beltrami operator. This new algorithm significantly improves existing semi-supervised learning and ranking algorithms on graphs. |
| Type Of Material | Computer model/algorithm |
| Year Produced | 2016 |
| Provided To Others? | Yes |
| Impact | Our new graph data analysis framework has been presented at an academic conference: European Conference on Computer Vision ECCV 2016. |
| URL | https://people.mpi-inf.mpg.de/~kkim/ldiff/index.html |
| Description | User-centric image organization: collaboration with Brown University and Max Planck Institute for Informatics (Note: grant transferred from EP/M00533X/1) |
| Organisation | Brown University |
| Country | United States |
| Sector | Academic/University |
| PI Contribution | In joint research on user-centric image organization, I contributed by developing new machine learning algorithms tailored for data organization: 1) Our new high-order regularization algorithms do not suffer from the degeneracy of conventional graph Laplacian-type regularization algorithms (see [1,6] Outputs section) and they facilitate the propagation of user-provided data annotations to the entire image database; 2) Our new predictor combination approach enables us to combine multiple heterogeneous predictors made by users benefiting from the sharing of knowledge gained from individual tasks across different data organization problems and users (see [5] Outputs section). |
| Collaborator Contribution | Prof. Tompkin at Brown University is an expert in computational photography and videography, and user-centric contents generation. He contributed by designing interactive systems that enable users to communicate their own data exploration criteria (supported by our new machine learning algorithms). Prof. Christian Theobalt at Max Planck Institute for Informatics who is an expert in human motion analysis, 3D image analysis and synthesis contributed with his significant technical and scientific knowledge in developing a framework that enables users to design their own gesture-based, animated character control interfaces [4]. |
| Impact | This collaboration has led to several important published (plus unpublished so far) outcomes: [1] K. I. Kim, J. Tompkin, H. Pfister, and C. Theobalt, Local high-order regularization on data manifolds, Proc. IEEE Computer Vision and Pattern Recognition, 2015. [2] K. I. Kim, J. Tompkin, H. Pfister, and C. Theobalt, Semi-supervised learning with explicit relationship regularization, Proc. IEEE Computer Vision and Pattern Recognition, 2015. [3] K. I. Kim, J. Tompkin, H. Pfister, and C. Theobalt, Context-guided diffusion for label propagation on graphs, Proc. International Conference on Computer Vision, 2015. [4] H. Rhodin, J. Tompkin, K. I. Kim, E. d. Aguiar, H. Pfister, H.-P. Seidel, and C. Theobalt, Generalizing Wave Gestures from Sparse Examples for Real-time Character Control, ACM Trans. Graphics (Proc. SIGGRAPH), 2015. [5] K. I. Kim, J. tompkin, and C. Richardt, Predictor Combination at Test Time, Proc. International Conference on Computer Vision, 2017. [6] J. tompkin, K. I. Kim, H. Pfister, and C. Theobalt, Criteria Sliders: Learning Continuous Database Criteria via Interactive Ranking, Proc. British Machine Vision Conference, 2017. |
| Start Year | 2015 |
| Description | User-centric image organization: collaboration with Brown University and Max Planck Institute for Informatics (Note: grant transferred from EP/M00533X/1) |
| Organisation | Max Planck Society |
| Department | Max Planck Institute for Informatics |
| Country | Germany |
| Sector | Charity/Non Profit |
| PI Contribution | In joint research on user-centric image organization, I contributed by developing new machine learning algorithms tailored for data organization: 1) Our new high-order regularization algorithms do not suffer from the degeneracy of conventional graph Laplacian-type regularization algorithms (see [1,6] Outputs section) and they facilitate the propagation of user-provided data annotations to the entire image database; 2) Our new predictor combination approach enables us to combine multiple heterogeneous predictors made by users benefiting from the sharing of knowledge gained from individual tasks across different data organization problems and users (see [5] Outputs section). |
| Collaborator Contribution | Prof. Tompkin at Brown University is an expert in computational photography and videography, and user-centric contents generation. He contributed by designing interactive systems that enable users to communicate their own data exploration criteria (supported by our new machine learning algorithms). Prof. Christian Theobalt at Max Planck Institute for Informatics who is an expert in human motion analysis, 3D image analysis and synthesis contributed with his significant technical and scientific knowledge in developing a framework that enables users to design their own gesture-based, animated character control interfaces [4]. |
| Impact | This collaboration has led to several important published (plus unpublished so far) outcomes: [1] K. I. Kim, J. Tompkin, H. Pfister, and C. Theobalt, Local high-order regularization on data manifolds, Proc. IEEE Computer Vision and Pattern Recognition, 2015. [2] K. I. Kim, J. Tompkin, H. Pfister, and C. Theobalt, Semi-supervised learning with explicit relationship regularization, Proc. IEEE Computer Vision and Pattern Recognition, 2015. [3] K. I. Kim, J. Tompkin, H. Pfister, and C. Theobalt, Context-guided diffusion for label propagation on graphs, Proc. International Conference on Computer Vision, 2015. [4] H. Rhodin, J. Tompkin, K. I. Kim, E. d. Aguiar, H. Pfister, H.-P. Seidel, and C. Theobalt, Generalizing Wave Gestures from Sparse Examples for Real-time Character Control, ACM Trans. Graphics (Proc. SIGGRAPH), 2015. [5] K. I. Kim, J. tompkin, and C. Richardt, Predictor Combination at Test Time, Proc. International Conference on Computer Vision, 2017. [6] J. tompkin, K. I. Kim, H. Pfister, and C. Theobalt, Criteria Sliders: Learning Continuous Database Criteria via Interactive Ranking, Proc. British Machine Vision Conference, 2017. |
| Start Year | 2015 |
| Title | Local high-order regularization on data manifolds |
| Description | Our software implements a new regularizer which is globally high order and is also sparse for efficient computation in semi-supervised learning applications. |
| Type Of Technology | Software |
| Year Produced | 2016 |
| Open Source License? | Yes |
| Impact | The software has been released under GPL on the machine learning open source software forum (mloss.org). |
| URL | http://mloss.org/software/search/?searchterm=Local+high+order+regularization&post= |
| Description | Presentation at Dongseo University |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | Local |
| Primary Audience | Undergraduate students |
| Results and Impact | I gave a talk on our semantics-level image exploration work at Dongseo University, Korea as a part of a recently started research collaboration with the Dept. of Digital Contents. |
| Year(s) Of Engagement Activity | 2016 |
| Description | Presentation at Imperial College London |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | Local |
| Primary Audience | Postgraduate students |
| Results and Impact | I was invited to Imperial College London to present our work on user-centric imagery framework: This led to research collaboration with Dr. Tae-Kyun Kim at Imperial Computer Vision & Learning Lab. |
| Year(s) Of Engagement Activity | 2016 |