Robust and Sensitive Methods for Non-rigid and Partial 3D model Retrieval
Lead Research Organisation:
CARDIFF UNIVERSITY
Department Name: Computer Science
Abstract
3D models have a broad range of applications in many different areas such as engineering, biology, chemistry, medicine, entertainment and cultural heritage. Many 3D models are available from the Internet and other sources, resulting in a problem of how to effectively and efficiently find required 3D models (i.e., 3D model retrieval). Current research on 3D model retrieval mainly focuses on global rigid 3D model retrieval, and algorithms for solving this problem are not effective for non-rigid and partial 3D model retrieval. Because many 3D models of interest are non-rigid (such as humans, and mechanisms), and because it is often important to consider just parts of a 3D model (e.g. find a model with a particular connector), finding an efficient way to retrieve non-rigid and partial 3D models is a pressing and challenging problem. This project intends to develop robust and sensitive algorithms for non-rigid and partial 3D model retrieval.
A typical shape-based 3D model retrieval algorithm consists of three main steps: model preprocessing, feature/shape descriptor extraction, and feature/shape indexing and matching. This project will investigate all three steps and develop new non-rigid and partial 3D model retrieval algorithms based on novel techniques from other research areas. Set-membership estimation from control theory will be introduced into model preprocessing and feature/shape descriptor extraction. New machine learning methods, such as affinity propagation, manifold learning and ranking, will be explored for extracting features/shape descriptors, and for feature/shape indexing and matching. The N-gram model from natural language processing will be adapted to feature/shape indexing and matching. Other new techniques from image processing and computer vision will be investigated regarding their effectiveness for non-rigid and partial 3D model retrieval.
This project will also consider potential applications of the newly developed techniques. The 3D model retrieval algorithms will be evaluated jointly with Delcam plc with a view to commercial exploitation. A practical non-rigid and partial 3D model search engine will be developed and deployed on the Internet for public use.
A typical shape-based 3D model retrieval algorithm consists of three main steps: model preprocessing, feature/shape descriptor extraction, and feature/shape indexing and matching. This project will investigate all three steps and develop new non-rigid and partial 3D model retrieval algorithms based on novel techniques from other research areas. Set-membership estimation from control theory will be introduced into model preprocessing and feature/shape descriptor extraction. New machine learning methods, such as affinity propagation, manifold learning and ranking, will be explored for extracting features/shape descriptors, and for feature/shape indexing and matching. The N-gram model from natural language processing will be adapted to feature/shape indexing and matching. Other new techniques from image processing and computer vision will be investigated regarding their effectiveness for non-rigid and partial 3D model retrieval.
This project will also consider potential applications of the newly developed techniques. The 3D model retrieval algorithms will be evaluated jointly with Delcam plc with a view to commercial exploitation. A practical non-rigid and partial 3D model search engine will be developed and deployed on the Internet for public use.
Planned Impact
3D geometric models are widely used in many different areas, such as engineering, chemistry, biology, medicine, cultural heritage, and entertainment. The broad availability of 3D models from the Internet and other sources has stimulated interest in the development of effective 3D model search/retrieval techniques. This project aims to develop robust and sensitive algorithms for non-rigid and partial 3D model retrieval. It will be of benefit to both academia and business. In addition, it is also expected to be beneficial to the advance in medical science and cultural heritage protection.
In engineering, computer-aided design (CAD) uses digital models of 3D objects for design. When many models are available, it is inefficient to manually search for required 3D models. This project will develop effective 3D model retrieval algorithms which can help CAD designers to find CAD models quickly. The algorithms can also be combined into existing CAD software packages to enhance their functionality. The project is potentially beneficial to both CAD end users and CAD software developers. We will cooperate with Delcam, one of the world's leading CAD software suppliers, to explore potential applications of our algorithms in their CAD software.
In chemistry and molecular biology, molecules and proteins are modelled as 3D objects, for structural classification and drug design, etc. Currently, structure classification is largely manual work. Given that protein databases are usually large, automatic classification using protein geometric features becomes very important. In classifying different proteins, there is a need for deformable (non-rigid) or partial shape matching. The local feature extraction and matching methods developed in this project are thus potentially usable for protein structure classification. In drug design, it is necessary to search for and compare deformable 3D models within molecular reference databases, so that suitable drug molecular structures can be designed to cure specific diseases. The 3D model search engine developed in this project has the potential to be used for this purpose.
In medical imaging applications, 3D volume data obtained by CT and MRI scans are currently used for diagnosis of organ deformations by matching actual images with medical databases of known deformations. Non-rigid 3D model retrieval algorithms are required since deformations are the prominent feature in these applications.
Many cultural heritage objects, such as old buildings, statues, pottery, and ceramics are digitized as 3D models in order to preserve cultural heritage, and also to make it available to a wider public. 3D model matching can be used to find similar artefacts for research purposes, or to compare classes of objects. For example, given pots from an archaeological site, they can be compared to known examples in a database as a basis for determing their the age or origin.
One of the most important application areas of non-rigid and partial 3D model retrieval is the field of entertainment. Animations in films or computer games need many non-rigid or part models of 3D characters for rendering. These models can be obtained from the Internet and 3D model manufacturers. However, 3D model databases are usually very large, and it is not easy to find specific 3D models from large databases. Our 3D search engines have the potential to save time for entertainment media production.
Besides the above-mentioned applications, 3D model retrieval also has potential applications in virtual geography environments (VGE), 3D spatial terrain, and robotics.
In engineering, computer-aided design (CAD) uses digital models of 3D objects for design. When many models are available, it is inefficient to manually search for required 3D models. This project will develop effective 3D model retrieval algorithms which can help CAD designers to find CAD models quickly. The algorithms can also be combined into existing CAD software packages to enhance their functionality. The project is potentially beneficial to both CAD end users and CAD software developers. We will cooperate with Delcam, one of the world's leading CAD software suppliers, to explore potential applications of our algorithms in their CAD software.
In chemistry and molecular biology, molecules and proteins are modelled as 3D objects, for structural classification and drug design, etc. Currently, structure classification is largely manual work. Given that protein databases are usually large, automatic classification using protein geometric features becomes very important. In classifying different proteins, there is a need for deformable (non-rigid) or partial shape matching. The local feature extraction and matching methods developed in this project are thus potentially usable for protein structure classification. In drug design, it is necessary to search for and compare deformable 3D models within molecular reference databases, so that suitable drug molecular structures can be designed to cure specific diseases. The 3D model search engine developed in this project has the potential to be used for this purpose.
In medical imaging applications, 3D volume data obtained by CT and MRI scans are currently used for diagnosis of organ deformations by matching actual images with medical databases of known deformations. Non-rigid 3D model retrieval algorithms are required since deformations are the prominent feature in these applications.
Many cultural heritage objects, such as old buildings, statues, pottery, and ceramics are digitized as 3D models in order to preserve cultural heritage, and also to make it available to a wider public. 3D model matching can be used to find similar artefacts for research purposes, or to compare classes of objects. For example, given pots from an archaeological site, they can be compared to known examples in a database as a basis for determing their the age or origin.
One of the most important application areas of non-rigid and partial 3D model retrieval is the field of entertainment. Animations in films or computer games need many non-rigid or part models of 3D characters for rendering. These models can be obtained from the Internet and 3D model manufacturers. However, 3D model databases are usually very large, and it is not easy to find specific 3D models from large databases. Our 3D search engines have the potential to save time for entertainment media production.
Besides the above-mentioned applications, 3D model retrieval also has potential applications in virtual geography environments (VGE), 3D spatial terrain, and robotics.
Publications
Ji Z
(2021)
ReliefNet: Fast Bas-relief Generation from 3D Scenes
in Computer-Aided Design
Ji Z
(2014)
Bas-Relief Modeling from Normal Images with Intuitive Styles.
in IEEE transactions on visualization and computer graphics
Ji Z
(2014)
Real-time Bas-Relief Generation from Depth-and-Normal Maps on GPU
in Computer Graphics Forum
Ji Z
(2021)
Normal manipulation for bas-relief modeling
in Graphical Models
Lian Z
(2015)
Non-rigid 3D Shape Retrieval
Pickup D
(2018)
An evaluation of canonical forms for non-rigid 3D shape retrieval
in Graphical Models
Pickup D
(2015)
Canonical Forms for Non-Rigid 3D Shape Retrieval
Pickup D
(2016)
Skeleton-based canonical forms for non-rigid 3D shape retrieval
in Computational Visual Media
Pickup D
(2015)
Euclidean-distance-based canonical forms for non-rigid 3D shape retrieval
in Pattern Recognition
Description | 1) Two benchmarking datasets were created for testing non-rigid 3D shape retrieval algorithms and canonical forms. 2) A linear time complexity method was proposed for computing a canonical form of non-rigid 3D shape, using Euclidean distances between pairs of a small subset of vertices. 3) A skeleton-based canonical form was proposed for non-rigid 3D shape retrieval. This canonical form results in computational speed-up, and reduced distortion of local shape detail. |
Exploitation Route | The researchers in the area of non-rigid 3D shape retrieval can use our benchmark datasets to test their algorithms, and further develop new algorithms that are robust to noise and distortion, and sensitive to tiny feature changes. Researchers in multimedia retrieval area will find new ways of retrieval, i.e., using analogies. In general applications of non-rigid 3D shape retrieval, the proposed linear time complexity method for computing a canonical form of non-rigid 3D shape will help to speed |
Sectors | Chemicals Creative Economy Digital/Communication/Information Technologies (including Software) Manufacturing including Industrial Biotechology Culture Heritage Museums and Collections |
URL | https://www.cs.cf.ac.uk/shaperetrieval/ |
Title | Non-Rigid 3D Human Models |
Description | We have created a new benchmarking dataset for testing non-rigid 3D shape retrieval algorithms, one that is much more challenging than existing datasets. Our dataset features exclusively human models, in a variety of body shapes and poses. |
Type Of Material | Database/Collection of data |
Year Produced | 2014 |
Provided To Others? | Yes |
Impact | 3D models of humans are commonly used within computer graphics and vision, and so the ability to distinguish between body shapes is an important shape retrieval problem. This dataset was submitted and accepted as a track of SHREC'14 - 3D Shape Retrieval Contest 2014. Nine groups from all over the world have submitted the results of a total of 22 different methods which have been tested on our new dataset. We published the testing results at EUROGRAPHICS 2014 - Workshop on 3D Object Retrieval. |
URL | http://www.cs.cf.ac.uk/shaperetrieval/shrec14/ |
Title | Non-Rigid 3D Models for Canonical Form Contest |
Description | This dataset was made by combining a selection of models from two existing databases. The two datasets are the SHREC'11 non-rigid dataset and the SHREC'14 non-rigid humans dataset. |
Type Of Material | Database/Collection of data |
Year Produced | 2015 |
Provided To Others? | Yes |
Impact | This dataset was submitted and accepted as a track of SHREC'15 - 3D Shape Retrieval Contest 2015. Two groups attended the contest and tested on this dataset. We published the testing results at EUROGRAPHICS 2015 - Workshop on 3D Object Retrieval. We have invited more groups from all over the world to test this dataset and submit their results. We are expecting to publish the extended test results to International Journal of Computer Vision or Pattern Recognition. |
URL | http://www.cs.cf.ac.uk/shaperetrieval/shrec15/ |
Description | Collaboration with Hangzhou Dianzi University |
Organisation | Royal Holloway, University of London |
Department | Department of Computer Science |
Country | United Kingdom |
Sector | Academic/University |
PI Contribution | Our research resulted in two Joint papers. |
Collaborator Contribution | Our research resulted in two Joint papers. |
Impact | Z. Ji, W. Ma, and X. Sun (2014), Bas-Relief Modeling from Normal Images with Intuitive Styles, IEEE Transactions on Visualization and Computer Graphics, 20(5): 675-685. Z. Ji, X. Sun, S. Li, and Y. Wang and (2014), Real-time Bas-Relief Generation from Depth-and-Normal Maps on GPU, Computer Graphics Forum, 33(5): 75-83 |
Start Year | 2014 |
Description | Collaboration with Peking University |
Organisation | Peking University |
Country | China |
Sector | Academic/University |
PI Contribution | Our research resulted in two Joint papers. |
Collaborator Contribution | Our research resulted in two Joint papers. |
Impact | Pickup, D., Liu, J., Sun, X., Rosin, P., Martin, R., Cheng, Z., Lian, Z., Nie, S., Jin, L., Shami, G., Sahillioglu, Y., Kavan, L.: An evaluation of canonical forms for non-rigid 3d shape retrieval. Gr. Models 97, 17-29 (2018) Liu, J., Rosin, P.L., Sun, X. et al. Image-driven unsupervised 3D model co-segmentation. Vis Comput 35, 909-920 (2019). https://doi.org/10.1007/s00371-019-01679-6 |
Start Year | 2018 |
Description | Collaboration with SCUT |
Organisation | South China University of Technology |
Country | China |
Sector | Academic/University |
PI Contribution | We submitted a joint proposal to NSFC-RS Exchange Scheme |
Collaborator Contribution | We submitted a joint proposal to NSFC-RS Exchange Scheme |
Impact | Joint proposal |
Start Year | 2015 |