REFRAME: Rethinking the Essence, Flexibility and Reusability of Advanced Model Exploitation
Lead Research Organisation:
University of Bristol
Department Name: Computer Science
Abstract
See the attached case for support.
Planned Impact
See the attached case for support.
People |
ORCID iD |
| Peter Flach (Principal Investigator) | |
| Oliver Ray (Co-Investigator) |
Publications
Hernández-Orallo J
(2013)
ROC curves in cost space
in Machine Learning
Kull M
(2017)
Beyond sigmoids: How to obtain well-calibrated probabilities from binary classifiers with beta calibration
in Electronic Journal of Statistics
Martinez-Plumed F
(2021)
CRISP-DM Twenty Years Later: From Data Mining Processes to Data Science Trajectories
in IEEE Transactions on Knowledge and Data Engineering
Nikolaou N
(2016)
Cost-sensitive boosting algorithms: Do we really need them?
in Machine Learning
| Description | Knowledge discovery aims to learn patterns from given data that are expected to apply to future data. Often there is relevant contextual information that, although it can have a considerable effect on the quality of the results, is rarely taken into account as it is not directly represented in the training data. The project has studied the role of context and context changes in the knowledge discovery process, and developed a range of practical techniques for context-aware machine learning and data mining, including changes in costs, data distribution and others. We have explored how data mining methodologies such as CRISP-DM can be extended to take context change into account. |
| Exploitation Route | The project has resulted in a range of academic publications as well as software and tutorial materials for use of other researchers and practitioners. |
| Sectors | Communities and Social Services/Policy,Creative Economy,Digital/Communication/Information Technologies (including Software),Healthcare |
| URL | http://reframe-d2k.org/ |
| Description | Newton Call |
| Amount | R$ 8,980 (BRL) |
| Organisation | FACEPE |
| Sector | Public |
| Country | Brazil |
| Start | 06/2017 |
| End | 12/2017 |
| Description | CHIST-ERA |
| Organisation | Institute of Genetics and Molecular and Cellular Biology (IGBMC) |
| Country | France |
| Sector | Academic/University |
| PI Contribution | The REFRAME project was awarded under the CHIST-ERA scheme; each partner was funded by their national research council. |
| Collaborator Contribution | All partners contributed to the REFRAME research programme. |
| Impact | See http://reframe-d2k.org/Publications |
| Start Year | 2013 |
| Description | CHIST-ERA |
| Organisation | Polytechnic University of Valencia |
| Country | Spain |
| Sector | Academic/University |
| PI Contribution | The REFRAME project was awarded under the CHIST-ERA scheme; each partner was funded by their national research council. |
| Collaborator Contribution | All partners contributed to the REFRAME research programme. |
| Impact | See http://reframe-d2k.org/Publications |
| Start Year | 2013 |
| Title | Precision-Recall-Gain curve software |
| Description | Precision-Recall analysis abounds in applications of binary classification where true negatives do not add value and hence should not affect assessment of the classifier's performance. Perhaps inspired by the many advantages of receiver operating characteristic (ROC) curves and the area under such curves for accuracy-based performance assessment, many researchers have taken to report Precision-Recall (PR) curves and associated areas as performance metric. We demonstrate in this work that this practice is fraught with difficulties, and show how to fix this by plotting PR curves in a different coordinate system. |
| Type Of Technology | New/Improved Technique/Technology |
| Year Produced | 2015 |
| Impact | This work was selected for a plenary spotlight presentation at NIPS 2015. |
| URL | http://www.cs.bris.ac.uk/~flach/PRGcurves/ |
| Title | REFRAME Github repository |
| Description | This is a Github repository collecting the software outputs of the REFRAME project. |
| Type Of Technology | Software |
| Year Produced | 2016 |
| Open Source License? | Yes |
| Impact | The PRG software was supporting a publication that was selected for a spotlight presentation at NIPS'15. The tutorial repository was used in a tutorial presented at ECML-PKDD'16. |
| URL | https://github.com/REFRAME |
| Description | LMCE |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Other audiences |
| Results and Impact | We organised two scientific workshops on Learning in Multiple Contexts at ECML-PKDD (LMCE) in the years 2014 (Nancy) and Porto (2015). |
| Year(s) Of Engagement Activity | 2014,2015 |
| URL | http://users.dsic.upv.es/~flip/LMCE2015/ |
| Description | MoReBikeS challenge |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Other audiences |
| Results and Impact | This was a data mining challenge organised in the context of the European Machine Learning and Data Mining conference (ECML-PKDD'15) in Porto. The challenge is carried out in the framework of historical bicycle rental data obtained from Valencia, Spain. Bicycles are continuously taken from and returned to rental stations across the city. The data consists of time series describing hourly availability of bikes at each station; information on weather and (local) holidays is also provided. The challenge motivation is based on the fact that, while we may have had the opportunity to learn and tune good models for old stations with historical data, we do not always have the same amount of data for new stations. With that in mind, participants will receive, in addition to limited data for the new stations, a large number of trained models for old stations. The task will be to make predictions (3 hours ahead) with regard to the number of bikes available for these new stations and within the next months. This situation fluctuates considerably depending on the time of year, the station's location, etc. The key point here is that by using models from other stations that have been learnt from data spanning more than one year, better predictions can be made for the new stations. In the end, this challenge aims at promoting the reusability of models rather than retraining a different model again and again each time the context changes. |
| Year(s) Of Engagement Activity | 2015 |
| URL | http://reframe-d2k.org/Challenge |
| Description | Tutorial on Context-Aware Knowledge Discovery |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Postgraduate students |
| Results and Impact | Traditionally, knowledge discovery aims to learn patterns from given data that are expected to apply to future data. Often there is relevant contextual information that, although it can have a considerable effect on the quality of the results, is rarely taken into account as it is not directly represented in the training data. This tutorial aims to elucidate the role of context and context changes in the knowledge discovery process, and to demonstrate how recent research advances in context-aware machine learning and data mining can be put to practical use. The tutorial will cover the main types of context changes, including changes in costs, data distribution and others. Participants will develop basic skills in choosing the appropriate modelling techniques and visualisation tools for the construction, selection, adaptation and understanding of versatile and context-aware models. We will discuss how data mining methodologies such as CRISP-DM can be extended to take context change into account. The tutorial will not only equip the attendees with new technical and methodological knowledge, but also encourage an anticipatory attitude towards context change. |
| Year(s) Of Engagement Activity | 2016 |
| URL | https://github.com/REFRAME/tutorial |