Newton STFC-NARIT; Data Flow and Archiving for Robotic Operations Present and Future

Lead Research Organisation: Liverpool John Moores University
Department Name: Astrophysics Research Institute

Abstract

Data archiving is axiomatically fundamental to the success of the knowledge economy. Where a primary production economy depends primarily on physical labour, there can be no doubt that the ability to efficiently manipulate and represent data to scientists, administrators and lay people is critical to any economy built on the value of human knowledge, analysis and experience.

Our own discussions with Thai researchers explicitly identified dissemination and coordination of information as a major stumbling block to effective and efficient governance and development in their economy. The Thai government collects much useful data but there is insufficient breadth of experience in the available workforce on matters such as database integration and exploitation in order to extract value from those data.

Looking further ahead, NARIT and LJMU share the common goal of collaboratively developing the world's largest and fastest responding robotic telescope which will address a critical capability gap foreseen in the global astronomy community for the 2020-2040 timescale. This new telescope focuses on exploitation of the anticipated new era in time domain astrophysics. For that to succeed, for either partner, UK or Thailand, dynamic and efficient data distribution is absolutely critical rather than simply desirable. Data management and distribution is a key work package for the LT2 project, so the Newton funded project proposed here will enable NARIT to be a developmental partner in this major piece of research infrastructure. Data handling is absolutely fundamental to any modern knowledge economy, so this proposal will develop widely transferable skills for both partners.

In this context, astronomy is purely one specific use case upon which to base training and capacity building in data flow management and archiving. It provides a convenient model and test bed for commercial and governmental data handling because there are no concerns about accidental disclosure of personal or confidential information. It is however global in scale involving collaboration of partners around the World and requires handling of diverse and high precision data sets. For example, storage is required for heterogeneous data sites such as imaging and spectroscopy and these need referencing with derived data products such as photometric catalogues.

The scope of the proposed project incorporates a survey of other observatories' and data servers' capabilities and software deployments, but more importantly, looks beyond software technologies already in use with astronomical databases. The objective is to assess and optimise the use of modern standards and technologies, such as relational vs non-relational databases (MySQL, Postgres vs. NoSql etc.) and multi-tiered software stacks with discrete database management, middleware and user interface layers. Astrophysics is just one implementation example of these highly transferable skills and technologies which have far reaching application globally in any knowledge-based economy.

Planned Impact

Being the core objective of the grant, industrial and economic impact within Thailand is discussed in the "Case for Support". Whilst some of the same arguments could be made, the UK economy is in a very different position on the world stage and thus the targeted impacts are different. The new archive's major societal significance is to the National Schools' Observatory and other public outreach activities. The NSO currently has about 3000 registered schools (primary, secondary and FE), the website attracts more than 1,500,000 visits per year and we have dealt with over 16,000 observing requests in the last year. Our existing science archive is used by the NSO to serve LT data to schools across the UK but the scope and scale of the NSO has grown enormously in the past decade, changing many of the originally foreseen requirements, and we will work closely with them to ensure optimum schools' use is integrated into the software design. To deliver our strategy, collaboration between the engagement and research teams is essential, with mutual benefits.

A related benefit of developing a more streamlined and open archive infrastructure is increasing the global visibility of UK astronomical research both within and outside the academic community. Increased integration with the virtual observatory advertises the availability of UK data and expertise to professional astronomers. Through the Liverpool Telescope, among other observing facilities, the UK has a leadership in robotic observatory operations and time domain astrophysics and increased public availability of these data advertises UK leadership in this highly technical and inspirational field of science.

Publications

10 25 50
 
Description Principal goals were ODA based and were capacity building within Thailand. This short term pilot study was no funded to continue but the Thai partners have moved ahead to continue the development independently. It has therefore acted as a catalyst to capacity building.
Exploitation Route Summaries of the relative merits of different database architectures were published in conference proceedings and are available as the basis for more efficient database design.
Sectors Digital/Communication/Information Technologies (including Software),Government, Democracy and Justice

 
Description STFC Official Development Assistance (ODA) Institutional Award
Amount £70,000 (GBP)
Organisation Science and Technologies Facilities Council (STFC) 
Sector Academic/University
Country United Kingdom
Start 03/2018 
End 12/2018
 
Description Joint UK/Thai data archive design 
Organisation National Astronomical Research Institute Of Thailand
Country Thailand 
Sector Public 
PI Contribution The Liverpool Telescope (LTs) group has provided both specific training and more generalised guidance and leadership of the data archive development program based on our long experience of operating the World's largest common-user robotic observatory. A large part of our routine collaboration meetings and the design documentation produced has been committed to knowledge transfer of both our successful and less successful experiments in data distribution for the specific case of robotic, remote and unsupervised observing modes. During an extended visit to our operations centre in the UK specific training sessions have also been arranged for the Thai operations staff on LT robotic operations, LT data handling and pipeline procedures, automated logging and telemetry analysis and distribution of data to external, remote observers. A joint training seminar was arranged between the UK and Thai software engineers to address issues of data protection compliance in an trans-national data archive. LTs long history of operation also provided a realistic sample of test data on which to base software prototyping experiments. Attended Newton Fund workshop organised by UK embassy in Thailand attended by several Thai research institutes and delivered a joint presentation with the NARIT partners on the project outcomes. I focussed on objectives and successes of the Newton process from a UK partner's perspective while the NARIT speaker addressed the project outcomes themselves.
Collaborator Contribution The Thai partners have performed all the interaction with Thai telescope users, providing us a simple single point of contact for understanding the wider requirements of the Thai community. The Thai partner provided a software engineer with specific experience of user interface programming to supplement the operations and science led experience of the UK team.
Impact The stated goal of the grant was to initiate and foster a joint development program for robotic operations between the two institutions. Deliverables from the first year study were never intended to include operational software. That was scheduled for the second part or the program. Since the development phase of the project was unfunded the only tangible ouputs so far are our internal development documents and software prototypes, all of which are publicly available and open source but without any commitment for long term availability, maintenance or support: https://github.com/NewtonFund/DataArchive, https://github.com/NewtonFund/DataArchive/wiki, https://www.dropbox.com/home/Newton_Archive. However contact between the two telescope operations groups has been initiated and some low level of cooperation may persist.
Start Year 2017
 
Description End of grant workshop. Results and lessons learnt 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Professional Practitioners
Results and Impact Workshop organised in Chiang Mai, Thailand, by UK embassy to mark the end of the first year of the UK/NARIT Newton collaborations and to kick-off the second. Attendees were primarily NARIT but included representatives Thai research and public organisations invited by the embassy to learn about Newton opportunities. Our project presented a joint talk with UK partners concentrating on the impact of Newton scheme and the Thai partner concentrating on the funded research project itself.
Year(s) Of Engagement Activity 2018
 
Description NARIT community input 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Professional Practitioners
Results and Impact During the visit by the UK partners to Thailand a small workshop was arranged involving 12 NARIT astronomers to discuss the diverse requirements of the project. This was a two way communication sharing our experience and soliciting their aspirations for what a new archiving system could provide to the Thai community.
Year(s) Of Engagement Activity 2017