Verifiable Autonomy

Lead Research Organisation: University of Liverpool
Department Name: Computer Science

Abstract

Autonomy is surely a core theme of technology in the 21st century. Within 20 years, we expect to see fully autonomous vehicles, aircraft, robots, devices, swarms, and software, all of which will (and must) be able to make their own decisions without direct human intervention. The economic implications are enormous: for example, the global civil unmanned air- vehicle (UAV) market has been estimated to be £6B over the next 10 years, while the world-wide market for robotic systems is expected to exceed $50B by 2025.

This potential is both exciting and frightening. Exciting, in that this technology can allow us to develop systems and tackle tasks well beyond current possibilities. Frightening in that the control of these systems is now taken away from us. How do we know that they will work? How do we know that they are safe? And how can we trust them? All of these are impossible questions for current technology. We cannot say that such systems are safe, will not deliberately try to injure humans, and will always try their best to keep humans safe. Without such guarantees, these new technologies will neither be allowed by regulators nor accepted by the public.

Imagine that we have a generic architecture for autonomous systems such that the choices the system makes can be guaranteed? And these guarantees are backed by strong mathematical proof? If we have such an architecture, upon which our autonomous systems (be they robots, vehicles, or software) can be based, then we can indeed guarantee that our systems never intentionally act dangerously, will endeavour to be safe, and will - as far as possible - act in an ethical and trustworthy way. It is important to note that this is separate from the problem of how accurately the system understands its environment. Due to inaccuracy in modelling the real world, we cannot say that a system will be absolutely safe or will definitely achieve something; instead we can say that it tries to be safe and decides to carry out a taskto its best ability. This distinction is crucial: we can only prove that the system never decides to do the wrong thing, we cannot guarantee that accidents will never happen. Consequently, we also need to make an autonomous system judge the quality of its understanding and require it to act taking this into account. We should also verify, by our methods, that the system's choices do not exacerbate any potential safety problems.

Our hypothesis is that by identifying and separating out the high-level decision-making component within autonomous systems, and providing comprehensive formal verification techniques for this, we can indeed directly tackle questions of safety, ethics, legality and reliability. In this project, we build on internationally leading work on agent verification (Fisher), control and learning (Veres), safety and ethics (Winfield), and practical autonomous systems (Veres, Winfield) to advance the underlying verification techniques and so develop a framework allowing us to tackle questions such as the above. In developing autonomous systems for complex and unknown environments, being able to answer such questions is crucial.

Planned Impact

Within 20 years, we expect to see fully autonomous vehicles, aircraft, robots, devices, swarms, and software, all of which will (and must) be able to make their own decisions without direct human intervention. This project will be relevant across many of these applications, from autonomous vehicles, such as driver-less cars and unmanned air vehicles, through to robotics, both in the home and in wider society. Since the control of autonomous systems is increasingly taken away from humans: how can we be sure that they will work; how can we be confident that they are safe; and how can we trust them? All of these are impossible questions for current technology. We cannot say that such systems are safe, will not deliberately try to injure humans, and will always try their best to keep humans safe. Without such guarantees, these new technologies will neither be allowed by regulators nor accepted by the public.

Consequently, the core problem with this technology is being able to effectively verify the inherent autonomy. This is the focus of our project and its potential impact reaches across all those in academia, industry, and government involved in the development, social acceptance, and legality of truly autonomous systems.

Specifically in WP6 and WP7 we have outlined our initial network of collaborating companies, covering Robotic Assistants (Elumotion) and Automotive (MIRA), later with (if sufficient progress) Nuclear Robotics (NNL), and potentially in Underwater (Thales), and Space Exploration (European Space Agency). However, we intend to broaden the industrial aspect and promote and expand the companies aware of, and developing, "verifiable autonomous systems". This promotion will build on our existing networks, for example the industrial members of the Autonomous and Intelligent Systems Partnership; industrial members of the Autonomous Systems National Technical Committee; industrial members of the British Automation and Robot Association; and industrial participants with the Virtual Engineering Centre.

We aim to engage closely with governmental agencies, professional societies, European and international institutions to contribute to the public debate on issues influenced by autonomous systems, leveraging and deepening our existing memberships and influence in such bodies. We aim to develop a suitable activity within one of the relevant learned societies to increase engagement with the scientific and policy communities.

We have already seen, particularly through Winfield's work on public engagement, how autonomous systems (typically, Robotics) can spark the public's imagination. We will build on this, expand to other forms of autonomous systems (UAVs and driver-less cars are natural possibilities), and target public engagement events such as the Cheltenham Science Festival and the British Science Festival (Winfield has already been involved with the former) later in the project.

In summary, through deeper engagement later in the project we aim to impact upon broader industry, and both knowledge transfer and business development branches of government. The team, together with their respective universities, have effective mechanisms for dealing with the media, with access to press offices who regularly publicise work nationally and internationally to non-academic audiences.

Publications

10 25 50

publication icon
Slavkovik M (2014) An abstract formal basis for digital crowds in Distributed and Parallel Databases

publication icon
Kamali M (2017) Formal verification of autonomous vehicle platooning in Science of Computer Programming

publication icon
Fisher M (2015) Misplaced Trust? in Engineering & Technology Reference

publication icon
Fernandes L (2017) A Rational Agent Controlling an Autonomous Vehicle: Implementation and Formal Verification in Electronic Proceedings in Theoretical Computer Science

publication icon
Dignum V (2018) Ethics by Design

publication icon
Dennis L (2018) Two-stage agent program verification in Journal of Logic and Computation

 
Description Developed techniques for verifying key aspects of autonomous robotic systems, as exemplified through two distinct (and very different routes): robot ethics and autonomous vehicle convoys.
Exploitation Route Further research and use in future standards.
Sectors Digital/Communication/Information Technologies (including Software),Healthcare,Transport

URL http://wordpress.csc.liv.ac.uk/va/
 
Description Britis Standards Institution: AMT/10 Committee on Robot Safety and Robot Ethics
Geographic Reach National 
Policy Influence Type Membership of a guideline committee
 
Description British Standards Institution: Artificial Intelligence standards committee [ART/1]
Geographic Reach National 
Policy Influence Type Membership of a guideline committee
URL https://standardsdevelopment.bsigroup.com/committees/50281655
 
Description Membership of International Standards Committee (IEEE P7009 Standard for Fail-Safe Design of Autonomous and Semi-Autonomous Systems)
Geographic Reach Multiple continents/international 
Policy Influence Type Membership of a guideline committee
URL https://standards.ieee.org/develop/project/7009.html
 
Description Membership of International Standards Committee (P7001 Standard for Transparency of Autonomous Systems)
Geographic Reach Multiple continents/international 
Policy Influence Type Membership of a guideline committee
URL https://standards.ieee.org/develop/project/7001.html
 
Description Chair in Emerging Technologies
Amount £129,999 (GBP)
Funding ID CiET 1718/6 
Organisation Royal Academy of Engineering 
Sector Learned Society
Country United Kingdom
Start 03/2019 
End 02/2028
 
Description Network on the Verification and Validation of Autonomous Systems
Amount £107,725 (GBP)
Funding ID EP/M027309/1 
Organisation Engineering and Physical Sciences Research Council (EPSRC) 
Sector Academic/University
Country United Kingdom
Start 08/2015 
End 08/2018
 
Description Software development support for Lego Rovers
Amount £15,816 (GBP)
Organisation University of Liverpool 
Sector Academic/University
Country United Kingdom
Start 11/2016 
End 03/2017
 
Title Model Checking Agent Programming Languages 
Description An Open Source toolkit for prototyping BDI agent programming languages and formally verifying programs written in these languages via an interface to the JavaPathfinder model checker. 
Type Of Material Technology assay or reagent 
Year Produced 2012 
Provided To Others? Yes  
Impact The tool has been used in other research projects and has had impact on external organisations looking at the verification of autonomous systems. 
URL https://sourceforge.net/projects/mcapl/
 
Title The MCAPL Framework 2017 
Description The MCAPL framework is a suite of tools for building interpreters for agent programming languages and model checking programs executing in those interpreters. It consists of the AIL toolkit for building interpreters for rational agent programming languages (BDI languages) and the AJPF model checker. AJPF extends the JavaPathfinder model checker to prove LTL properties of BDI agents. This distribution also contains a number of programming languages implemented in the AIL. Chief among these are Gwendolen, the EASS variant of Gwendolen that can be used to program hybrid autonomous systems and GOAL. This is the 2017 release of the framework. 
Type Of Technology Software 
Year Produced 2017 
Open Source License? Yes  
Impact The Framework underpins a number of publications and results relating to the work on Verifiable Autonomy. 
URL https://sourceforge.net/projects/mcapl/
 
Description Agent Verification workshop 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Postgraduate students
Results and Impact The aim of the workshop is to provide an overview of the field of Agent Verification and associated topics primarily within the UK. We hope this will be of particular value to PhD students in the area but expect it will also provide a useful networking opportunity for more established people. The format of the workshop will be a series of invited talks with the aim of providing good coverage of the research area followed by a panel session and discussion on current research challenges.
Year(s) Of Engagement Activity 2015
URL http://cgi.csc.liv.ac.uk/~maryam/AVWorkshop15.html
 
Description Dangerous Science Summer School (Ethics and Autonomous Systems) 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Other audiences
Results and Impact A debate-style event presenting issues surrounding ethics and artificial intelligence/autonomous systems. This was presented at the University of Lliverpool's Dangerous Science Summer School to groups of teenagers selected as part of the university's widening participation agenda. The teenagers engaged in a lively debate about various ethical issues associated with AI/Autonomous Systems.
Year(s) Of Engagement Activity 2017
 
Description Invited talk at Verification of Autonomous Systems Working Group, Washigton DC, USA [Nov 2018] 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Invited talk at Verification of Autonomous Systems Working Group, Washigton DC, USA [Nov 2018]
Year(s) Of Engagement Activity 2018
URL http://www.robotistry.org/vaswg/nist2018/nist2018.html
 
Description Lego Rovers a Great Hollingworth Academy 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Schools
Results and Impact A workshop on Mars Rovers and AI Programming was delivered to a 5 classes as part of a Year 9 enrichment day.
Year(s) Of Engagement Activity 2018
 
Description Lego Rovers at Aintree Davenhill 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Schools
Results and Impact Four robot programming workshops delivered to primary school children
Year(s) Of Engagement Activity 2018
URL http://legorovers.csc.liv.ac.uk
 
Description Lego Rovers at Great Ashton STEM Academy 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Schools
Results and Impact A workshop on Mars Rovers and AI Programming was delivered to a STEM Club.
Year(s) Of Engagement Activity 2017
URL http://legorovers.csc.liv.ac.uk
 
Description Lego Rovers at Green Park School 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Schools
Results and Impact Five robot programming workshops delivered to primary school children.
Year(s) Of Engagement Activity 2018
URL http://legorovers.csc.liv.ac.uk
 
Description Lego Rovers at Holy Rosary Primary 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Schools
Results and Impact Two Lego Rovers workshops delivered to Year 5 children.
Year(s) Of Engagement Activity 2018
URL http://legorovers.csc.liv.ac.uk
 
Description Lego Rovers at Hudson Primary 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Schools
Results and Impact A workshop on Mars Rovers and AI Programming was delivered to three classes of Primary School children.
Year(s) Of Engagement Activity 2018
URL http://legorovers.csc.liv.ac.uk
 
Description Lego Rovers at LIPA Primary 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Schools
Results and Impact A workshop on Mars Rovers and AI Programming was delivered to two classes of Primary School children.
Year(s) Of Engagement Activity 2018
URL http://legorovers.csc.liv.ac.uk
 
Description Lego Rovers at Lydiate Primary 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Schools
Results and Impact Three Lego Rovers workshops delivered to primary children.
Year(s) Of Engagement Activity 2018
URL http://legorovers.csc.liv.ac.uk
 
Description Lego Rovers at Melland High School 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Schools
Results and Impact Lego Rover workshop delivered to a mixed group of SEN children.
Year(s) Of Engagement Activity 2018
URL http://legorovers.csc.liv.ac.uk
 
Description Lego Rovers at Melling Primary 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Schools
Results and Impact Four workshops run for primary children in Key Stage 2. Sparking questions and discussion afterwards.
Year(s) Of Engagement Activity 2018
URL http://legorovers.csc.liv.ac.uk
 
Description Lego Rovers at Speke Hall 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Public/other audiences
Results and Impact 9 lego robot programming workshops were run for families over three days. This was to tie in with the National Trusts Women and Power theme, and Speke Hall's Vigorous Victorians theme which featured Ada Lovelace. The workshops were well received with most families reporting that it got them to discuss issues around robots and artificial intelligence with their family and friends.
Year(s) Of Engagement Activity 2018
 
Description Lego Rovers at St. Andrew's 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Schools
Results and Impact Two Lego Rover workshops delivered to year 3 classes.
Year(s) Of Engagement Activity 2018
URL http://legorovers.csc.liv.ac.uk
 
Description Lego Rovers at St. Anne's Primary 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Schools
Results and Impact Three Lego Robot programming workshops run at a primary school
Year(s) Of Engagement Activity 2019
URL http://legorovers.csc.liv.ac.uk
 
Description Lego Rovers at St. George's Primary 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Schools
Results and Impact A programming workshop using Lego Robots was delivered to four classes in Key Stage 2.
Year(s) Of Engagement Activity 2018
URL http://legorovers.csc.liv.ac.uk
 
Description Lego Rovers at St. Gregory's 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Schools
Results and Impact Two workshops run for primary children in Key Stage 2. Sparking questions and discussion afterwards.
Year(s) Of Engagement Activity 2018
 
Description Lego Rovers at St. Hilda's Primary 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Schools
Results and Impact A workshop on Mars Rovers and AI Programming was delivered to a class of Primary School children.
Year(s) Of Engagement Activity 2017
URL http://legorovers.csc.liv.ac.uk
 
Description Lego Rovers at St. John Bosco 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Schools
Results and Impact Two workshops run for primary children in Key Stage 2. Sparking questions and discussion afterwards.
Year(s) Of Engagement Activity 2018
 
Description Lego Rovers at St. Mary's Crosby 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Schools
Results and Impact A workshop on Mars Rovers and AI Programming was delivered to two classes of Primary School children.
Year(s) Of Engagement Activity 2018
 
Description Lego Rovers at St. Oswald's 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Schools
Results and Impact Two Lego programming workshops delivered to a primary school audience.
Year(s) Of Engagement Activity 2018
URL http://legorovers.csc.liv.ac.uk
 
Description Lego Rovers at St. Thomas' 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Schools
Results and Impact A Lego Robot Programming workshop was delivered to 4 classes of primary children
Year(s) Of Engagement Activity 2018
URL http://legorovers.csc.liv.ac.uk
 
Description Lego Rovers at Summerhill Primary 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Schools
Results and Impact Four robot programming workshops delivered to primary school children.
Year(s) Of Engagement Activity 2018
 
Description Lego Rovers at Writing on the Wall 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Public/other audiences
Results and Impact A workshop ranging over robots in science fiction, robots in the future with a practical programming component using lego robots. The event was a collaboration between the English and Computer Science and allowed a group of adults to explore current technology and future ideas about robotics and autonomous systems. There was a great deal of interested discussion.
Year(s) Of Engagement Activity 2017
 
Description Lego Rovers at the AISB Convention 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Public/other audiences
Results and Impact Lego Rovers workshop for families run as part of the AISB Convention
Year(s) Of Engagement Activity 2018
URL http://legorovers.csc.liv.ac.uk
 
Description Lego Space Rovers at Manchester Science Festival 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Public/other audiences
Results and Impact A series of 20 workshops on programming autonomous systems delivered to families as part of Manchester Science Festival. Children were highly engaged and there were questions on future directions from parents.
Year(s) Of Engagement Activity 2017
URL http://legorovers.csc.liv.ac.uk
 
Description Liverpool Light Night 
Form Of Engagement Activity Participation in an open day or visit at my research institution
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Public/other audiences
Results and Impact As part of Liverpool Light Night, the University of Liverpool organised an event showcasing university research which included a stand showing ideas about Autonomous Systems involving Lego Robots.
Year(s) Of Engagement Activity 2017
 
Description Meeting with Liverpool City Region Transport Policy unit 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Policymakers/politicians
Results and Impact Meeting with Liverpool City Region Transport Policy unit
Year(s) Of Engagement Activity 2019
 
Description Platform for Investigation 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Public/other audiences
Results and Impact Participation in a Platform for Investigation event at the Museum of Science and Industry to celebrate the 70th anniversary of the Manchester Baby Computer
Year(s) Of Engagement Activity 2018
 
Description Reasoning Machines talk for the University of the Third Age 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Public/other audiences
Results and Impact A talk covering the implementation of reasoning in machines and philosophical issues associated with computers and thought.
Year(s) Of Engagement Activity 2018
 
Description Robots at Tate Liverpool 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Public/other audiences
Results and Impact From Summer 2016 to Spring 2017 we worked with the artist Cecile B. Evans to produce an exhibit comprising robotic performances. This was hosted by the Tate Gallery at Liverpool and currently has had over 130,000 visitors.

In addition to the exhibit, involving 2 humanoid robots, a robot dog, and many videos, we held a talk/workshop at which the artist and Prof. Fisher engaged with members of the public about both the exhibit and about future developments. See: http://www.tate.org.uk/whats-on/tate-liverpool/talk-and-lecture/cecile-b-evans/conversation-michael-fisher-and-karl-tuyls
Year(s) Of Engagement Activity 2016,2017
URL http://www.tate.org.uk/whats-on/tate-liverpool/exhibition/cecile-b-evans
 
Description Space:Late at Manchester Museum of Science and Industry 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Public/other audiences
Results and Impact A stand demonstrating lego Mars Rovers featuring as part of an evening "Space:Late" event at the Museum of Science and Industry in Manchester. Many enthusiastic and interested discussions with members of the public.
Year(s) Of Engagement Activity 2018
URL https://www.msimanchester.org.uk/what-was-on/space-late
 
Description Stargazing at the World Museum 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Public/other audiences
Results and Impact A stand demonstrating lego Mars Rovers featuring as part of an evening "stargazing" event at the World Museum in Liverpool. Many enthusiastic and interested discussions with members of the public.
Year(s) Of Engagement Activity 2018
URL http://legorovers.csc.liv.ac.uk
 
Description Talk at Widnes Scibar 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Public/other audiences
Results and Impact Presented a talk on Automated Reasoning and Artificial Intelligence "Reasoning Machines: Could a Machine Think?" to Widnes Scibar. There were many questions and lively discussion after the talk.
Year(s) Of Engagement Activity 2017
 
Description Talks at Kirkby Scibar 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Public/other audiences
Results and Impact A general talk on Artificial Intelligence and Philosophical issues at Kirkby Scibar resulted in a lively discussion and was followed up with an invitation to return and talk more specifically about the Verification of Autonomous Systems. This was again followed by a lively discussion.
Year(s) Of Engagement Activity 2017
 
Description Tomorrow's World Live: Me and My Robot 
Form Of Engagement Activity A press release, press conference or response to a media enquiry/interview
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Public/other audiences
Results and Impact Dr. Dennis participated in a panel interview on the future of robotics run by the BBC at Manchester Science Festival in front of a live audience. It was live-streamed on various social media channels and subsequently made available on iPlayer.
Year(s) Of Engagement Activity 2017
URL http://www.bbc.co.uk/programmes/p05lph23
 
Description Verification and Validation of Autonomous Systems: Ethical, Social and Trustworthy behaviour 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Other audiences
Results and Impact This workshop, held on 23-24 November 2017, brought together researchers who are developing ethical, social and trustworthy behaviour in autonomous robotics, with researchers across the range of verification and validation techniques, in an informal setting to discuss these issues, to share understanding, and to stimulate collaboration. The workshop was funded by EPSRC, through both the Network on the Verification & Validation of Autonomous Systems, and the Verifiable Autonomy research project.
Year(s) Of Engagement Activity 2017
URL https://vavas.org/pastevents/workshop-verification-and-validation-of-autonomous-systems-ethical-soci...