Combining Expert Judgements

You are being invited to participate in a research study. Before you decide whether to participate, it is important for you to understand why the research is being done and what it will involve. Please take time to read the following information carefully and feel free to ask us if you would like more information or if there is anything that you do not understand. Please also feel free to discuss this with your friends, relatives, and GP if you wish. We would like to stress that you do not have to accept this invitation and should only agree to take part if you want to.

Thank you for reading this.

What is the purpose of the study?

Decisions often need to be made in group environments, but people often have differing perspectives and knowledge that influence a group’s decision-making. The purpose of this study is to investigate how the knowledge of multiple people can best be combined in an effective and accurate way.

Why have I been chosen to take part?

You have not been specifically targeted for recruitment, and there are no specific requirements for participation, other than that you are 18+ years old, a student at the University of Liverpool, and that you are able to consent to participate. To accurately answer our research question, we aim to collect data from a large number of diverse participants. Your participation in this study would be extremely valuable in helping us achieve this goal.

Do I have to take part?

No. Please note that participation is voluntary. You will be asked to indicate if you want to take part by giving your consent to participate in this study. Only people who give their informed consent will be included in this project. If you do decide to take part, you may withdraw your consent and leave the study for any reason by informing the researcher in the room with you. If you decide to withdraw, then there will be no negative consequences resulting from you doing so. During the study, you can also always decide to skip a question if you do not want to answer.

You may also be told about the opportunity to take part in a follow-up study. Your participation in this is also voluntary, and you may choose not to take part in the second stage if you wish. Registering your interest in the follow up study will also not mandate taking part in that study. Choosing not to do the follow-up study will not affect your ability to complete this study.

What will happen if I take part?

In the study, you will work in a small group to make numeric judgements (e.g., how many sweets are in a packet, or how many tigers are in the wild). These questions will have a numerical answer, which teams will work together to guess as accurately as possible. You will be asked to give both your own independent estimate and come up with an estimate as a group at various points in the study. The study should take no more than 1.5 hours.

You will also be filmed (both audio and video) in the study. The video will be taken with a 360° camera, which will capture everybody in the room at the same time.

Are there any risks in taking part?

There are no direct risks to participating in this research. Some people may find making judgements on limited evidence mildly uncomfortable or find group environments difficult and/or frustrating. If the questions asked, or the group environment, make you feel upset or uncomfortable at any point, please remember that you can always decide to take a short break or withdraw from the study altogether by notifying the facilitator. If you choose to withdraw and you are part of the EPR scheme, you will still be awarded EPR points pro rata on the amount of time you have spent being tested, with a minimum of one point.

Are there any benefits to taking part?

First-year undergraduate Psychology students at the University of Liverpool are eligible to receive EPR points. There will be no other immediate benefit to you from participating in the study. However, you will be aiding in research in decision-making and combining expert knowledge, which will contribute to improving the way experts contribute their knowledge to solve problems across many disciplines such as medicine, engineering, and ecology.

How will my data be used?

The data we collect from this study will contribute towards understanding how we can combine expert judgements more effectively. For this purpose, the data may be used as part of research articles in scientific journals, academic conference presentations, and student dissertations.

Your anonymised responses and anonymized written transcripts of your group’s discussions will be made publicly available under a Creative Commons license, so that they can be useful for as many people and purposes as possible. Importantly, we will ensure that individual participants will not be and cannot be uniquely identified in this public dataset. If you and everyone else in your group agrees, video and audio data will be made shared under a safeguarded data approach, meaning that the data will be put in an open data repository that can only be accessed by other researchers with legitimate interest, and not by the general public. This data will be available for non-commercial research purposes only and cannot be used for training artificial intelligence. We will ask you to give us consent for sharing video and audio data separately, and your decision will not affect whether you can participate in this study.

The University processes personal data as part of its research and teaching activities in accordance with the lawful basis of ‘public task’, and in accordance with the University’s purpose of “advancing education, learning and research for the public benefit”. Under UK data protection legislation, the University acts as the Data Controller for personal data collected as part of the University’s research. The Principal Investigator/Supervisor acts as the Data Processor for this study, and any queries relating to the handling of your personal data can be sent to the Project Supervisor, Dr Angelika Stefan, on angelika.stefan@liverpool.ac.uk.

Further information on how your data will be used can be found in further questions below.

How will my data be collected?

Data will be collected via surveys and video and audio recording.

How will my data be stored?

Survey data will be initially stored on third party servers managed by Rforms.org, then downloaded to a secure University of Liverpool server. The anonymised response and transcript data will be published and stored on an open data repository such as the Open Science Framework (OSF) or a similar open access platform. Data will always be hosted by GDPR compliant servers either within the UK or the EU.

Audio and video recording data will initially be saved on a memory card in the camera, from which it will be transferred to a secure University of Liverpool drive after each experimental session, or, if multiple sessions are conducted back-to-back, at the end of the working day. If everyone in your group consents, this data will also be made available on a case-by-case basis to other researchers through a safeguarded data approach, where access will only be permitted for non-commercial research purposes and the data cannot be used for training artificial intelligence. You will be asked at the start of the study to consent to storing your data in this way. If you do not consent to the safeguarded data approach, your data may still be used for analysis but will only be accessible to the research team of the present study.

How long will my data be stored for?

Data in an open data repository will be stored indefinitely because many different people may keep copies of it. The research team will store all data for at least 10 years, as mandated by the University of Liverpool’s Research Data Management Policy.

What measures are in place to protect the security and confidentiality of my data?

In survey components, you are free to avoid providing any data that you believe could make you identifiable, such as basic demographics. If you provide us with data that we believe may make you identifiable, then we will remove the identifying data before making it publicly available. We will also remove anything that may make you identifiable from the study transcript. However, you will be identifiable on audio and video recordings. These will be stored in password protected files on University of Liverpool servers and, group consent provided, will be made available on a case-by-case basis as safeguarded data on an open science repository.

Will my data be anonymised?

We aim to keep the collection of personal / demographic information that may make you identifiable to a minimum (age, gender, education level). During the study, you will be assigned a pseudonym (like a username or a participant ID). This means that your name will not appear in the dataset, but your responses will be linked back to this pseudonym. Your pseudonym will also be used to connect your written responses to the video/audio recordings. However, the audio and video recordings will only be made available to researchers who meet the safeguarding criteria together with the video/audio data. We will fully anonymize audio transcriptions before making them publicly available.

How will my data be used?

Your data will be analysed with a range of quantitative and qualitative techniques to try and improve our understanding of judgements. The data collected in this study will be analysed for a PhD and possible subsequent publications. It may also be used in future dissertations/research projects. However, the research findings may also feature in other academic output, such as conference presentations, peer-reviewed publications, and blog posts.

Who will have access to my data?

Only the study team will have access to the primary data. Anonymised data will be made publicly available. Video and audio data will be made available on a case-by-case basis to other researchers with legitimate interest as safeguarded data for non-commercial research.

Will my data be archived for use in other research projects in the future?

Yes, the anonymised and safeguarded data will be made available to comply with open science standards, meaning that researchers from all over the world could potentially use it in future research projects.

How will my data be destroyed?

At the end of the project, the primary data will be deleted from Rform.org servers. However, it will be saved on university drives, and we do not intend to destroy it. The anonymised and safeguarded dataset will be made publicly available to comply with open science standards and will therefore be unlikely to ever be destroyed.

Expenses and / or payments

There is no monetary reimbursement available for study participation. Participants who are undergraduate psychology students at the University of Liverpool are eligible to receive EPR points as part of the EPR scheme for their participation.

What will happen to the results of the study?

The collected data will be analysed and may be used for research publications, as well as undergraduate and postgraduate dissertations from the Department of Psychology.

What will happen if I want to stop taking part?

You do not have to make any judgements or answer any questions you do not want to answer. You may also withdraw your participation in the study at any time, without explanation. Results up to the period of withdrawal may be used, though you can request your data to be withdrawn prior to anonymisation. You will not receive any penalty or be disadvantaged in any way for not answering questions, withdrawing your data, or stopping your participation. If you would like your data to be withdrawn, please contact carys.hoggan@liverpool.ac.uk within two weeks after your study participation. After this period of time, your data may have been anonymised and published, so that a withdrawal is impossible.

What if I am unhappy or if there is a problem?

If you are unhappy, or if there is a problem, please feel free to let us know by contacting carys.hoggan@liverpool.ac.uk, and we will try to help. If you remain unhappy or have a complaint which you feel you cannot come to us with then you should contact the Research Ethics and Integrity Office at ethics@liv.ac.uk. When contacting the Research Ethics and Integrity Office, please provide details of the name or description of the study (so that it can be identified), the researcher(s) involved, and the details of the complaint you wish to make.

The University strives to maintain the highest standards of rigour in the processing of your data. However, if you have any concerns about the way in which the University processes your personal data, it is important that you are aware of your right to lodge a complaint with the Information Commissioner's Office by calling 0303 123 1113.

Who has reviewed this research?

The University of Liverpool’s Human Research Ethics Committee has approved this research [Reference number: 17702].

Who can I contact if I have further questions?

If you have any further questions, please contact either the Principal or Student Investigator:

Principal Investigator
Angelika Stefan
Angelika.stefan@liverpool.ac.uk

Co-Investigator
Carys Hoggan
Carys.hoggan@liverpool.ac.uk

Department of Psychology
Institute of Population Health
The University of Liverpool
Eleanor Rathbone Building
74 Bedford Street South, Liverpool, L69 7ZA

Accessibility: To request this document in an alternative format, please contact the Principal Investigator listed above.