Measurement, evaluation, and learning (MEL) strategies are critical to advancing open science, assessing impact, refining strategies, and adapting to evolving challenges.
Complete
Overview
Measurement, evaluation, and learning (MEL) strategies are critical to advancing open science, assessing impact, refining strategies, and adapting to evolving challenges. By applying greater rigor to open science interventions, we can better ensure accountability, optimize resource allocation, and foster continuous improvement. To this end, ORCA collaborated with the Data Innovation Lab at UC Berkeley’s Haas School of Business to develop a robust, adaptable set of qualitative and quantitative criteria to evaluate the impact of open science mandates implemented by research funders.
As part of this work, ORCA designed a methodology to evaluate the real-world impact of open science policies and piloted it with a funding organization, surveying nearly 800 awardees who had been subject to a data-sharing policy on projects that concluded five years earlier. We explored with awardees how sharing their research outputs affected areas such as outcomes visibility, innovation, career advancement, and collaboration.
The pilot surfaced several key findings. Self-reported policy compliance was high. Roughly half found compliance straightforward, while over a third described it as somewhat or very difficult, pointing to a meaningful gap between understanding a policy and operationalizing it. Barriers fell into two categories: practical obstacles like unfamiliarity with output-sharing practices, technical or repository challenges, and lack of time; and psychosocial ones, including a lack of recognition or reward, concerns about data misuse, and doubts about relevance. Most did not receive support from the funder or their institutions to help them share their outputs. When asked about the broader benefits of the policy, PIs most consistently cited increased transparency, while other intended outcomes, like accelerated discovery or reduced duplication of effort, were largely invisible to them.
These findings pointed to a broader challenge in the field: evaluating the impact of an open science policy requires a pre-existing baseline, and most funders (including the dozen-plus consulted for this work) did not yet have one in place; typically, compliance tracking was as far as monitoring went. This pilot helped establish that baseline for the participating funder and, in the process, identified the specific barriers awardees faced in complying with open-sharing policies. This insight could directly inform how funders design compliance support for future awardees. Notably, none of the surveyed researchers had access to compliance support at the time; the funder has since put dedicated support in place for new and current awardees, and the anticipation is that this will improve compliance rates going forward.
Because most researchers were not positioned to see the full downstream impact of open-sharing policies (even though their day-to-day experience remained one of the most valuable sources of insight into where policies succeeded or fell short), future evaluation work will likely need to draw on complementary methods. These could include tracking citations or reuse of published datasets, or analyzing whether public data availability accelerated publication, to build a fuller picture of open science’s broader effects on the field.
Target Communities
Research funders that have adopted open science policies interested in understanding whether those policies are having an effect
Research funders interested in adopting open sharing policies and also interested in understanding their effects
Goals
Build a rigorous, adaptable evaluation framework for assessing the impact of open science policies, beyond compliance tracking.
Pilot the methodology with a research funder to establish a baseline and prove the framework’s feasibility in practice.
Identify the practical barriers researchers face in complying with open-sharing policies, so funders know where to target support.
Funders
Gordon and Betty Moore Foundation (GBMF12347.01)
How to Get Involved
This is a service that ORCA provides for free to any interested organization. If you would like to learn more, please contact Greg Tananbaum at greg@orcaopen.org.