Visit our sister organization:

The Open Data for Assessment Fund

Competition updates will be added once the series launches. As we prepare for our initial launch, we are actively seeking out datasets. Please see the final section, Get Involved, for more information on sharing potential ODAF datasets with the Lab. Funding is available!

BACKGROUND

The Open Data for Assessment Fund (ODAF) was designed to respond to the current lack of high-quality, open source assessment datasets in education. When datasets are open and available, innovators and researchers can develop new solutions (e.g., artificial intelligence and machine learning) that can reduce the cost and time to develop and administer assessments.

For example, the Automated State Assessment Prize (ASAP) dataset has become central to the field of writing assessment. ASAP, hosted in 2012, was the first study that publicly examined the ability of computers to score student essays. The dataset consists of 22,000 essays scored by human raters. The dataset was constructed to address a key pain point identified by educators – the length of time it takes to manually grade essays. This leads test companies to produce assessments made up of faster-to-grade tasks such as multiple-choice questions.
 
Through the ASAP dataset, tools were created that allowed for the testing and validation of automated essay scoring – producing rich information about student learning and student work in a fraction of the time, while also supporting rich assessments. While ASAP laid a solid foundation, the competition allowed participants to keep intellectual property, meaning that the solutions produced were not required to be publicly accessible.
 
Despite the power of open datasets, very few assessment datasets have been released. This is primarily because:
  • Almost all large educational assessment datasets are proprietary (like ASAP), held by large testing companies for competitive advantage
  • Federal funding focuses more on education interventions and research than the development of open datasets
  • Few researchers create datasets given the considerable logistical hurdles, and lack of connection to funding and their own career advancement.
This lack of assessment-focused datasets has become a major bottleneck to innovation, making advancement in the field difficult and expensive. While there have been promising accomplishments in the field, these have been isolated successes. 
 
The ODAF will address the challenges described above by collecting and releasing datasets; helping other experts collect and release datasets; and supporting the creation of data science competitions that will help draw attention to the assessment datasets. In sum, the ODAF will serve as a clearinghouse for open source assessment datasets. 

GET INVOLVED

As the ODAF is an ongoing project, we are always open to reviewing new datasets! If you have a dataset, or an idea for a dataset, that might be a good fit for the ODAF series– focused on assessment and aligned with the selection criteria– we would love to know more! Funding is available!

Please feel free to email natalie@the-learning-agency.com.

© Copyright 2022. The Learning Agency LAB. All rights Reserved.