Automated Assessment of AI Literacy Competencies Aligned with International Frameworks
Automated Assessment of AI Literacy Competencies Aligned with International Frameworks
Description
Details
Context and Problem Statement
International frameworks define expected AI-literacy competencies but rarely provide validated measurement instruments. Existing assessments often rely on self-report, which may differ substantially from demonstrated competence.
Research Question
Can a situated-task assessment be constructed with established psychometric validity and reliability to measure critical competencies related to generative AI?
Proposed Approach
Design tasks requiring learners to detect factual errors, identify bias, and judge AI-output appropriateness. Use multimodal models for assisted scoring with human oversight and validate the instrument through item analysis, reliability testing, and comparison with self-reported competence.
Expected Contribution
An open and psychometrically validated AI-literacy assessment instrument suitable for Francophone and Arabic-speaking contexts.
Expected Prototype
An online platform generating situated tasks, competency profiles, and aggregated institutional reports.
Datasets
UNESCO competency frameworks, a newly constructed item bank, and consent-based student responses.
Challenges
Large sample requirements, rapid item obsolescence, cultural bias, and potential misuse for selection.
Research Question
Innovation
Expected Deliverable
Technologies
Required Skills
- Psychometrics and Statistics
- Educational Assessment Design
- Web Development and Platform Engineering
- Foundations of Generative AI
Datasets
- UNESCO competency frameworks for students and teachers
- Assessment item bank created during the project
- Responses collected from volunteer students