Sürdürülebilir Kalkınma Amaçları
Yayınlar
- MakaleYalnız künye2026Scopus SJRQ4Evaluating Large Language Models in Turkish Short Answer Scoring: Validity, Reliability, and Fairness Perspectives
Kara, Abdulkadir; Yıldırım, Serkan
This study examines the performance of large language models (LLMs) in Turkish short-answer assessments within the measurement and evaluation theory framework. The GPT, Gemini, Gemma, and LLaMA models were evaluated under zero-shot and one-shot conditions with rubric support. The results show that LLMs have high self-consistency, but decision reliability can vary depending on prompt format and example sensitivity. Formulating rubrics with clear and concrete performance indicators increases model-human alignment and assessment fairness. Furthermore, error analyses revealed that while models often exhibit systematic low-scoring tendencies, they also display an ‘inverted-U’ error pattern, showing higher reliability at score extremes but struggling significantly with evaluating partial knowledge (intermediate scores). The results indicate that LLMs can support teachers in formative assessment when properly structured rubrics are used, but ethical oversight and pedagogical responsibility remain indispensable in final decisions.
- MakaleAçık Erişim2026Scopus SJRQ3From black box to pedagogical partner: Students’ sense-making of LLM-based automated assessment
Kara, Abdulkadir; Yıldırım, Serkan
While current educational research on automated assessment heavily emphasizes technical validity, a significant gap remains in understanding students’ sustained, real-world experiences with these systems in authentic learning environments. This study examines students’ experiences with a large language model-based automated assessment system embedded in the regular flow of a university course. The study employed a mixed-methods design with 47 university students over a seven-week period. Quantitative data were obtained from system interaction logs and student feedback ratings, while qualitative data were collected from focus group interviews with 24 students and 175 written feedback responses. The results reveal that students perceive the LLM-based assessment system as a learning assistant, an impartial evaluator, and a self-assessment tool. The transparency of explanations was identified as a decisive factor in building trust in the algorithmic system by helping students understand the rationale behind scores and feedback. Sustained interaction with the system triggered a shift from high-frequency trial and error to more efficient and strategic participation, indicating that the assessment criteria were gradually internalized. The system appeared to create an environment free from perceived social judgment, providing favorable conditions for productive failure, repeated attempts, and self-directed revision. Overall, the study demonstrates that artificial intelligence can be positioned as a tool that scales pedagogical intent without replacing the teacher and can be effectively integrated within the framework of human–AI complementarity in higher education.
- MakaleYayıncıda açık erişim2025Answer-based and reference-based BERT models for automatic scoring of Turkish short answers: The decisive role of task complexity
Kara, Abdulkadir; Kara, Zeynep Avinc; Yildirim, Serkan
In measurement and evaluation processes, natural language responses are often avoided due to time, workload, and reliability concerns. However, the increasing popularity of automatic short-answer grading studies for natural language responses means such answers can now be measured more quickly and reliably. This study aims to build models for predicting automatic short answer scores using the pre-trained BERT deep learning language model and to reveal their effectiveness. For this purpose, two different score prediction models were created using an answerbased approach that aligns student answers with expert judgements and a referencebased approach that matches student answers with reference answers. The dataset includes answers from 246 Physics department students responding to 4 physicsrelated questions. The performance of these models was evaluated on four physics questions representing varying levels of cognitive complexity, using Cohen's Kappa for statistical comparison of agreement with expert scores. Our findings reveal a clear interaction between model architecture and task complexity. The answer-based model was unequivocally superior for the most complex, multi-class task, effectively capturing diverse, nuanced responses. Conversely, the reference-based model demonstrated a statistically significant advantage for a well-defined, mediumcomplexity binary task. This study concludes that the optimal model for ASAG in Turkish is contingent on the cognitive demands of the assessment task, suggesting that a onesize-fits-all solution may not be the most effective approach. This provides a critical framework for practitioners, demonstrating not only that effective models are feasible for complex languages, but that their selection must be guided by task complexity.
AtıfWoS1OpenAlex2
İncele - MakaleAçık Erişim2024The New Face of Thinking in Education: Bibliometric Map of Design Thinking
Yılmaz, Turkan Karakuş; Kara, Zeynep Avinç; Kara, Abdulkadir
In the age of information and technology, students are expected to develop skills such as critical thinking, problem solving, and creative solutions. Design thinking stands out as an effective approach to develop these skills in education. Design thinking is a method that encourages empathy, creativity, collaboration, and user-oriented solutions and increases student participation in education. As educators are expected to continue applying and researching this approach, this study aims to analyze the current status and development trends of design thinking in the field of education, to reveal the research dynamics, cooperation networks, and impact areas in the field. The bibliometric analysis method was used to provide a holistic perspective in a data-based, numerical, and objective way. Within the scope of the research, the studies published in the ‘Education-Educational Research’ category with the keyword ‘design thinking’ in Web of Science between 2010-2024 were analysed. The bibliometric analysis of research on design thinking in education (DOD) between 2010 and 2024 shows that this field has become an essential subject of study. In these studies led by the USA and China, themes such as creativity, higher education, and experiential learning come to the fore. The concentration of studies in specific journals and by a limited number of prolific authors creates opportunities for new researchers. Citation data emphasise the impact and importance of TOD research on educational practices and outcomes. In addition, the findings of the research may be useful in developing new perspectives and approaches for future studies on design thinking and in examining the existing literature and previous studies from a broader perspective.
AtıfOpenAlex1İncele
