Research Seminar: A Hypothesis-Driven Learning Analytics Approach to Assessing Students' Computer Science Focal Knowledge, Skills, and, Abilities

26 Aug 2020 - Arun Balajiee

This talk was presented Dr. Bita Akram. The talk was mainly to situate and present her work in the area of CS education and building games that provide supports to acquire the necessary skills. This baiscally involves Learning Analytics for computer science education using Artifical Intelligence. This process involves computer science student modelling using block based programming and a design for the CS curriculm. In the game built by Dr. Bita, specifically the students leanr to use binary numbers and translate binary numbers to decimal form in order to unlock different levels in the game. The game has several such levels which are unlocked by understanding and applying different concepts in computer science, without any prior knowledge. Hence, the students learn by doing by playing the game over multiple levels.

To be able to gauge how much the student learns, the actions of the student in the game is tracked and this data is collected to construct an N-gram model that is based on patterns of actions. Further, the students are clustered into three different tasks based on their actions in these different levels of the game. This way an individual student is anonymized using a collective group of students with similar characteristics, each working on a task with different characteristics.The game logic then uses an LSTM with 5-fold cross validation to be able to cluster students to their designated tasks and take their progress over different slices of time slotted by the different actions they take over the course of the game. This way the student’s change in strategy to play the game and the skills that they learn over the course of playing the game becomes trackable. The second part of talk discussed the methods implemented by Dr. Bita in collaboration with other researchers to support Automated Assessment. The idea of Automated Assemesnt is to generalize the design of a good assessment of computer science students and working with K-12 children to integrate computer science education in their classroom.

Dr. Bita and her collaborators propose a framework that labels the dataset of student’s work in a meaninful ways – that is labelling the student’s good or bad performance based on the known computer science course conditionals and policies. For example, in one of the levels of the game discussed above, the student is expected to implement the block-based program for bubblesort algorithm. This way the student’s understanding of bubble stort algorithm along with their programming abilities can be assessed. The model of automated assessment is adapted from the framework by Grover et al., 2017. The idea is to build an automated assessment framework with integrated task design, domain modelling of computer science and practicie and realizing the type of data (quantitative or qualitative). That is, during the domain modelling using the rubrics and concepts of the computer science curricula, that task related rubrics can also be integrated as a part of the design. To do this, the focal concepts are constructed from the data using Abstract Syntax Trees. An LSTM model is trained to be able to construct this Abstract Syntaxt Tree with structural n-gram based feature engineering. The vertical structure of the Abstract Syntax Tree looks at the hierarchical ordering of different tasks for a rubric and the nodes at the same level of the Abstract Syntax Tree represents the student’s knowledge on the focal concepts. Regression is then used to be able to classify and predict the learning pathway and assessment of students.

From both the presentations by Dr. Bita, I was inspired to extend the interesting topics that I am currently working to be incorporated into a few long term goals of my PhD. By working towards building a more robust educational tool, I could possibly be able to create a learning system that could potentially replace existing education systems. A learning management system with preset curricular goals and assessments would be highly beneficial for students who cannot afford to be in class environments or in cases such as the recent times of COVID-19 where remote learning has become the new normal. Another really interesting aspect to think about here is the idea of making educational games. It is refreshing to see a beneficial application of games for learning after reading research in the past such as video games instilling violence in children. Students like playing games and usually a lot good games such as God of War, Assassin’s Creed, Tomb Raider, Batman: Arkham Asylum, Portal have levels with intriguing puzzles. When young adults play these games, they tend to use a lot of intellectual faculties in cracking these puzzles and going to higher levels of the game. An educational game designed in these lines, but without violent content, with promising results, is indeed a interesting sign of what lies ahead in the future for educational technologies. It is also really amazing to notice that simple n-gram modelling can be possible to encode the information regarding student activities and when coupled with LSTM classifiers as well as regression techniques to be able to implement task based learning and assessments. Overall, I think it was really an intellectually enthralling presentation, with really practical benefits in the field of Human Computer Interaction and Educational Technologies.