Report on MC Tasks

Log Data of Shuffeld and Unshuffeld MC Tasks

Test Course

To investigate the exact behaviour of JACK and data logging we run an example course with the task Komposition Aussagen.

The course has two task:

  • First task (01_no_shuffle): disabled shuffling
  • Second task (02_shuffle): enabled shuffling

Log Data

test.raw_data <- readr::read_csv2('00_data/Testkurs_20251105.csv', progress = FALSE) %>%
  janitor::clean_names()

Combinations

Positions, Rules & feldinhalt

test.raw_data %>%
  group_by(exercise_name) %>%
  distinct(exercise_name, position_ao, regel, feldinhalt) %>%
  dplyr::mutate(feldinhalt = str_extract(feldinhalt, "(?<=var=)[A-Za-z0-9_]+")) %>%
  dplyr::arrange(exercise_name, position_ao) %>%
  dplyr::ungroup() %>%
  pivot_wider(
    names_from = exercise_name,
    values_from = regel) %>%
    knitr::kable(escape = TRUE)
position_ao feldinhalt 01_no_shuffle 02_shuffle
0 VAR_richtig1 CORRECT CORRECT
1 VAR_richtig2 CORRECT CORRECT
2 VAR_falsch1 WRONG WRONG
3 VAR_falsch2 WRONG WRONG
4 VAR_falsch3 WRONG WRONG

There isn’t any variiation!

Pattern of Answers

Shuffle
No Yes No Yes No Yes
100 Punkte
11000 11000
50 Punkte
10100
10010 10010
10001 10001
10000 10000
01100 01100
01010 01010
01001 01001
01000
0 Punkte
00110 00110
00100
00011 00011
00001

Example – 50 Percent

example_1 <- test.raw_data %>%
  dplyr::select(zeitstempel, user_login, kursstart, eingereichte_antwort, punkte) %>%
  dplyr::filter(punkte == 50,
                user_login == 'snjeklen') %>%
  dplyr::group_by(kursstart, user_login) %>%
  dplyr::slice_max(zeitstempel, n = 1, with_ties = FALSE) %>%
  dplyr::distinct(eingereichte_antwort, .keep_all = TRUE) %>%
  dplyr::ungroup()
zeitstempel user_login kursstart eingereichte_antwort punkte
2025-11-05 13:56:35 snjeklen 2025-11-05 13:54:51 01001 50
2025-11-05 14:00:49 snjeklen 2025-11-05 13:59:57 01100 50

Investigating the first data point, with the answer pattern 01001 and course start 2025-11-05 13:54:51 UTC

  • Saving Kursstart
kursstart_example_1 <- example_1 %>%
  dplyr::slice_min(zeitstempel, n = 1, with_ties = FALSE) %>%
   dplyr::pull(kursstart)
2025-11-05 13:54:51.626
  • Log data for that submission
example_1_data <- test.raw_data %>%
  dplyr::filter(kursstart == kursstart_example_1,
                user_login == 'snjeklen',
                exercise_name == '02_shuffle') %>%
  dplyr::mutate(feldinhalt = str_extract(feldinhalt, "(?<=var=)[A-Za-z0-9_]+"))
  • Final submit
final_submit <- example_1_data %>%
  dplyr::slice_max(zeitstempel, n = 1, with_ties = FALSE) %>%
  dplyr::distinct(eingereichte_antwort) %>%
  dplyr::pull()
01001
  • Log data for analyzing statements
analyzing_data <- example_1_data %>%
  dplyr::group_by(position_ao) %>%
  dplyr::filter(variablenname == feldinhalt) %>%
  dplyr::select(position_ao, zeitstempel, user_login, feldinhalt, regel, eingereichte_antwort) %>% 
  dplyr::slice_max(zeitstempel, n = 1, with_ties = FALSE) %>%
  dplyr::distinct() %>%
  dplyr::mutate(answer_pos = substr(eingereichte_antwort, position_ao + 1, position_ao + 1)) %>%
  dplyr::mutate(stud_answer_correct = dplyr::case_when(
    regel == "CORRECT" && answer_pos == "1" || regel == "WRONG" && answer_pos == "0" ~ TRUE,
    TRUE ~ FALSE
  ))
position_ao zeitstempel user_login feldinhalt regel eingereichte_antwort answer_pos stud_answer_correct
0 2025-11-05 13:56:35 snjeklen VAR_richtig1 CORRECT 01001 0 FALSE
1 2025-11-05 13:56:35 snjeklen VAR_richtig2 CORRECT 01001 1 TRUE
2 2025-11-05 13:56:35 snjeklen VAR_falsch1 WRONG 01001 0 TRUE
3 2025-11-05 13:56:35 snjeklen VAR_falsch2 WRONG 01001 0 TRUE
4 2025-11-05 13:56:35 snjeklen VAR_falsch3 WRONG 01001 1 FALSE

Pattern & Variables in JACK

\qquad

Learning

  • I am stupid and got confused
  • Better documentation
  • The Log is not shuffeld and therefore the matching can be made on the position and feldinhalt
  • (Reading) access to the course would be helpful