Usage of Hints on Coding-Based Summative Assessments

A recent in-class exam in a software engineering course included a section in which students had to write code and/or tests under conditions they might face on real software engineering projects, including complex, multi-part coding questions in which later parts of the question build on earlier parts. To avoid "cascading penalties" on later questions that build on earlier ones, students could reveal the answer for any question during the exam for a "reveal penalty": If the student identified the correct place in the code to copy and paste the revealed answer, they would get 20% of the question points. That is, by simply revealing every answer and copy-pasting it correctly, even the weakest student should be able to achieve 20% of the points for the coding portion of the exam. In this paper, we study how students used this mechanism. Surprisingly, some students chose to take zero points for certain questions rather than revealing the answer, and some who asked for an answer to be revealed were unable to correctly incorporate it to get the question correct. We also find a correlation between student scores on the non-coding multiple-choice section of the exam, which had no hints/reveals available, and the coding section, even though students who performed poorly on the non-coding first half of the exam should have been able to score well on the coding part by using the reveal mechanism. That is, weak students did not benefit as much as would be expected from revealable answers. Pedagogical interpretation of these results informs our future use of the "buy a hint" format for coding-based exams.