Make your SCT robust

For larger exercises, you’ll often want to be flexible: if students get the end result right, you don’t want to be picky about how they got there. However, when they do make a mistake, you want to be specific about the mistake they are making. These seemingly conflicting requirements can be satisfied with check_correct() and check_or().

check_correct()

To explain the concept of check_correct(), consider this example:

# setup
import numpy as np
arr = np.array([1, 2, 3, 4, 5, 6])

# calculate result
result = np.mean(arr)

You want the SCT to pass when the student manages to store the correct value in the object result. How result was calculated, does not matter to you: as long as result is correct, the SCT should accept the submission. If something about result is not correct, you want to dig a little deeper and see if the student used the np.mean() function correctly. The following SCT will do just that:

Ex().check_correct(
    check_object("result").has_equal_value(),
    check_function("numpy.mean").check_args("a").has_equal_value()
)

Inside check_correct(), two SCT chains are specified, separated by a comma:

  • A check chain, that has to pass in all cases, but when it fails, it doesn’t immediately stop the SCT execution and fail the exercise.
  • A diagnose chain, that is only execute if the check chain failed silently.

In the example, we’re checking the end value of result first. Only if this is not correct, will the check_function() chain be run, to verify if the student used numpy.mean. If the diagnose chain does not fail, the check` chain is executed again ‘loudly’.

Let’s see what happens in case of different student submissions:

  • The student submits result = np.mean(arr)
    • check_correct() runs the check_object() chain.
    • This test passes, so check_correct() stops.
    • The SCT passes.
  • The student submits result = np.sum(arr) / arr.size
    • check_correct() runs the check_object() chain.
    • This test passes, so check_correct() stops before running check_function().
    • The entire SCT passes even though np.mean() was not used.
  • The student submits result = np.mean(arr + 1)
    • check_correct() runs the check_object() chain.
    • This test fails, so check_correct() continues with the diagnose part, running the check_function() chain.
    • This chain fails, since the argument passed to numpy.mean() in the student submission does not correspond to the argument passed in the solution.
    • A meaningful, specific feedback message is presented to the student: you did not correctly specify the arguments inside np.mean().
  • The student submits result = np.mean(arr) + 1
    • check_correct() runs the check_object() chain.
    • This test fails, so check_correct() continues with the diagnose part, running the check_function() chain.
    • This function passes, because np.mean() is called in exactly the same way in the student code as in the solution.
    • Because there is something wrong - result is not correct - the check chain is executed again, and this time its feedback on failure is presented to the student.
    • The student gets the message that result does not contain the correct value.

Multiple functions in diagnose and check

It is perfectly possible for your check and diagnose SCT chains to branch out into different sub-branches with multi():

Ex().check_correct(
    multi(
      check_object('a').has_equal_value(), # multiple check SCTs
      check_object('b').has_equal_value()
    ),
    check_function("numpy.mean").check_args("a").has_equal_value()
)

Why use check_correct()

You will find that check_correct() is an extremely powerful function& to allow for different ways of solving the same problem. You can use check_correct() to check the end result of a calculation. If the end result is correct, you can go ahead and accept the entire exercise. If the end result is incorrect, you can use the diagnose part of check_correct() to dig a little deeper.

It is also perfectly possible to use check_correct() inside another check_correct().

check_or()

check_or() tests whether one of the SCTs you specify inside it passes. Suppose you want to check whether people correctly printed out any integer between 3 and 7. A solution could be:

print(4)

To test this in a robust way, you could use has_code() with a suitable regular expression that covers everything, or you can use check_or() with three separate has_code() functions:

Ex().check_or(has_code('4'),
                has_code('5'),
                has_code('6'))

You can consider check_or() a logic-inducing function. The different calls to pythonwhat functions that are in your SCT are actually all tests that _have_ to pass: they are AND tests. With check_or() you can add chunks of OR tests in there.