Make your SCT robust¶
For larger exercises, you’ll often want to be flexible: if students get the end result right, you don’t want to be picky about how they got there.
However, when they do make a mistake, you want to be specific about the mistake they are making.
These seemingly conflicting requirements can be satisfied with check_correct()
and check_or()
.
check_correct()
¶
To explain the concept of check_correct()
, consider this example:
# setup
import numpy as np
arr = np.array([1, 2, 3, 4, 5, 6])
# calculate result
result = np.mean(arr)
You want the SCT to pass when the student manages to store the correct value in the object result
.
How result
was calculated, does not matter to you: as long as result
is correct, the SCT should accept the submission.
If something about result
is not correct, you want to dig a little deeper and see if the student used the np.mean()
function correctly.
The following SCT will do just that:
Ex().check_correct(
check_object("result").has_equal_value(),
check_function("numpy.mean").check_args("a").has_equal_value()
)
Inside check_correct()
, two SCT chains are specified, separated by a comma:
- A
check
chain, that has to pass in all cases, but when it fails, it doesn’t immediately stop the SCT execution and fail the exercise. - A
diagnose
chain, that is only execute if thecheck
chain failed silently.
In the example, we’re checking the end value of result
first. Only if this is not correct, will the check_function()
chain be run,
to verify if the student used numpy.mean
. If the diagnose
chain does not fail, the check`
chain is executed again ‘loudly’.
Let’s see what happens in case of different student submissions:
- The student submits
result = np.mean(arr)
check_correct()
runs thecheck_object()
chain.- This test passes, so
check_correct()
stops. - The SCT passes.
- The student submits
result = np.sum(arr) / arr.size
check_correct()
runs thecheck_object()
chain.- This test passes, so
check_correct()
stops before runningcheck_function()
. - The entire SCT passes even though
np.mean()
was not used.
- The student submits
result = np.mean(arr + 1)
check_correct()
runs thecheck_object()
chain.- This test fails, so
check_correct()
continues with thediagnose
part, running thecheck_function()
chain. - This chain fails, since the argument passed to
numpy.mean()
in the student submission does not correspond to the argument passed in the solution. - A meaningful, specific feedback message is presented to the student: you did not correctly specify the arguments inside
np.mean()
.
- The student submits
result = np.mean(arr) + 1
check_correct()
runs thecheck_object()
chain.- This test fails, so
check_correct()
continues with thediagnose
part, running thecheck_function()
chain. - This function passes, because
np.mean()
is called in exactly the same way in the student code as in the solution. - Because there is something wrong -
result
is not correct - thecheck
chain is executed again, and this time its feedback on failure is presented to the student. - The student gets the message that
result
does not contain the correct value.
Multiple functions in diagnose
and check¶
It is perfectly possible for your check
and diagnose
SCT chains to branch out into different sub-branches with multi()
:
Ex().check_correct(
multi(
check_object('a').has_equal_value(), # multiple check SCTs
check_object('b').has_equal_value()
),
check_function("numpy.mean").check_args("a").has_equal_value()
)
Why use check_correct()¶
You will find that check_correct()
is an extremely powerful function& to allow for different ways of solving the same problem.
You can use check_correct()
to check the end result of a calculation.
If the end result is correct, you can go ahead and accept the entire exercise.
If the end result is incorrect, you can use the diagnose
part of check_correct()
to dig a little deeper.
It is also perfectly possible to use check_correct()
inside another check_correct()
.
check_or()
¶
check_or()
tests whether one of the SCTs you specify inside it passes. Suppose you want to check whether people correctly printed out any integer between 3 and 7. A solution could be:
print(4)
To test this in a robust way, you could use has_code()
with a suitable regular expression that covers everything,
or you can use check_or()
with three separate has_code()
functions:
Ex().check_or(has_code('4'),
has_code('5'),
has_code('6'))
You can consider check_or()
a logic-inducing function. The different calls to pythonwhat functions that are in your SCT are actually all tests that _have_ to pass:
they are AND
tests. With check_or()
you can add chunks of OR
tests in there.