Checking function calls¶
Basic functionality¶
Take the following example that checks whether a student used the round()
function correctly:
# solution
round(2.718282, ndigits = 3)
# sct
Ex().check_function("round").multi(
check_args("number").has_equal_value(),
check_args("ndigits").has_equal_value()
)
# submissions that pass:
round(2.718282, 3)
round(2.718282, ndigits = 3
round(number=2.718282, ndigits=3)
round(ndigits=3, number=2.718282)
val=2.718282; dig=3; round(val, dig)
val=2.718282; dig=3; round(number=val, dig)
int_part = 2; dec_part = 0.718282; round(int_part + dec_part, 3)
- check_function() checks whether
round()
is called by the student, and parses all the arguments. check_args()
checks whether a certain argument was specified, and zooms in on the expression used to specify that argument.has_equal_value()
will rerun the expressions used to specify the arguments in both student and solution process, and compare the results.
Note
In check_args()
you can refer to the argument of a function call both by argument name and by position.
Customizations¶
If you only want to check the number
parameter, just don’t include a second chain with check_args("ndigits")
:
Ex().check_function("round").check_args("number").has_equal_value()
If you only want to check whether the number
parameter was specified, but not that it was specified correctly, drop has_equal_value()
:
Ex().check_function("round").check_args("number")
If you just want to check whether the function was called, drop check_args()
:
Ex().check_function("round")
If you want to compare the ‘string versions’ of the expressions used to set the arguments instead of the evaluated result of these expressions,
you can use has_equal_ast()
instead of has_equal_value()
:
Now, the following submissions would fail:
val=2.718282; dig=3; round(val, dig)
– the string representation ofval
in the student code is compared to2.718282
in the solution code.val=2.718282; dig=3; round(number=val, dig)
– sameint_part = 2; dec_part = 0.718282; round(int_part + dec_part, 3)
– the string representation ofint_part + dec_part
in the student code is compered to2.718282
in the solution code.
As you can see, doing exact string comparison of arguments is not a good idea here, as it is very inflexible. There are cases, however, where it makes sense to use this, e.g. when there are very big objects passed to functions, and you don’t want to spend the processing power to fetch these objects from the student and solution processes.
Functions in packages¶
If you’re testing whether function calls of particular packages are used correctly, you should always refer to these functions with their ‘full name’.
Suppose you want to test whether the function show
of matplotlib.pyplot
was called, use this SCT:
Ex().check_function("matplotlib.pyplot.show")
check_function()
can handle it when a student used aliases for the python packages (all import
and import * from *
calls are supported).
If the student did not properly call the function, check_function()
will automatically generate a feedback message that corresponds to how the student imported the modules/functions.
has_equal_value? has_equal_ast?¶
In the customizations section above, you could already notice the difference between has_equal_value()
and has_equal_ast()
for checking
whether arguments are correct. The former reruns the expression used to specify the argument in both student and solution process
and compares their results, while the latter simply compares the expression’s AST representations. Clearly, the former is more robust, but there
are some cases in which has_equal_ast()
can be useful:
- For better feedback. When using
has_equal_ast()
, the ‘expected x got y’ message that is automatically generated when the arguments don’t match up will use the actual expressions used.has_equal_value()
will use string representations of the evaluations of the expressions, if they make sense, and this is typically less useful. - To avoid very expensive object comparisons. If you are 100% sure that the object people have to pass as an argument is already correct (because
you checked it earlier in the SCT or because it was already specified in the pre exercise code) and doing an equality check on this object between
student and solution project is likely going to be expensive, then you can safely use
has_equal_ast()
to speed things up. - If you want to save yourself the trouble of building exotic contexts. You’ll often find yourself checking function calls in e.g. a for loop.
Typically, these function calls will use objects that were generated inside the loop. To easily unit test the body of a for loop, you’ll typically
have to use
set_context()
andset_env()
. For exotic for loops, this can become tricky, and it might be a quick fix to be a little more specific about the object names people should use, and just usehas_equal_ast()
for the argument comparison. That way, you’re bypassing the need to build up a context in the student/solution process and do object comparisions.
Signatures¶
The round()
example earlier in this article showed that a student can call the function in a multitude of ways,
specifying arguments by position, by keyword or a mix of those. To be robust against this, pythonwhat uses the concept of argument binding.
More specifically, each function has a function signature. Given this signature and the way the function was called,
argument binding can map each parameter you specified to an argument. This small demo fetches the signature of the open
function and tries to
bind arguments that have been specified in two different ways. Notice how the resulting bound arguments are the same:
>>> import inspect
>>> sig = inspect.signature(open)
>>> sig
<Signature (file, mode='r', buffering=-1, encoding=None, errors=None, newline=None, closefd=True, opener=None)>
>>> sig.bind('my_file.txt', mode = 'r')
<BoundArguments (file='my_file.txt', mode='r')>
>>> sig.bind(file = 'my_file.txt', mode = 'r')
<BoundArguments (file='my_file.txt', mode='r')>
When you’re using check_args()
you are actually selecting these bound arguments.
This works fine for functions like round()
and open()
that have a list of named arguments,
but things get tricky when dealing with functions that take *args
and *kwargs
.
*args
example¶
Python allows functions to take a variable number of unnamed arguments through *args
, like this function:
def multiply(*args):
res = 1
for num in args:
res *= num
return res
Let’s see what happens when different calls are bound to their arguments:
>>> import inspect
>>> inspect.signature(multiply)
<Signature (*args)>
>>> sig = inspect.signature(multiply)
>>> sig
<Signature (*args)>
>>> sig.bind(1, 2)
<BoundArguments (args=(1, 2))>
>>> sig.bind(3, 4, 5)
<BoundArguments (args=(3, 4, 5))>
Notice how now the list of arguments is grouped under a tuple with the name args
in the bound arguments.
To be able to check each of these arguments individually, pythonwhat allows you to do repeated indexing in check_args()
.
Instead of specifying the name of an argument, you can specify a list of indices:
# solution to check against
multiply(2, 3, 4)
# corresponding SCT
Ex().check_function("multiply").multi(
check_args(["args", 0]).has_equal_value(),
check_args(["args", 1]).has_equal_value(),
check_args(["args", 2]).has_equal_value()
)
The check_args()
subchains each zoom in on a particular tuple element of the bound args
argument.
**kwargs
example¶
Python allows functions to take a variable number of named arguments through **kwargs
, like this function:
def my_dict(**kwargs):
return dict(**kwargs)
Let’s see what happens when different calls are bound to their arguments:
>>> import inspect
>>> sig = inspect.signature(my_dict)
>>> sig.bind(a = 1, b = 2)
<BoundArguments (kwargs={'b': 2, 'a': 1})>
>>> sig.bind(c = 2, b = 3)
<BoundArguments (kwargs={'b': 3, 'c': 2})>
Notice how now the list of arguments is grouped under a dictionary name kwargs
in the bound arguments.
To be able to check each of these arguments individually, pythonwhat allows you to do repeated indexing in check_args()
.
Instead of specifying the name of an argument, you can specify a list of indices:
# solution to check against
my_dict(a = 1, b = 2)
# corresponding SCT
Ex().check_function("my_dict").multi(
check_args(["kwargs", "a"]).has_equal_value(),
check_args(["kwargs", "b"]).has_equal_value()
)
The check_args()
subchains each zoom in on a particular dictionary element of the bound kwargs
argument.
Manual signatures¶
Unfortunately for a lot of Python’s built-in functions no function signature is readily available because the function has been implemented in C code.
To work around this, pythonwhat already includes manually specified signatures for functions such as print()
, str()
, hasattr()
, etc,
but it’s still possible that some signatures are missing.
That’s why check_function()
features a signature
parameter, that is True
by default.
If pythonwhat can’t retrieve a signature for the function you want to test,
you can pass an object of the class inspect.Signature
to the signature
parameter.
Suppose, for the sake of example, that check_function()
can’t find a signature for the round()
function.
In a real situation, you will be informed about a missing signature through a backend error.
To be able to implement this SCT, you can use the sig_from_params()
function:
sig = sig_from_params(param("number", param.POSITIONAL_OR_KEYWORD),
param("ndigits", param.POSITIONAL_OR_KEYWORD, default=0))
Ex().check_function("round", signature=sig).multi(
check_args("number").has_equal_value(),
check_args("ndigits").has_equal_value()
)
You can pass sig_from_params()
as many parameters as you want.
param
is an alias of the Parameter
class that’s inside the inspect
module.
- The first argument of param()
should be the name of the parameter,
- The second argument should be the ‘kind’ of parameter. param.POSITIONAL_OR_KEYWORD
tells check_function
that the parameter can be specified either through a positional argument or through a keyword argument.
Other common possibilities are param.POSITIONAL_ONLY
and param.KEYWORD_ONLY
(for a full list, refer to the docs).
- The third optional argument allows you to specify a default value for the parameter.
Note
If you find vital Python functions that are used very often and that are not included in pythonwhat by default, you can let us know and we’ll add the function to our list of manual signatures.
Multiple function calls¶
Inside check_function()
the index
argument (0
by default), becomes important when there are several calls of the same function.
Suppose that your exercise requires the student to call the round()
function twice: once on pi
and once on Euler’s number:
# Call round on pi
round(3.14159, 3)
# Call round on e
round(2.71828, 3)
To test both these function calls, you’ll need the following SCT:
Ex().check_function("round", 0).multi(
check_args("number").has_equal_value()
check_args("ndigits").has_equal_value()
)
Ex().check_function("round", 1).multi(
check_args("number").has_equal_value()
check_args("ndigits").has_equal_value()
)
The first check_function()
chain, where index=0
, looks for the first call of round()
in both student solution code,
while check_funtion()
with index=1
will look for the second function call. After this, the rest of the SCT chain behaves as before.
Methods¶
Methods are Python functions that are called on objects. For testing this, you can also use check_function()
.
Consider the following examples, that calculates the mean()
of the column a
in the pandas data frame df
:
# pec
import pandas as pd
df = pd.DataFrame({ 'a': [1, 2, 3, 4] })
# solution
df.a.mean()
# sct
Ex().check_function('df.a.mean').has_equal_value()
```
The SCT is checking whether the method df.a.mean
was called in the student code, and whether rerunning the call in both student and solution process is returning the same result.
As a more advanced example, consider this example of chained method calls:
# pec
import pandas as pd
df = pd.DataFrame({ 'type': ['a', 'b', 'a', 'b'], 'val': [1, 2, 3, 4] })
# solution
df.groupby('type').mean()
# sct
Ex().check_function('df.groupby').check_args(0).has_equal_value()
Ex().check_function('df.groupby.mean', signature=sig_from_obj('df.mean')).has_equal_value()
Here:
- The first SCT is checking whether
df.groupby()
was called and whether the argument fordf.groupby()
was specified correctly to be'type'
. - The second SCT is first checking whether
df.groupby.mean()
was called and whether calling it gives the right result. Notice several things:- We describe the entire chain of method calls, leaving out the parentheses and arguments used for method calls in between.
- We use
sig_from_obj()
to manually specify a Python expression that pythonwhat can use to derive the signature from. If the string you use to describe the function to check evaluates to a method or function in the solution process, like for'df.groupby'
, pythonwhat can figure out the signature. However, for'df.groupby.mean'
will not evaluate to a method object in the solution process, so we need to manually specify a valid expression that will evaluate to a valid signature withsig_from_obj()
.
In this example, you are only checking whether the function is called and whether rerunning it gives the correct result.
You are not checking the actual arguments, so there’s actually no point in trying to match the function call to its signature.
In cases like this, you can set signature=False
, which skips the fetching of a signature and the binding or arguments altogether:
# pec
import pandas as pd
df = pd.DataFrame({ 'type': ['a', 'b', 'a', 'b'], 'val': [1, 2, 3, 4] })
# solution
df.groupby('type').mean()
# sct
Ex().check_function('df.groupby').check_args(0).has_equal_value()
Ex().check_function('df.groupby.mean', signature=False).has_equal_value()
Warning
Watch out with disabling signature binding as a one-stop solution to make your SCT run without errors.
If there are arguments to check, argument binding makes sure that various ways of
calling the function can all work. Setting signature=False
will skip this binding, which can
cause your SCT to mark perfectly valid student submissions as incorrect!
Note
You can also use the sig_from_params()
function to manually build the signature from scratch,
but this this more work than simply specifying the function object as a string from which to extract the signature.