Automating Grading of Procedural Python Program

Let's assume you want to test basic programming skills of students. For example:

Write a Python module, student_submission.py that

1. assigns to variable myString the string "$stackoverflow is great",
2. remove '$' at the beginning
3. splits it into words
4. replace the word 'great' with 'awesome'.

Assume each step is 10 points and there is no partial credit. I assume we need to mock print as well.

Students submit a single file, how should we go about testing/marking student_submission.py? Can we do this via unittest or pytest? It would be great if we could store the student grade in a dictionary, say student_dict = {'step1':0,'step2':10,'step3':10,'step4':0}

# student_submission.py
# Step 1: Assign a string to myString
myString = "$stackoverflow is great" 

# Step 2: Remove any non-alphanumeric character at the beginning
myString = myString.lstrip("$")

# Step 3: Split the string into words
words = myString.split()

# Step 4: Replace the word 'great' with 'awesome'
for i in range(len(words)):
    if words[i] == 'great':
        words[i] = 'awesome'

# Print the final result
final_string = ' '.join(words)
print(final_string)

Adapting from the answer by @chikitin and @Alexey,

The for loop can be shorten for better readability by:

for t, s in zip(target_lines, submitted_lines):
    # ...

---

For any examples below, assume the following unless otherwise specified:

target_text = "Hello\nWorld\nPython"
submitted_text = "Hello\nWorld\nJava"

No partial credit (full marks or zero points)

As the question mentioned, there may by no partial credit.

def compare_text(target, submitted, partial_credit=True):
    total_points = 0
    target_lines = target.split('\n')
    submitted_lines = submitted.split('\n')

    for t, s in zip(target_lines, submitted_lines):
        if t == s:
            total_points += 10
        elif not partial_credit:
            return 0

    return total_points
    
# Partially correct
points = compare_text(target_text, submitted_text, partial_credit=False)
print("Total points:", points) # Total points: 0

# Fully correct
submitted_text = target_text
points = compare_text(target_text, submitted_text, partial_credit=False)
print("Total points:", points) # Total points: 30

Customize point of each step (same points per step)

You might want to reuse the code for different questions, where every step has the same points, but different questions/assignment may "value" each step at different points.

Of course, you can specify what is the default points per step (for all questions/assignment. In this case, I have given a default of 10 points per step.

def compare_text(target, submitted, partial_credit=True, pts=10):
    total_points = 0
    target_lines = target.split('\n')
    submitted_lines = submitted.split('\n')

    for t, s in zip(target_lines, submitted_lines):
        if t == s:
            total_points += pts
        elif not partial_credit:
            return 0
            
    return total_points

# Give 5 points per step
points = compare_text(target_text, submitted_text, pts=5) 
print("Total points:", points) # Total points: 10

# When not specified, give 10 points per step
points = compare_text(target_text, submitted_text)
print("Total points:", points) # Total points: 20

Customize point for each step (different points per step)

You may value different steps different. Some steps may be more "valuable".

import numbers

def compare_text(target, submitted, partial_credit=True, pts=10):
    total_points = 0
    target_lines = target.split('\n')
    submitted_lines = submitted.split('\n')

    if isinstance(pts, numbers.Number):
        pts = [pts] * len(submitted_lines)
    elif len(pts) != len(submitted_lines):
        print('Error: Not all steps are assigned points')
        return
        # alternative, use `raise Exception(your_error_message)`

    for t, s, p in zip(target_lines, submitted_lines, pts):
        if t == s:
            total_points += p
        elif not partial_credit:
            return 0         

    return total_points

# Give 5 points per step
points = compare_text(target_text, submitted_text, pts=5) 
print("Total points:", points) # Total points: 10

# Give 2.5 points per step
points = compare_text(target_text, submitted_text, pts=2.5) 
print("Total points:", points) # Total points: 5

# Step 3 is more important, give more points than step 1 and 2
points = compare_text(target_text, submitted_text, pts=[5, 5, 10])
print("Total points:", points) # Total points: 10

# Forgot to give step 3 some points
points = compare_text(target_text, submitted_text, pts=[5, 5]) # error: not all steps have points assigned