Learn
By the end of this unit
- Record an actual user task without substituting a supplied example for observation.
- Use a defect to revise the solution, retest it, and qualify the resulting claim.
Planned guided work including the short textbook read-and-apply: 80–130 minutes, plus other project work. This is a design estimate, not measured learner time.
How to read this lesson
The labels describe how to use a passage. Color does not prove a claim is true or that a response is correct.
- Explain — Understand the idea.
- Try / predict — Make a choice or prediction before the reveal.
- Evidence — Inspect the named source or supplied data.
- Worked reasoning — Compare the steps after your own attempt.
- Caution — Check a limit, assumption or unsupported conclusion.
Each figure has its own local legend for the data. Blue selection outlines mark the current selection; other outlines are explained by each figure’s local legend. Selection is not a correctness judgment.
Observe the work rather than perform it for the user
A user test asks another person to complete a defined task using your actual version. Give the task and ordinary instructions, then observe. If you must explain a missing step, record the assistance. Helping someone recover is appropriate; hiding the help makes the kit appear more usable than the evidence supports.
Confirm that the reviewer can open the files and has the permitted inputs. State what you are observing and record only the information needed for the exercise. A classroom colleague using fictional questions can provide real evidence about that classroom attempt. The case facts remain fictional; the event, actions, difficulty, and timing are observed.
| Version | Observation | Interpretation | Next action |
|---|---|---|---|
| 1 | The reviewer cannot find the cancellation time field and needs help. | Intake instructions do not support independent use. | Move the required field before the drafting step. |
| 2 | The reviewer finds the field on a new incomplete-input case and asks for the missing fact. | The revised step worked in this retest. | Check an ordinary and exception case; do not claim universal usability. |
A revision should respond to a consequential finding. Changing a decorative color after a false refund promise does not address the defect. Keep the failed example, revised instruction, and retest result together. A defect is useful learning evidence when you explain its cause, consequence, and correction.
Compare like with like. Report complete effort and quality using the definitions you chose before testing. If a task is incomplete, show that status rather than inserting a favorable time. If one person performs the second attempt faster, practice may explain part of the difference. A bounded improvement claim is stronger than an unsupported causal claim.
Required textbook reading and application · about 15 minutes
Read, then apply one idea
Read only the passages listed here. Mathematical derivations, algorithms, code, and end-of-chapter exercises are not assigned. The time estimate includes the brief passage and one connection to your existing project update; reading the passage alone usually takes much less time. It is a planning estimate, not a timer.
7.1: Learning Issues
David L. Poole and Alan K. Mackworth. Artificial Intelligence: Foundations of Computational Agents. 3rd edition. Cambridge University Press, 2023.
Read: Read the initial Task, Data, and Measure of improvement definitions. Then jump to Measuring success and read that block, stopping before Bias. The illustrative P and N classifier paragraph may be skimmed; no symbolic work is assigned.
Why this unit: Distinguishes success on familiar examples from performance on cases not used to improve the system.
7.7: Social Impact
David L. Poole and Alan K. Mackworth. Artificial Intelligence: Foundations of Computational Agents. 3rd edition. Cambridge University Press, 2023.
Read: Start at the paragraph beginning "A model that performs well" and read the remaining section text, ending with the discussion of values, preferences, and available actions. Stop before the navigation links. Earlier examples and section 7.6 are not assigned.
Why this unit: Explains why a test result can fail to transfer to the actual setting, and why a prediction alone is not a decision.
Course connection: Use an observed failure to change the artifact or workflow, then try the revision on new comparable cases. Here, improving prompts or instructions is not the same as training the underlying AI model. The retained cases help test your revised solution; they do not certify all future uses.
Course connection: A successful classroom test may not represent the people, inputs, or conditions of intended use. Check those differences before extending the claim. Even an accurate prediction does not choose an action: the responsible person must consider the affected people, goals, available actions, and consequences.
Apply one idea: Choose one prompt for the brief connection in your existing project update.
- Record one observed failure, one specific revision, and the outcome on a new case. Explain what your evidence supports and what it cannot yet establish.
- Identify one way your pilot cases differ from intended use and one human judgment needed beyond the output or prediction. State a limited next-use or retest decision your evidence can support.
Read this textbook section here
The official page below may include material before or after your assigned selection. The reader does not automatically scroll or crop to that selection.
Assigned selection — 7.1: Learning Issues: Read the initial Task, Data, and Measure of improvement definitions. Then jump to Measuring success and read that block, stopping before Bias. The illustrative P and N classifier paragraph may be skimmed; no symbolic work is assigned.
Read this textbook section here
The official page below may include material before or after your assigned selection. The reader does not automatically scroll or crop to that selection.
Assigned selection — 7.7: Social Impact: Start at the paragraph beginning "A model that performs well" and read the remaining section text, ending with the discussion of values, preferences, and available actions. Stop before the navigation links. Earlier examples and section 7.6 are not assigned.
Add a brief connection to your existing project update: identify the book and section, explain one relevant idea in your own words, and show the decision it informs. This is part of the same update, not another assignment.
If the official reader is unavailable or inaccessible, use the original lesson explanation and worked example, apply the same question, and label that access alternative honestly. Report the problem through the course support route; do not claim to have read an unavailable text.
Optional further reading
Strategyzer — How to select the next best test ↗
Read the experiment-selection principles and the discovery/validation distinction. Name the uncertainty your next test addresses. The article’s suggested experiment cadence is not an added course requirement.
Try
Make a claim the pilot record can support
A separate supplied Cedar Desk cohort has 12 invitees, 9 starters, 6 completers, and 4 who complete another task within 14 days after invitation. Every invitee has the same follow-up window. These fictional counts are separate from the discovery interviews and reply-processing cases.
Inspect the people behind the percentages, then commit a completion claim with a denominator and period. Reveal the changed-denominator challenge and decide whether your wording needs revision. Explain whether any person's behavior changed.
Worked explanation and evidence limits
Use this after trying the decision, or as the complete reading version of the activity. These fixed teaching comparisons are separate from your exploration settings.
A changed denominator is a changed question
| Question | Numerator | Denominator | Result |
|---|---|---|---|
| How many invitees completed a task? | 6 completers | 12 invited people | 6/12 = 50% |
| How many starters completed a task? | 6 completers | 9 people who started | 6/9 ≈ 66.7% |
No additional person completes a task when the denominator changes. The 50% claim describes everyone invited; the 66.7% claim describes only those who started. Both can be correct if labeled. Neither establishes improved behavior, universal usability, paid demand, or a monthly adoption forecast.
Return among initial completers is 4/6 ≈ 66.7%; return among invitees is 4/12 ≈ 33.3%. Here “return” means completing another task within 14 days after invitation. “Most users returned” is ambiguous. “Four of six initial completers completed another task within 14 days after invitation” is inspectable.
Account for everyone: three did not start, three started but did not complete, two completed only one task, and four completed another task. Those non-overlapping groups sum to twelve. The counts alone do not tell you whether access, need, instructions, assistance, or something else explains the outcomes. Ask before choosing a remedy.
A new reviewer and a new but comparable case would help establish whether revised instructions work for an unfamiliar user. They would still not establish willingness to pay or reliability across every situation. Supplied observations are for critique, not entries to copy into your actual pilot record.
Carry forward: specify the invited population and observation window. Record the actual task, assistance, completion, defect, revision, and retest. Leave an event pending until it happens; retain failed attempts as evidence.
Decision lab · practice with your Learning Companion
The retest illusion
Keep pilot and retest cohorts separate and explain what a repair does not prove.
Take the role of evidence auditor. Commit to a decision, uncover evidence, revise your reasoning, then carry one action into Build. Use this as the unit’s reasoning rehearsal; reuse the response in your existing project update.
Open the AI Learning Companion section below this interactive lesson to enter this unit’s decision lab. Return here for Build. You can also use the complete reading route below.
Complete reading route · no AI service needed
Fictional teaching scenario. A fictional pilot has 6 acceptable results from 10 attempts. After a repair, a retest has 4 acceptable results from 5 attempts. Choose the headline.
- Success rose to 10 out of 10
- Pilot 6/10; retest 4/5; investigate the improvement
- The repair caused a 20-point improvement everywhere
Choose one and write why before revealing the evidence.
Reveal evidence and challenge
- Pilot: 6/10 = 60% acceptable; four failures identify a recurring instruction problem.
- Retest: 4/5 = 80% acceptable on the retest cases after the instruction repair.
- Design limit: Different cases and familiarity may affect the result; the samples are small.
Explain the 20 percentage-point difference without claiming the repair caused it. What case would you test next?
Compare with the worked reasoning
Report the cohorts separately: 60% in the pilot and 80% in the retest, a 20 percentage-point observed difference. If reporting all attempts together, 10/15 is about 66.7%, not a retest rate. Examine remaining failures and test fresh matched cases. This is supplied teaching evidence, not a learner’s own pilot.
Apply to your project: Name one repair, the version and cohort for the retest, and what result would make you revise again.
Write your initial choice, evidence-based revision, and project action in your own document.
Practice records are not grades or Canvas submissions.
Build
Apply this step to your chosen problem. Cedar Desk is the supplied classroom case; keep one project throughout the course.
Prepare Milestone 3: pilot and learning
- Run the approved classroom or partner task with the versioned kit. Record date, user role, input, access, assistance, result, complete effort, and quality.
- Compare the observation with the baseline or explain why a fair numerical comparison is not available.
- Select one meaningful defect or uncertainty and revise the kit.
- Retest the affected path and check that an ordinary path still works. Keep the new version and observations.
- Write a proportionate continue/revise/stop decision and one alternative explanation.
Do not fill gaps with the supplied benchmark or AI-generated observations. If no other person has tried the kit, that requirement is pending. You can finish bench testing and prepare materials while arranging an appropriate reviewer. In self-paced study, invite a willing colleague or peer to perform the bounded task; no instructor access is implied.
For an instructor-led offering, the individual check near this milestone asks you to explain a calculation, a design choice, or an evidence limit using your own work. Keep the record concise enough that another person can inspect it. The next unit asks whether the observed gain could translate into useful impact within available resources.
Add your project evidence and reflection to the check-in in Canvas.
Your project record
Write here or use your own document. Notes stay in this browser; export a backup to keep them. Nothing is submitted automatically.
Finish
What will you carry forward?
Your Canvas check-in
In the supplied cohort,6 of 12 invitees complete a task and 4 of those 6 return. Which claim is precise?
Use the evidence from this lesson to explain your reasoning in Canvas. Continue with Canvas Next below; submit your check-in in Canvas.
The simulation and Learning Companion support practice. They do not submit work, record Canvas completion, or assign a grade.
In your project record, state what you decided, the evidence you used, and one uncertainty to test next. Explain the reasoning in your own words.
Milestone 3 of 5
Pilot and learning
Save a snapshot of your project record. In a taught offering, submit it using your instructor’s arrangements and apply the feedback before the next dependent task.
Evidence and assessment criteria
Show the predeclared pilot question and measures, actual user-task record, defensible comparison, complete effort and quality, meaningful revision, and retest. Keep supplied benchmark figures separate from what happened with your version.
The result may support continuation, narrowing, or stopping. Grade quality of evaluation and judgment, not a positive AI result. The individual explanation near this milestone is ungraded formative preparation. The individual case exam is the sole assessed 40-point individual component within Milestone 5; this earlier explanation contributes no separate score.
| Criterion | Maximum points | Meets the goal | Strong transfer | Scoring anchors |
|---|---|---|---|---|
| Test design and comparison | 25 | Defines version, user task, permitted inputs, baseline/comparison, quality, full-process timing, and decision rules before observing outcomes. | Addresses order, practice, case differences, or selection effects and explains what the design can and cannot establish. | Strong evidence: 25; Meets the goal: 20; Developing: 12.5; Not demonstrated: 0. Intermediate points within this criterion's maximum are allowed when the instructor explains the criterion-specific evidence. |
| Actual observation and quantitative integrity | 30 | Records an actual classroom or partner attempt with assistance, completion, quality, complete effort, units, denominators, and missing values correctly labeled. | Uses an inspectable table/figure to expose an exception or tradeoff hidden by a favorable average and reconciles it with the recommendation. | Strong evidence: 30; Meets the goal: 24; Developing: 15; Not demonstrated: 0. Intermediate points within this criterion's maximum are allowed when the instructor explains the criterion-specific evidence. |
| Meaningful revision and retest | 25 | Links a consequential finding to a changed artifact or process and documents an actual retest of the affected path. | Checks transfer to a changed case and an unaffected path, explaining any new limitation rather than only reporting success. | Strong evidence: 25; Meets the goal: 20; Developing: 12.5; Not demonstrated: 0. Intermediate points within this criterion's maximum are allowed when the instructor explains the criterion-specific evidence. |
| Interpretation and next decision | 20 | Separates supplied from observed evidence, states limitations, and gives a proportionate continue/revise/stop decision. | Considers an alternative explanation and identifies the next evidence that would most reduce uncertainty. | Strong evidence: 20; Meets the goal: 16; Developing: 10; Not demonstrated: 0. Intermediate points within this criterion's maximum are allowed when the instructor explains the criterion-specific evidence. |
How criterion scores are assigned
Each rubric criterion keeps its published maximum points. The instructor judges the evidence against that criterion's Meets the goal and Stronger evidence descriptions and uses these numerical anchors.
| Evidence level | Anchor | What the evidence shows |
|---|---|---|
| Strong evidence | 100% of the criterion maximum | Meets the goal and demonstrates the criterion's stronger evidence or transfer. |
| Meets the goal | 80% of the criterion maximum | Demonstrates the stated goal with accurate, usable evidence. |
| Developing | 50% of the criterion maximum | Shows a relevant attempt, but important evidence is incomplete, unreliable, or needs correction. |
| Not demonstrated | 0% of the criterion maximum | Provides no usable evidence for this criterion. |
These are judgment anchors, not automatic all-or-nothing ratings. Intermediate points are allowed within the criterion maximum, with a comment identifying the evidence and remaining gap. For a 25-point criterion the anchors are 25, 20, 12.5 and 0 points. A defensible decision to narrow, revise, or stop can earn strong credit. These anchors do not change criterion maximums, assessment totals, course weights, or previously recorded scores.
A favorable project result is not required; the quality of evidence and judgment matters.
Before saving this snapshot
You can revisit any step. Completion records study progress; it does not confirm a submission, grade, or project quality.