Explore the lesson here. Canvas holds your check-ins, submissions, feedback, and progress.

← Learning path

Unit 06 of 12

Design a fair test

What would count as improvement, unacceptable quality, or an inconclusive result?

Learn

By the end of this unit

  • Compare policies on the same cases using quality and complete effort.
  • Write a bounded test with predeclared measures, decision rules, and evidence limits.

Planned guided work including the short textbook read-and-apply: 83–133 minutes, plus other project work. This is a design estimate, not measured learner time.

How to read this lesson

The labels describe how to use a passage. Color does not prove a claim is true or that a response is correct.

  • Explain — Understand the idea.
  • Try / predict — Make a choice or prediction before the reveal.
  • Evidence — Inspect the named source or supplied data.
  • Worked reasoning — Compare the steps after your own attempt.
  • Caution — Check a limit, assumption or unsupported conclusion.

Each figure has its own local legend for the data. Blue selection outlines mark the current selection; other outlines are explained by each figure’s local legend. Selection is not a correctness judgment.

Explain

Compare the complete task on a common basis

A fast draft and a finished reply are different outputs. A useful comparison gives each method the same cases, sources, and completion criteria. Record preparation, checking, correction, and handoff, not only typing. Define what counts as an acceptable final reply before seeing which method is fastest.

Fictional benchmark on the same 12 cases; supplied teaching data, not your pilot
PolicyAcceptable repliesMaterial errorsComplete effort
Existing manual process12/12096 minutes
AI draft without verification9/12324 minutes
AI draft with source check and exception review12/12084 minutes

The unchecked method is fastest but creates three material errors. The reviewed method releases 96 − 84 =12 minutes, or 12 ÷ 96 =12.5% of the manual total, in this supplied case set. Average complete effort falls from 8 to 7 minutes per reply. At an assumed $24/hour, twelve minutes has an assigned capacity value of $4.80. Unchanged payroll means no demonstrated cash saving, and tool costs remain separate.

Set a quality boundary and a value question. A possible classroom decision rule is no unsupported commitments in the observed cases and at least 10% lower complete effort. These are chosen teaching thresholds, not universal standards. A serious failure should trigger review even when the average score looks good. Record unresolved or skipped cases as such; never count them as successful.

Why zero observed errors is not proof of zero risk

A small supplied case set can omit rare or unfamiliar situations. Even in a simplified model with independent, representative cases and a true 10% error chance, seeing zero errors in 12 trials has probability 0.9 raised to 12, approximately 28%. You do not need this formula to make the decision: zero observed errors supports a bounded next test, not a guarantee about every future customer. The actual classroom cases are not automatically a random sample.

Required textbook reading and application · about 18 minutes

Read, then apply one idea

Read only the passages listed here. Mathematical derivations, algorithms, code, and end-of-chapter exercises are not assigned. The time estimate includes the brief passage and one connection to your existing project update; reading the passage alone usually takes much less time. It is a planning estimate, not a timer.

1.4: Experimental Design and Ethics

Barbara Illowsky and Susan Dean. Introductory Statistics 2e. 2nd edition. OpenStax, Rice University, 2023.

Read: Read the opening of section 1.4, from the introduction through the definition of a double-blind experiment. Stop before Example 1.19. No medical experiment or statistical calculation is assigned.

Why this unit: Introduces changed and measured variables, comparison groups, lurking variables, random assignment, and blinding.

7.4: Overfitting

David L. Poole and Alan K. Mackworth. Artificial Intelligence: Foundations of Computational Agents. 3rd edition. Cambridge University Press, 2023.

Read: Read the opening explanation and Example 7.15 about restaurant ratings. Stop before the next discussion of model complexity and Example 7.16. The polynomial example and equations are not assigned.

Why this unit: Provides an accessible example of why a result based on a small or favorable sample can disappoint later.

Course connection: The passage uses strong language about experiments proving causation. Read that claim with its design conditions: causal interpretation depends on the allocation, measurements, implementation, and assumptions, and does not automatically generalize beyond the study. Compare the current approach with the AI-assisted approach on comparable tasks using the same scoring rules. Keep some new cases for a later retest. Random assignment or alternating task order can reduce particular sources of bias, but a small classroom pilot does not guarantee balanced groups or prove a general causal effect. Record limitations instead of claiming certainty.

Course connection: The book discusses learned models. Apply its caution to testing your solution: repeatedly tuning instructions to familiar examples can make those examples look successful without showing that the solution handles new cases. Prompt revision is not training the underlying AI model. Keep some comparable cases out of development and use the same scoring rule when you later test them.

Apply one idea: Choose one prompt for the brief connection in your existing project update.

  • Name the factor you will change, the result you will measure, the comparison approach, and one factor that could make the comparison unfair. Describe how you will reduce that problem.
  • Name one case used while developing your instructions and one new case you will hold back. Explain why success on the familiar case might mislead you, and state the common scoring rule you will use to compare performance on the new case.
Read this textbook section here

The official page below may include material before or after your assigned selection. The reader does not automatically scroll or crop to that selection.

Assigned selection — 1.4: Experimental Design and Ethics: Read the opening of section 1.4, from the introduction through the definition of a double-blind experiment. Stop before Example 1.19. No medical experiment or statistical calculation is assigned.

Official textbook section ↗

Read this textbook section here

The official page below may include material before or after your assigned selection. The reader does not automatically scroll or crop to that selection.

Assigned selection — 7.4: Overfitting: Read the opening explanation and Example 7.15 about restaurant ratings. Stop before the next discussion of model complexity and Example 7.16. The polynomial example and equations are not assigned.

Official textbook section ↗

Add a brief connection to your existing project update: identify the book and section, explain one relevant idea in your own words, and show the decision it informs. This is part of the same update, not another assignment.

If the official reader is unavailable or inaccessible, use the original lesson explanation and worked example, apply the same question, and label that access alternative honestly. Report the problem through the course support route; do not claim to have read an unavailable text.

Optional further reading

OpenStax — Confidence Intervals introduction ↗

Read the opening explanation of point and interval estimates, stopping before the mathematical example. State why a sample estimate is uncertain. Derivations are optional.

Try

Case file: approved Cedar Desk facts
Supplied source facts

Fictional classroom facts. Use these to check every policy claim in the exercise.

Use these supplied facts when drafting or checking replies
SourceApproved fact
S1 — SessionA Standard Workshop lasts 90 minutes and costs $40 per participant, including materials. Participants must be at least 16.
S2 — CancellationAt least 72 hours before the session: refund eligibility. From 24 hours up to, but not including, 72 hours: credit valid for 90 days. Under 24 hours: no automatic refund or credit; the owner reviews exceptional requests. Do not promise a payment-processing date.
S3 — ReschedulingA request made at least 48 hours before the session may be considered, subject to available space and coordinator confirmation. No availability is supplied in this packet.
S4 — Missing factsAsk only for the missing fact needed to apply a policy. Classroom booking references are fictional. A draft is not confirmation that a booking was changed or a refund was processed.
S5 — Special requestsGroup prices, accessibility arrangements, age exceptions, and policy exceptions require owner review. The packet supplies no basis for promising them.
S6 — ReleaseA person checks the source, conditions, tone, and next step before a reply leaves the service. A human decides exceptions and authorizes commitments.
Try / predict

Commit a policy before testing its resilience

Use the same twelve cases and the classroom decision rule: zero material errors and at least 10% lower complete effort than the 96-minute manual process. Choose the process you would investigate next and explain the consequence you are prioritizing. Commit the recommendation before revealing the added-rework challenge.

After the challenge, keep or revise your recommendation using both boundaries. An unfavorable result is useful evidence. Your decision concerns a bounded next test; it is not permission to promise reliability for every customer.

Worked explanation and evidence limits

Use this after trying the decision, or as the complete reading version of the activity. These fixed teaching comparisons are separate from your exploration settings.

Worked reasoning

Does the reviewed process still meet the declared rule?

The fixed challenge keeps the twelve cases, manual comparison, and quality counts unchanged. It adds six hypothetical rework minutes to the reviewed process. This is a sensitivity exercise, not another observed experiment. The classroom effort limit is 96 × (1 − 0.10) = 86.4 minutes; a result at or below that limit meets the at-least-10% rule.

Fixed rework challenge; complete effort across the same twelve cases
QuantityReviewed baseWith six added rework minutes
Complete effort84 minutes90 minutes
Time released versus 96 minutes12 minutes6 minutes
Effort reduction12 ÷ 96 = 12.5%6 ÷ 96 = 6.25%
Acceptable replies / material errors12/12 / 012/12 / 0; held fixed by assumption
At least 10% lower effort?Meets the 86.4-minute limitMisses the 86.4-minute limit
Assigned capacity value at $24/hour$4.80$2.40

The reviewed base meets both chosen classroom boundaries on this supplied set. Added rework leaves the assumed quality unchanged but removes enough of the time advantage to miss the effort rule. Revise the workflow, narrow the task, or investigate another justified benefit before scaling. Do not lower the threshold after seeing the result without recording that it is a new decision rule.

The unchecked process takes 24 minutes but leaves three material errors, so it fails the quality boundary. Keeping the manual process is a valid fallback; the supplied reviewed result also gives a reason to investigate a bounded alternative. Zero errors in twelve supplied cases is not proof of zero future risk. Tool cost is not in this time comparison, and unchanged payroll means no demonstrated cash saving.

Design a fairer actual comparison

If one person repeats identical questions, the second method may benefit from memory. Use matched task versions or vary the order across reviewers when feasible, record the order, and keep instructions and quality criteria consistent. Small classroom tests still have limits. Describe them instead of claiming that a before-and-after difference proves AI caused every improvement.

Carry forward: state the capstone task, common cases, acceptable-output definition, complete timing, reviewer, and continue/revise/stop rule before your next comparison. Name the unfamiliar cases and operating costs that still need evidence.

Decision lab · practice with your Learning Companion

The event table that invents a room

Check a structured AI output against its input and compare complete effort on matched cases without lowering quality.

Take the role of extraction test designer. Commit to a decision, uncover evidence, revise your reasoning, then carry one action into Build. Use this as the unit’s reasoning rehearsal; reuse the response in your existing project update.

Open the AI Learning Companion section below this interactive lesson to enter this unit’s decision lab. Return here for Build. You can also use the complete reading route below.

Complete reading route · no AI service needed

Fictional teaching scenario. A fictional community calendar needs a table from twelve permitted event notices. The output has three columns: Event, Date, Location. You may use the supplied candidate table without an AI account; no live extractor has been run. A fast method advertises only its table-generation time. Choose how you would compare it with manual entry.

  1. Pick the lowest draft time
  2. Compare complete effort and acceptable outputs on the same cases
  3. Give the fast method easier notices

Choose one and write why before revealing the evidence.

Reveal evidence and challenge
Supplied evidence
  • Source notices and candidate rows: Supplied fictional inputs: A — “Repair meetup, 12 May, Library Room 2.” B — “Garden exchange, 15 May; location to follow.” Candidate rows (Event | Date | Location): Repair meetup | 12 May | Library Room 2; Garden exchange | 15 May | Library Room 2. Compare each field with its own notice; no year is supplied.
  • Declare the quality rule first: An acceptable row matches its notice and records an absent field as “Not supplied”; it must not borrow another event’s detail or guess a year. Use all twelve notices for every method. Time the full attempt from opening inputs through saving results, including setup, checking and corrections within the stated stages. The chosen classroom rule is 12/12 acceptable final rows and at least 10% lower effort than manual entry.
  • Matched methods: complete effort and quality: Supplied extraction benchmark on the same twelve notices: manual entry uses 80 total minutes with 12/12 acceptable rows. Fast extraction uses 20 draft + 40 review + 30 attempted repair = 90 total minutes; 9/12 rows are acceptable after that work, so three still fail the rule. The reviewed method uses a field instruction, the “Not supplied” rule, source checks and corrections: 30 draft + 28 review + 10 repair = 68 total minutes; 12/12 final rows are acceptable. These extraction times are separate from the reply-policy benchmark in the lesson. They are fictional benchmark counts and times, not your observations or a live tool-performance claim.

Repair the unsupported field in the Garden exchange row. Which method earns a bounded next test under both declared rules? Calculate its reduction from the 80-minute manual-entry baseline. If the reviewed extractor needs eight more repair minutes, does your recommendation change? Use this extraction table, not the lesson’s reply-policy table.

Compare with the worked reasoning
Worked reasoning

Garden exchange’s location is “Not supplied”; Library Room 2 belongs only to Repair meetup. Preserve 15 May as stated and do not invent a year. Reviewed extraction uses 68 rather than 80 minutes: 12 minutes less, or 12/80 = 15%, with 12/12 acceptable rows in this supplied set. It meets both chosen classroom rules. Fast extraction uses 90 minutes and leaves three unacceptable rows. Eight added repair minutes raise the reviewed total to 76: (80 − 76)/80 = 5%, missing the 10% effort rule even if quality stays 12/12. These are the extraction task’s times, not the reply-policy table’s times. Retain a manual fallback and revise or investigate another justified benefit. Twelve fictional cases cannot establish future reliability; keep input cases, quality definitions, assistance and timing consistent.

Apply to your project: For your same capstone, define an inspectable input/output contract and one missing or ambiguous input. If extracting, name the columns and missing-field rule; if using another capability, name its equivalent quality check. Predeclare matched cases, complete-effort timing and a continue/revise/stop rule.

Write your initial choice, evidence-based revision, and project action in your own document.

Practice boundary

Practice records are not grades or Canvas submissions.

Build

Apply this step to your chosen problem. Cedar Desk is the supplied classroom case; keep one project throughout the course.

Write the pilot card before running the task

Add one page containing the question, version, intended user, permitted inputs, task, comparison, quality definition, timing start/stop, planned cases, and continue/revise/stop rules. Name the reviewer and the person who may authorize any external use. A classroom task with fictional information needs no live external integration.

Use a measurement sheet with one row per attempt: case, method, order, completion, preparation time, checking time, rework time, handoff time, material errors, assistance, and note. Mark a quantity “not measured” when necessary. A blank cell is not zero. Include ordinary, incomplete, and exception inputs; justify any additional cases needed by your scope.

Check readiness with the instructor when the offering requires it. If the test is not ready for a partner, practice privately with the supplied case. In the next unit, collect an actual observation and keep it separate from this benchmark.

Add your project evidence and reflection to the check-in in Canvas.

Your project record

Write here or use your own document. Notes stay in this browser; export a backup to keep them. Nothing is submitted automatically.

Finish

What will you carry forward?

Your Canvas check-in

Extra rework increases the reviewed total from 84 to 90 minutes against a 96-minute baseline. What is the new reduction?

Use the evidence from this lesson to explain your reasoning in Canvas. Continue with Canvas Next below; submit your check-in in Canvas.

The simulation and Learning Companion support practice. They do not submit work, record Canvas completion, or assign a grade.

Extra rework increases the reviewed total from 84 to 90 minutes against a 96-minute baseline. What is the new reduction?

In your project record, state what you decided, the evidence you used, and one uncertainty to test next. Explain the reasoning in your own words.

You can revisit any step. Completion records study progress; it does not confirm a submission, grade, or project quality.