Estimate Values from Experience
The dispatcher observes a few outcomes after each route choice.
Read the full lesson and open its transcript below. A narrated video will be added after production.
What you will learn
The dispatcher observes a few outcomes after each route choice.
A practical question
The dispatcher has tried the alternate route only three times. Those trips went well, but is the route reliably better? A value estimate describes expected future reward under an action or policy, averaged over the uncertainty in outcomes.
The useful idea
Three observations give a noisy estimate. Historical choices were also made for reasons: perhaps the alternate route was used only on clear days. Comparing its recorded rewards with the usual route without accounting for those conditions can mislead us.
Work through the example
Show the estimated value together with the number and conditions of the trips behind it. Update the estimate as controlled new outcomes arrive. It is an expectation for comparable situations, not a promise for the next truck. An estimated value improves as the system sees outcomes from comparable decisions, but a few lucky trips are weak evidence. Track how many observations support each estimate and whether route conditions changed. If a new action looks unusually good after only one trial, do not send every important delivery down it. Review uncertainty and operational cost together. A value estimate is a summary of experience under a particular policy, not a promise for every future trip. Now check the idea on a concrete example: Update an estimate from experience while tracking how much evidence supports it. A possible trap is worth naming: Noisy rewards and biased historical choices can distort value estimates. The answer follows from that evidence: A value estimate summarizes expected future reward under a policy or action. Put the three alternate-route outcomes next to the older route history. Even if all three succeeded, the evidence is thin. A cautious value estimate should reflect both those outcomes and uncertainty about untried weather or traffic conditions.
Limits and next step
Next, we'll decide when it is acceptable to try an uncertain route to learn more about it.
Worked example
Update an estimate from experience while tracking how much evidence supports it.
Quick check: test your understanding
Try it yourself: Consider this case: Update an estimate from experience while tracking how much evidence supports it. Explain how you would check the result. In your answer, address this limitation: Noisy rewards and biased historical choices can distort value estimates.
Check the reasoning
A value estimate summarizes expected future reward under a policy or action. Noisy rewards and biased historical choices can distort value estimates.
What is the main idea behind estimate values from experience?
A value estimate summarizes expected future reward under a policy or action.
How is it used in the example?
Update an estimate from experience while tracking how much evidence supports it.
What limitation should be checked?
Noisy rewards and biased historical choices can distort value estimates.
Further reading
Machine Learning — Tom M. Mitchell.
Original ToolsInVision teaching examples. Editorial approach · Suggest a correction
Accessible referenceLesson 71 transcript298 words · Narration ready for the upcoming videoOpen transcript
The dispatcher has tried the alternate route only three times. Those trips went well, but is the route reliably better? A value estimate describes expected future reward under an action or policy, averaged over the uncertainty in outcomes.
Three observations give a noisy estimate. Historical choices were also made for reasons: perhaps the alternate route was used only on clear days. Comparing its recorded rewards with the usual route without accounting for those conditions can mislead us.
Show the estimated value together with the number and conditions of the trips behind it. Update the estimate as controlled new outcomes arrive. It is an expectation for comparable situations, not a promise for the next truck. An estimated value improves as the system sees outcomes from comparable decisions, but a few lucky trips are weak evidence. Track how many observations support each estimate and whether route conditions changed. If a new action looks unusually good after only one trial, do not send every important delivery down it. Review uncertainty and operational cost together. A value estimate is a summary of experience under a particular policy, not a promise for every future trip. Now check the idea on a concrete example: Update an estimate from experience while tracking how much evidence supports it. A possible trap is worth naming: Noisy rewards and biased historical choices can distort value estimates. The answer follows from that evidence: A value estimate summarizes expected future reward under a policy or action. Put the three alternate-route outcomes next to the older route history. Even if all three succeeded, the evidence is thin. A cautious value estimate should reflect both those outcomes and uncertainty about untried weather or traffic conditions.
Next, we'll decide when it is acceptable to try an uncertain route to learn more about it.