PyTorch explained: tensors, automatic differentiation and your model
Build a PyTorch gradient-check notebook with explicit expected values
A learning exercise compares analytical derivatives and framework outputs without hiding failed checks.
What you will learn
- Start with a small function
- Test the common confusion
- Make failures useful
Before you start
- Basic Python and calculus
- An isolated environment for later exercises
A learning exercise compares analytical derivatives and framework outputs without hiding failed checks.
Key takeaways
- The notebook is a proposed exercise.
- Expected and observed values are different evidence.
- Tolerance changes need explanation.
Start with a small function
Create a notebook for x squared and a short vector sum, recording analytical derivatives before running code. Keep shape, dtype and device visible beside every result.
This is a proposed exercise, not a notebook executed in this series. Begin on CPU without external models or datasets so unexpected outputs are easier to trace.
Test the common confusion
Compare backward accumulation with autograd.grad return values. Include a fresh forward for each pass and a deliberate reset to None, keeping expected and observed values in separate columns.
Add finite-difference checks with a chosen step and tolerance, explaining that numerical approximation is sensitive to precision. Do not demand exact equality for every floating-point experiment.
Make failures useful
Show a failed check with the code, environment and discrepancy intact. Avoid silently changing tolerance until every result turns green; justify any tolerance change.
The completed exercise should explain where each derivative came from and whether it was stored or returned. GPU timing, model training and production deployment remain separate projects.
Decision guide
| Criterion | Option A | Option B |
|---|---|---|
| Best when | You need predictable behavior and easy auditing | You need adaptive optimization and have reliable telemetry |
| Main risk | May leave performance on the table | Can become difficult to explain or debug |
Implementation steps
- 1
Write analytical expectations first.
- 2
Compare accumulation and returned derivatives.
- 3
Preserve failures and justified tolerances.
Copy-ready example
{"function":"x*x","x":3,"analytical_gradient":6,"observed_gradient":null,"executed":false}Frequently asked questions
Must the notebook download a model?
No. Small mathematical functions are enough.
What demonstrates completion?
Reproducible checks that explain derivative values, storage and any numerical tolerance.
Sources
- PyTorch / torch/autograd/__init__.pySource checked 2026-09-23
- PyTorch / README.mdSource checked 2026-09-23