PyTorch explained: tensors, automatic differentiation and your model
Reading backward versus grad: where a derivative is stored
Use the source accumulation flag to explain two commonly confused APIs.
What you will learn
- Read backward’s result contract
- Read grad’s result contract
- Keep the source scope honest
Before you start
- Basic Python and calculus
- An isolated environment for later exercises
A learning exercise compares analytical derivatives and framework outputs without hiding failed checks.
Key takeaways
- backward stores gradients.
- grad returns requested derivatives.
- The example is analytical, not a runtime result.
Read backward’s result contract
backward returns no derivative value and accumulates into leaf gradient fields unless its input selection narrows the targets. A loop that forgets to reset those fields can combine multiple passes.
The inspected source passes accumulate_grad=True into the engine. This is stronger evidence for the storage contract than inferring behavior from a variable named loss.
Read grad’s result contract
autograd.grad returns derivatives for the requested inputs rather than accumulating them into .grad. Its ordinary engine branch explicitly sets accumulate_grad=False.
For x squared at x=3, the expected returned derivative is 6 and x.grad remains unset for this isolated call. Recompute the expression if another differentiation pass is needed.
Keep the source scope honest
This chapter reads revision 4de991c; a future release can change supported input forms. Check your installed version instead of assuming every main-branch overload exists in an older binary.
The code sample was not executed locally because torch was unavailable. No source extraction can substitute for testing native autograd behavior; the expected scalar result follows calculus.
Decision guide
| Criterion | Option A | Option B |
|---|---|---|
| Best when | You need predictable behavior and easy auditing | You need adaptive optimization and have reliable telemetry |
| Main risk | May leave performance on the table | Can become difficult to explain or debug |
Implementation steps
- 1
Decide whether you need stored or returned gradients.
- 2
Inspect accumulation before repeating passes.
- 3
Verify against the installed version.
Copy-ready example
import torch
x = torch.tensor(3.0, requires_grad=True)
(g,) = torch.autograd.grad(x * x, x)
print(g, x.grad) # Analytical expectation: 6 and NoneFrequently asked questions
Why is x.grad None after autograd.grad?
That API returns derivatives instead of accumulating into the field.
Was native autograd tested?
No. The article is fixed-source analysis with an unexecuted scalar exercise.
Sources
- PyTorch / torch/autograd/__init__.pySource checked 2026-09-23