PyTorch explained: tensors, automatic differentiation and your model
PyTorch autograd architecture: Python checks before engine execution
Trace the public backward path without pretending the wrapper is the entire engine.
What you will learn
- Normalize the request
- Follow the engine handoff
- Separate higher-order differentiation
Before you start
- Basic Python and calculus
- An isolated environment for later exercises
A learning exercise compares analytical derivatives and framework outputs without hiding failed checks.
Key takeaways
- The wrapper prepares engine inputs.
- backward and grad have different accumulation contracts.
- Higher-order work needs deliberate memory handling.
Normalize the request
The inspected backward function normalizes tensors and requested inputs, handles supported overrides and prepares gradient arguments. These Python steps shape the request before engine execution.
A non-scalar output needs an appropriate gradient argument; a scalar example can use the implicit seed. Shape and device mistakes should be diagnosed at this boundary before assuming a kernel defect.
Follow the engine handoff
After preparing gradients, backward resolves retain_graph from create_graph when unspecified and calls _engine_run_backward with accumulate_grad set to true.
The ordinary autograd.grad branch calls the engine with accumulation disabled and returns derivatives. The Python wrapper shows the contract, while actual graph scheduling and kernels live beyond this inspected file.
Separate higher-order differentiation
create_graph builds derivative computations for higher-order work; retain_graph concerns retaining the original graph. They are related defaults, not interchangeable fixes for every backward error.
The source warns that backward with create_graph can create reference cycles. Review the documented alternative and gradient cleanup rather than enabling both flags indiscriminately to suppress an error.
Decision guide
| Criterion | Option A | Option B |
|---|---|---|
| Best when | You need predictable behavior and easy auditing | You need adaptive optimization and have reliable telemetry |
| Main risk | May leave performance on the table | Can become difficult to explain or debug |
Implementation steps
- 1
Trace input and gradient normalization.
- 2
Compare engine accumulation flags.
- 3
Distinguish graph retention and derivative construction.
Copy-ready example
inputs -> normalization -> gradient preparation
backward -> engine(accumulate_grad=True)
grad -> engine(accumulate_grad=False)Frequently asked questions
Does this inspect the whole C++ engine?
No. It traces the Python entry point and its engine call.
Should retain_graph always be true?
The source says it is unnecessary in nearly all ordinary cases.
Sources
- PyTorch / torch/autograd/__init__.pySource checked 2026-09-23