CRPC-TR89003:  On Automatic Differentiation

Andreas Griewank

1989

In comparison to symbolic differentiation and numerical differencing, the
chain rule based technique of automatic differentiation is shown to
evaluate partial derivatives accurately and cheaply. In particular it is
demonstrated that the reverse mode of automatic differentiation yields any
gradient vector at no more than five times the cost of evaluating the
underlying scalar function. After developing the basic mathematics we
describe several software implementations and briefly discuss the
ramifications for optimization.