Delta error

Gradient descent Back propagation Calculus NN Learning

In backpropagation, we define the error at a neuron in layer l as

$${\delta }_j^l=\frac{\partial C}{\partial z_j^l}$$

where C is the cost function and $z_j^l$ is the weighted input to neuron j in layer l

So, delta error is not the cost itself. It’s a local measure of how much changing that neuron’s input would change the overall cost. It’s the signal that tells each neuron how “wrong” it was, layer by layer.

 How is it different from the cost?

• Cost is a global measure: e.g., mean squared error or cross-entropy, telling us how far the network’s output is from the target .
• Delta error is a local gradient: it tells each neuron, “your weighted input contributed this much to the overall cost.”

Think of cost as the “final exam grade” (overall performance), while delta error is the “personal feedback” each student (neuron) gets on what they need to fix.

Why is delta error significant?

• It makes backpropagation possible: instead of recalculating global cost gradients for every weight from scratch, we propagate these local deltas backward efficiently.
• It allows learning to be distributed: each neuron only needs its own delta to update its weights and biases.

Formulas like

$$\frac{\partial C}{\partial w_{jk}^l}=a_k^{l-1}\cdot {\delta }_j^l$$

show that weight updates are simply the input activation times the delta of the output neuron

✅ In short:

• Cost = overall network error measure.
• Delta error = per-neuron gradient of cost w.r.t. weighted input.
• Significance: Delta is the bridge between the global cost and local weight updates, making backpropagation efficient and biologically intuitive.