Understanding neural networks | Part 3: Activation function
15 min read
Part 2 (https://x.com/siddhantangore/status/2085986321705873679) ended with 2 layers and a problem. My second layer did nothing. I had worked out the single layer that produces its exact output, printed both, and watched them agree to 15 decimal places.
This part fixes that, and keeps going until the program has something no version before it had:** a number that says how wrong it is**.
3 bugs on the way. All 3 returned correct answers.
The full source is at the end.
One bend
2 layers collapse into one because both do the same kind of thing. Multiply by weights, add a bias, twice over, is the same as doing it once with different numbers. The algebra folds one into the other.
ReLU does something else. Take a number. Pass it through if it is above zero. Return zero if it is below.
That is the whole function. Nothing to train, and it always cuts at zero. Because it is a different kind of operation, it cannot be folded into the layers on either side.
func relu(inputs [][]float64) [][]float64 {
for index, input := range inputs {
input[index] = math.Max(0, input[index])
}
return inputs
}
Which gave me this:
output1: [[4.8 1.21 2.385] [8.9 -1.8100000000000005 0.19999999999999996] [1.4100000000000004 1.0509999999999997 0.025999999999999912]]
activated: [[4.8 1.21 2.385] [8.9 0 0.19999999999999996] [1.4100000000000004 1.0509999999999997 0.025999999999999912]]
The one negative value became zero. Everything else passed through. All 9 numbers correct.
The function is broken!
The bug #1: Count the passes, not the values
Do not look at the output. Count how many times the loop runs.
inputs has 3 rows, so the loop body runs 3 times. There are 9 numbers. 6 are never looked at.
Which 3 does it touch? index counts rows: 0, 1, 2. input is a row. So input[index] uses the row number as a column number. Pass 0 touches [0][0], pass 1 touches [1][1], pass 2 touches [2][2].
The diagonal. I had written a diagonal ReLU.
It printed the right answer because there was exactly one negative number in those 9 cells, sitting at [1][1]. Row 1, column 1. On the diagonal. One of the three cells the loop happened to visit.
I ran the broken version and the correct version on that data afterwards. Byte-identical output.
When a wrong function returns a right answer, find the coincidence. Do not conclude the function is right.
The bug #2
input[index] = math.Max(0, input[index])
That writes into the row it was handed. A Go slice is a view onto an array, not a copy, so writing through input writes into output1 itself. relu was not returning anything new. It was changing its argument and giving the same thing back.
One line at the end of main proved it:
fmt.Printf("output1 post activation: %v\n", output1)
The -1.81 had become 0. Same variable, different value, and nothing in main had assigned to it.
That costs nothing today. Later, working out which values ReLU flattened needs the numbers the layer produced _before _the activation ran. Those are exactly the numbers this destroys.
The fix is to** build new slices instead of writing into the old ones**:
func relu(inputs [][]float64) [][]float64 {
output := make([][]float64, len(inputs))
for sampleIndex, sample := range inputs {
row := make([]float64, len(sample))
for valueIndex, value := range sample {
row[valueIndex] = math.Max(0, value)
}
output[sampleIndex] = row
}
return output
}
ReLU did not bend anything
With ReLU between the layers, I ran the collapsed single layer against the 2-layer network again. Before ReLU they had agreed to 15 decimal places.
sample 0: max difference 4.44e-16
sample 1: max difference 1.32e+00
sample 2: max difference 3.33e-16
2 of the 3 samples still match perfectly. Only 1 diverged.
Sample 1 is the only sample whose layer-1 output had a negative number in it. It is the only place ReLU did anything. For the other two every value was already positive, so ReLU passed them straight through and the network computed exactly what it would have computed with no activation at all.
So ReLU does not make the network curved. It makes it straight in pieces. Feed it a point where nothing gets flattened and the network is a plain straight-line function, and my collapsed layer still predicts it exactly. Feed it a point where something gets flattened and a different straight-line function applies, because one path through the network has been cut.
ReLU does not bend the function. It swaps in a different straight one depending on which neurons went negative.
Weights I typed in, are not weights
Every weight in the program was a number I had typed by hand. That blocks everything. No layer of 64 neurons, and no training, because training means changing weights.
So: random weights, zero biases.
Why random rather than some fixed value? My answer was that zero weights ignore the inputs. True, but that only rules out zero. Set every weight to 0.5 and all 64 neurons compute the identical number from the same input, get the identical correction during training, and move identically. They stay identical forever. 64 neurons behaving as one.
Random is not there to avoid zero. It is there to make the neurons different from each other so they can end up specialising.
Zero biases are fine because the weights already made the neurons different. There is no sameness left for a random bias to break.
Then: why multiply the random values by something small? _My answer was to keep them between 0 and 1. Wro_ng. rand.NormFloat64() has a spread of 1 around zero, and I sampled a hundred thousand of them: lowest -4.852, highest 4.286, and 31.6% land outside -1 to 1. Multiplying by 0.01 shrinks everything. It does not put a lid on anything.
5 columns of numbers
The real reason is that each layer's effect piles onto the next. I built 5 layers of 64 neurons, chained them, and printed the largest value after each one.
weights scaled by 0.05
after layer 1: largest 0.9072 factor 0.33
after layer 3: largest 0.06886 factor 0.27
after layer 5: largest 0.005263 factor 0.24
weights scaled by 1
after layer 1: largest 21.03 factor 9.53
after layer 3: largest 953 factor 8.52
after layer 5: largest 2.224e+04 factor 7.33
Neither is a layer being wrong. Each one did something modest. 5 layers turned modest into a two-hundredth, and into twenty-two thousand.
Too big and the next step raises e to those numbers.** e to the power of 22,240 is infinity,** and once one value is infinity every number after it is garbage. Too small and everything reaching the last layer is a millionth, and there is nothing left to make a decision with.
Photocopying a photocopy. Each copy is slightly darker than the last. One copy, unnoticeable. 20 copies and it is a black page. A copier that darkens by 5% is not broken. It is a copier you cannot use 20 times.
A per-layer factor of anything other than 1 is a factor raised to the power of the depth.
Both failures are a steady factor per layer, one below 1 and one above. So there is a scale between them where the factor is exactly 1. I went looking.
scale layer 1 layer 5 ratio
0.05 0.9072 0.005263 0.01
0.1 1.514 0.1546 0.10
0.2 5.196 5.346 1.03
0.5 11.08 639.2 57.69
1.0 21.03 2.224e+04 1057.54
Somewhere between 0.1 and 0.2. Dividing each run's factor by its scale gave a flat column at about 6, which means the factor is proportional to the scale. So the crossover is one sixth. About 0.167.
I could measure it and could not explain it.
4 coin flips
The explanation turned out to have nothing to do with neural networks.
Flip a coin. Heads you gain a rupee, tails you lose one. 4 flips: how far from zero do you end up?
My instinct said 2, and 2 is right. Then for 100 flips I said 50, and 50 is wildly wrong. The real answer is about 10.
50 would need 75 heads and 25 tails, with 45 heads never cancelled by a tail. With 100 independent coins that does not happen. And go from 100 flips to 400: you get 4 times as many chances to wander away from 0, and 4 times as many chances to wander back. Both grow together. The distance goes from 10 to 20. Doubled, not quadrupled.
4 times the flips, twice the distance. That is the square root, and my first instinct had it. I abandoned it and switched to "half of them" for the bigger number.
A neuron with n inputs multiplies n pairs and adds them up. The weights are random, half positive and half negative, so each product could go either way. That is n coin flips.
I checked, 20,000 runs per size:
4 products -> typical sum 1.99 sqrt(4) = 2.00
64 products -> typical sum 7.93 sqrt(64) = 8.00
1024 products -> typical sum 31.92 sqrt(1024) = 32.00
4 decimal places. And with the cancelling taken away, every product forced positive:
4 products -> typical sum 3.18
64 products -> typical sum 51.06
1024 products -> typical sum 817.61
Now it grows like n. So the cancelling does all the work, and the square root is what survives it.
One thing I had to get straight. Minus 10 is as likely as plus 10. So where do you end up on average? Zero. How far from zero are you? About 10, every time, in one direction or the other.
Position cancels. Distance does not, because distance is never negative.
That is also why my largest function uses math.Abs. A neuron outputting -50 is as broken as one outputting +50.
Solving it instead of hunting for it
A layer multiplies size by sqrt(n) times the weight scale. Set that to 1:
sqrt(n) × s = 1
s = 1 / sqrt(n)
For 64 inputs, 1/8 = 0.125. For 784 inputs, 1/28 = 0.0357.
Then ReLU's bill. It flattens roughly half the values to zero, which shrinks the output. To make up for that the weights need to be bigger by sqrt(2):
s = sqrt(2) / sqrt(n) = sqrt(2 / n)
sqrt(2/64) = 0.1768. My measured crossover was 0.167, a 6% gap because I was tracking the single largest value out of 64 rather than the typical spread.
It has a name, He initialisation, and a 2015 paper. I found it by reading five columns of my own numbers.
The float64(...) is required. Go will not divide a float by an int.
scale := math.Sqrt(2.0 / float64(numberOfInputs))
The real problem with 0.01 was not that it was too small. It was that it was a constant. 4 inputs wants 0.7071. 64 wants 0.1768. 784 wants 0.0505. No single number works for all 3.
When a constant works at one size and fails at another, stop tuning it and find what it should have been a function of.
5 chained layers with the new constructor:
after layer 1: largest 3.588 factor 1.33
after layer 3: largest 2.238 factor 0.72
after layer 5: largest 1.994 factor 0.95
3.588 to 1.994. The factors bounce, because one extreme value out of 64 is a noisy thing to measure. What is gone is the trend.
The run before that one had a bug worth recording. I wrote layer.Forward(inputBatch) instead of layer.Forward(activations), so every layer was fed the original input and nothing was chained together. The output was 3.685, 2.554, 3.249, 3.545, 3.113, which looks beautifully stable. It was five separate one-layer calculations. A stable-looking sequence produced by a program that was not stacking anything.
3 points at the origin
Hand-typed inputs mean nothing, so: a spiral. 3 classes wound around each other, deliberately impossible for a straight line to separate.
total: 300
point 0: [0 0] label 0
point 1: [0.0022272767690251997 0.009852392767996451] label 0
point 2: [0.011323556708313948 0.01673017285375231] label 0
Points 0, 100 and 200 are all exactly [0 0], with labels 0, 1 and 2. 3 identical inputs with 3 different correct answers. Nothing can get more than 1 of them right. That is a ceiling built into the data before any network exists.
3 numbers that cannot say how sure they are
Two inputs, 64 in the middle, 3 outputs, 1 per class.
sample 3: [-0.02949227326743917 0.005162064869399488 -0.031384753472476525]
sample 4: [-0.03848959786636551 0.011837775619161231 -0.041539116199199935]
Which class is sample 4? The middle number is highest, so class 1. That part works.
But class 1 beat class 0 by about 0.05 in sample 4, and by about 0.034 in sample 3. Is the first one more confident? There is no way to say. These are sums of products with no ceiling and no floor. Add 1 to all three and class 1 still wins, and now all three are positive. Nothing changed about the answer.
Training needs a number saying how wrong the network was, not which answer it picked. Being wrong by 0.05 when the numbers are around 0.03 is a disaster. Being wrong by 0.05 when they are around 800 is nothing.
A raw score tells you which answer won. It cannot tell you by how much.
Softmax fixes both. Raise e to each value, add them up, divide each by the total. e to anything is positive, whatever the sign going in, and dividing by the total forces the row to add up to 1.
Half a make and half an append
My first softmax had 4 bugs.
output := make([][]float64, len(inputs)) // 300 empty rows already exist
output = append(output, row) // this adds a 301st
make([][]float64, 300) creates 300 rows. Appending puts more after them. The result is 600 rows, the first 300 empty. Same mistake inside the loop: three zeros from make, then three appended, giving six.
I had written both patterns correctly elsewhere in the same file. Here I took half of each.
#3 bug: I worked out the total and never divided by it. The rows did not add up to 1. The largest still won, so it looked nearly right.
The #4 bug, which cannot happen yet
e to a power grows viciously. math.Exp(710) is infinity, and infinity divided by infinity is not a number.
My spiral scores are around 0.03, so this could not happen. Once training pushes the scores apart it will, and it produces garbage with no error and no line number.
The fix is 1 line. Take the row's largest value away from every element before raising e to it:
[1, 2, 3] -> subtract 3 -> [-2, -1, 0]
The largest becomes 0, so e^0 is 1, and nothing can overflow. The answer does not change at all, because the same factor appears above and below the division line and cancels. I checked on my own sample 4:
naive : [0.328023, 0.344954, 0.327024]
shifted : [0.328023, 0.344954, 0.327024]
And on a row where it matters:
naive on [900, 901, 902] : OverflowError
shifted on [900, 901, 902]: [0.090031, 0.244728, 0.665241]
When a formula overflows, look for a change that leaves the answer identical and moves the numbers somewhere safe.
sample 0: [0.3333333333333333 0.3333333333333333 0.3333333333333333] sum 1.000000
sample 4: [0.31730520658214784 0.32929534624483453 0.35339944717301763] sum 1.000000
Every row adds to exactly 1. Sample 0 is exactly a third for each class because that point is at the origin, so both inputs are zero, every weighted sum is zero, and the biases are zero too. The check I ran on my very first neuron, showing up again three hundred lines later.
Indexing a row by the correct answer
The labels had been thrown away since the spiral was written. For each sample the correct class is labels[i], so the number that matters is the probability given to that class:
probabilities[i][labels[i]]
sample 0, label 0, confidence 0.3333
sample 1, label 0, confidence 0.3291
sample 2, label 0, confidence 0.3242
That one expression is the whole idea. Index the row by the correct answer. Everything else in the row is already accounted for, because softmax made it add to 1.
Now turn 300 confidences into one number where lower is better.
Direction is the simpler half. Scale is not. Compare a network giving the right class 0.5 against one giving it 0.01. Subtracting from 1 gives 0.5 and 0.99, which says the second is twice as bad. It is nearly certain of the wrong answer while the first is merely undecided. Far worse than twice.
-log(p) has the right shape:
p = 1.0 -> 0
p = 0.5 -> 0.6931
p = 0.3333 -> 1.0987
p = 0.01 -> 4.6052
p = 0.001 -> 6.9078
p -> 0 -> infinity
Being right costs nothing. Halving the confidence adds a fixed amount. Approaching zero costs without limit.
loss: 1.1971
A baseline that is mine
-log(1/3) is 1.0986. That is the score of something that knows nothing about three classes.
My network scored 1.1971. Worse than knowing nothing, which is correct: random weights are not a neutral position, they are an arbitrary opinion, and an arbitrary opinion is usually worse than admitting you have none.
Any training that works has to push the loss below 1.0986. Stuck there, nothing is happening. Rising, something is actively wrong.
That number does not come from anyone's book and does not depend on matching anyone's printed output. It falls out of the shape of the problem: three classes, no knowledge.
The full source
Known gaps at this point in the story. Nothing changes a weight. Layer does not check that the number of biases matches the number of neurons. Forward takes a copy of the layer, which will break silently once the backward pass needs to store anything on it. crossEntropy has no guard for a confidence of exactly zero, which is infinity waiting to happen once training pushes the scores apart.
package main
import (
"fmt"
"math"
"math/rand"
)
type Layer struct {
weights [][]float64
biases []float64
}
func NewLayer(numberOfInputs, numberOfNeurons int) Layer {
weights := make([][]float64, numberOfNeurons)
biases := make([]float64, numberOfNeurons)
scale := math.Sqrt(2.0 / float64(numberOfInputs))
for neuronIndex := range weights {
neuronWeights := make([]float64, numberOfInputs)
for weightIndex := range neuronWeights {
neuronWeights[weightIndex] = rand.NormFloat64() * scale
}
weights[neuronIndex] = neuronWeights
}
return Layer{weights: weights, biases: biases}
}
func (layer Layer) Forward(inputs [][]float64) [][]float64 {
layerOutput := [][]float64{}
for _, input := range inputs {
sampleOutput := []float64{}
for neuronIndex, neuronWeight := range layer.weights {
neuronOutput := dot(input, neuronWeight)
neuronOutput += layer.biases[neuronIndex]
sampleOutput = append(sampleOutput, neuronOutput)
}
layerOutput = append(layerOutput, sampleOutput)
}
return layerOutput
}
func dot(a, b []float64) float64 {
if len(a) != len(b) {
panic(fmt.Sprintf("dot: length mismatch: %d vs %d", len(a), len(b)))
}
sum := 0.0
for index, value := range a {
sum += value * b[index]
}
return sum
}
func relu(inputs [][]float64) [][]float64 {
output := make([][]float64, len(inputs))
for sampleIndex, sample := range inputs {
row := make([]float64, len(sample))
for valueIndex, value := range sample {
row[valueIndex] = math.Max(0, value)
}
output[sampleIndex] = row
}
return output
}
func softmax(inputs [][]float64) [][]float64 {
output := make([][]float64, len(inputs))
for sampleIndex, sample := range inputs {
largest := sample[0]
for _, value := range sample {
if value > largest {
largest = value
}
}
row := make([]float64, len(sample))
sum := 0.0
for valueIndex, value := range sample {
row[valueIndex] = math.Exp(value - largest)
sum += row[valueIndex]
}
for valueIndex := range row {
row[valueIndex] /= sum
}
output[sampleIndex] = row
}
return output
}
func crossEntropy(probabilities [][]float64, labels []int) float64 {
if len(probabilities) != len(labels) {
panic(fmt.Sprintf("crossEntropy: %d samples but %d labels", len(probabilities), len(labels)))
}
sum := 0.0
for sampleIndex, row := range probabilities {
confidence := row[labels[sampleIndex]]
sum += -math.Log(confidence)
}
return sum / float64(len(probabilities))
}
func spiralData(pointsPerClass, numberOfClasses int) ([][]float64, []int) {
points := make([][]float64, 0, pointsPerClass*numberOfClasses)
labels := make([]int, 0, pointsPerClass*numberOfClasses)
for classIndex := range numberOfClasses {
for pointIndex := range pointsPerClass {
radius := float64(pointIndex) / float64(pointsPerClass-1)
angle := float64(classIndex)*4 + radius*4 + rand.NormFloat64()*0.2
x := radius * math.Sin(angle*2.5)
y := radius * math.Cos(angle*2.5)
points = append(points, []float64{x, y})
labels = append(labels, classIndex)
}
}
return points, labels
}
func main() {
points, labels := spiralData(100, 3)
layer1 := NewLayer(2, 64)
layer2 := NewLayer(64, 3)
output := layer2.Forward(relu(layer1.Forward(points)))
probabilities := softmax(output)
for i := 0; i < 5; i++ {
row := probabilities[i]
fmt.Printf("sample %d: %v sum %.6f label %d confidence %.4f\n",
i, row, row[0]+row[1]+row[2], labels[i], row[labels[i]])
}
fmt.Printf("\nloss: %.4f (chance baseline: %.4f)\n",
crossEntropy(probabilities, labels), -math.Log(1.0/3.0))
}