Activation Functions
Meet the gate that decides which signals move forward.
Activation Functions explained visually
See how a motion light explains the gate inside an artificial neuron.
The gate after a neuron
After a neuron combines its weighted inputs and bias, an activation function shapes the value that continues to the next layer. It acts like a gate: some information passes, some is reduced, and some is stopped.
Different gates serve different jobs
A step function makes a simple on-or-off decision. Sigmoid produces a value between zero and one, while tanh produces values between negative one and positive one. ReLU blocks negative values and lets positive values continue.
Softmax compares choices
At the end of a multi-class classifier, softmax can turn scores for dog, zebra, and horse into probabilities that add up to one. The largest probability becomes the predicted class.
Nonlinearity makes networks powerful
Activation functions let a network represent curved and complicated relationships. Without this nonlinearity, stacking many layers would still behave like one simple linear calculation.
Quick check
What happens before activation?
The neuron combines its inputs, weights, and bias.
What does ReLU do?
It changes negative values to zero and allows positive values to continue.
When is softmax useful?
When a classifier must compare several possible categories.
Lesson 3 transcript
Imagine a motion-sensitive light outside a house. Small movements from leaves should not turn it on. A person walking toward the door should. The light needs a rule that decides when the incoming signal matters.
An artificial neuron needs a similar rule. First, it combines its inputs, weights, and bias into one value. Then an activation function transforms that value into the signal sent to the next layer.
The simplest example is a step function. If the value passes a threshold, the neuron turns on. If it does not, the neuron stays off. This resembles a basic yes-or-no decision.
Modern neural networks often need smoother and more flexible behavior. A sigmoid function turns any input into a value between zero and one. This can be useful when the output represents a probability, such as the probability that an email contains spam.
The hyperbolic tangent function, often called tanh, produces values between negative one and positive one. Its output can represent signals that push a decision in opposite directions.
Another widely used function is ReLU, which means Rectified Linear Unit. ReLU changes negative inputs to zero and allows positive inputs to continue. Its calculation is simple, and it often helps deep networks train efficiently.
For a classification problem with several choices, the final layer may use softmax. Imagine that a network must choose between a dog, zebra, and horse. Softmax converts the final scores into probabilities that add up to one. The category with the largest probability becomes the prediction.
Activation functions are important because they allow a network to learn relationships that a straight line cannot represent. Without this nonlinearity, adding more layers would still leave the network unable to capture many complicated patterns.
You do not need to memorize every function yet. Remember the purpose: an activation function shapes a neuron's signal and helps the network represent useful decisions.
In the next lesson, we will connect many neurons into layers and see why deeper networks can recognize complex objects.