Learn more – What is Deep Learning?

Explore DL courses

Real-life analogy for Activation Functions

Imagine that a neural network is a hose:

It takes to water (takes some input), carries it somewhere (modifies your input), and pushes the water out (produces some output).

Without an activation function, your hose will act more like a steel pipe: fixed and inflexible. Sometimes that’s good enough. Nothing wrong with using a pipe to deliver your water:

A rigid steel pipe won’t fit, no matter how you rotate it. An activation function is handy here because it allows your function to be more flexible.

In this case, a neural net with an activation function would act like a plastic garden hose. You can bend it to your specific needs and carry your water to a lot more places that are impossible to reach with a steel pipe:

So, the purpose of an activation function is to add flexibility to your hose (nonlinearity to your neural net).

Free Deep Learning Courses from Top e-learning Platforms

Why use activation functions?

1. Activation functions’ main objective is to add non-linearities into the network so that it can model more intricate and varied interactions between inputs and outputs. In the absence of activation functions, the network would only be capable of performing linear transformations, which cannot adequately represent the complexity and nuances of real-world data. Since neural networks need to implement complex mapping functions, non-linear activation functions must be used to introduce the much-needed nonlinearity property that allows approximating any function. 

2. Normalizing each neuron in the network’s output is a key benefit of utilizing activation functions. Depending on the inputs it gets and the weights associated to those inputs, a neuron’s output can range from extremely high to extremely low. Activation functions make ensuring that each neuron’s output falls inside a defined range, which makes it simpler to optimise the network during training.

Types of Activation Functions

Sigmoid activation function

The sigmoid activation function is a mathematical function used in artificial neural networks to classify information. It maps any input onto a value between 0 and 1, which can then be interpreted as either true or false. A common example of this is when an image recognition system needs to decide whether an object in the image is a cat or not; if the output from the sigmoid activation function for that particular object exceeds 0.5, then it’s classified as being “cat-like”; otherwise, it isn’t. The advantage of using this activation function over others lies in its ability to smooth out data points so that small variations don’t affect results too much – making predictions more reliable overall. 

Note: This function suffers from vanishing gradient problems.

Tanh activation function

The tanh (hyperbolic tangent) activation functions are similar but have some distinct differences compared with sigmoid: instead of mapping inputs onto values between -1 and 1 rather than just 0/1 like Sigmoid do – allowing for more nuance when classifying objects into categories based on their similarity scores across all features considered by the network at once (i.e., multi-dimensional classification). Tanh also has better gradient properties than Sigmoid functions – meaning they allow faster learning rates during training because gradients can be propagated back through layers with less distortion due to curvature effects along each axis (as opposed to flat lines like those produced by Sigmoid). This makes them ideal for deep learning applications where accuracy matters most! 

Note: This function suffers from vanishing gradient problems.

Softmax activation function

Softmax functions are often written as a combination of multiple sigmoid. We know that Sigmoid returns a value between 0 and 1. This can be treated as the probability of a data point belonging to a particular class. Therefore, sigmoid are often used for binary classification problems.

The softmax function can be used for multiclass classification problems. This function returns the probability of a data point belonging to each unique class. Here is the formula for the same −

ReLU activation function

In the case of hidden layers, Relu is the most effective option to use. It is computationally very effective. It also suffers from a vanishing gradient problem as if the value is less than 0. Then the output will be 0 means constant.

Note: If you need more clarification about your choice of activation function, especially for hidden layers, go for the Relu function.

Leaky ReLU activation function

Leaky ReLU is the most popular and effective way to solve the dying ReLU problem. Adds a small slope(as shown in fig.) in the negative direction to prevent ReLU problems from disappearing. Leaky Relu is a variant of ReLU. Instead of being 0 for z < 0, leaky ReLUs allow a small constant non-zero gradient α (typically α = 0.01).