Activation Functions

An activation function in a neural network defines how the weighted sum of the input is transformed into an output from a node or nodes in a layer of the network.

Why do we use Activation Function (AF)?

The neuron in NN performs linear computation. But, our data may have a non-linear relationship. Here comes the activation function to introduce non-linearity to the model & to improve its performance.
After computation is performed the results are passed to activation function. Then we get the output.

Most popular AF are ReLU, Sigmoid & Hyperbolic Tangent (Tanh).

ReLU - Rectified Linear Unit - is the simplest yet it took more than a decade to implement in NN after the a brief stagnant.

Image: Analytics Vidhya

ReLU keeps only the positive values & nullifies all negatives. It is the most commonly used activation function.

Sigmoid is a variant & advanced form of ReLU. It, like ReLU has only the positive values but the transition to the positive is smoother than the former.


The 3rd type of activation function, Hyperbolic Tangent (Tanh) is symmetric around the axis.

Image: Wolfram Mathworld

Thus after receiving inputs, computation is performed & results are passed to AF. When the output result reaches beyond a certain value, AF activates the neuron.




dreamer, explorer and inventor…

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Understanding and Implementing Architectures of ResNet and ResNeXt for state-of-the-art Image…

Everything Has Its Price — How to Price Words for Ad Bidding, etc

Explainable AI

Machine Learning Course In Delhi Trends to Watch Out for in 2020.

Machine Learning institute In Delhi

The Tale of Probability : Naive Bayes

Text classification with transformers in Tensorflow 2: BERT

[Hayao,Paprika]Convert AnimeGANv2 to Core ML and use it on iOS [There is a converted model]

Train your Neural Network on

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Hilaal Alam

Hilaal Alam

dreamer, explorer and inventor…

More from Medium

Getting the MNIST

Recognizing the domestic violence signal using Deep Learning and Surveillance cameras

Quick Tutorial on Matlab Deep Learning Toolbox Part 1

Cortical Columns: in vivo and in silico Perspectives