Activation function

If a line has a positive slope, on the other hand, it may reflect the increase in firing rate that occurs as input current increases. Such a function would be of the form

A special class of activation functions known as radial basis functions (RBFs) are used in RBF networks. These activation functions can take many forms, but they are usually found as one of the following functions:

• Gaussian: ϕ(𝐯)=(‖𝐯−𝐜‖2+a2)−12
• Polyharmonic splines

where σ are parameters affecting the spread of the radius.

Periodic functions can serve as activation functions. Usually the sinusoid is used, as any periodic function is decomposable into sinusoids by the Fourier transform.

Quadratic activation maps

Folding activation functions are extensively used in the pooling layers in convolutional neural networks, and in output layers of multiclass classification networks. These activations perform aggregation over the inputs, such as taking the mean, minimum or maximum. In multiclass classification the softmax activation is often used.

The following table compares the properties of several activation functions that are functions of one fold x from the previous layer or layers:

Name	Plot	Function, g′(x)	Range	Order of continuity
Identity		(−∞,∞)	0	step		(0,1)	1−g(x)2	smht⁡(x)≐eax−e−bxecx+e−dx		x1+|x|	C1
Rectified linear unit (ReLU)		0\n \\end{cases} \\\\\n = {} &\\max(0,x) = x \\textbf{1}_{x>0}\n \\end{align}"}}'>(x)+≐{0if x≤0xif x>0=max⁡(0,x)=x𝟏x>0	0\n \\end{cases}"}}'>{0if x<01if x>0		Φ(x)+12xϕ(x) where C∞
Softplus		(0,∞)	α	0 \n \\end{cases}"}}'>{αexif x<01if x>0	λ{α(ex−1)if x<0xif x≥0 with parameters λ{αexif x<01if x≥0	{0.01xif x≤0xif x>0	0\n \\end{cases}"}}'>{0.01if x<01if x>0	{αxif x<0xif x≥0 with parameter (−∞,∞)	gλ,σ,μ,β(x)=(x−λ)1{x⩾λ}1+e−sgn⁡(x−μ)(|x−μ|σ)β	C0
Sigmoid linear unit (SiLU, Sigmoid shrinkage, SiL, or Swish-x1+e−x	C∞
Exponential Linear Sigmoid SquasHing (ELiSH)		[−0.881…,∞)	−2xe−x2	sin⁡x	C∞

The following table lists activation functions that are not functions of a single fold ∂gi(x→)∂xj Range Order of continuity Softmax J C∞ Maxout (−∞,∞) ^ For instance, Hinton, Geoffrey; Deng, Li; Deng, Li; Yu, Dong; Dahl, George; Mohamed, Abdel-rahman; Jaitly, Navdeep; Senior, Andrew; Vanhoucke, Vincent; Nguyen, Patrick; Sainath, Tara; Kingsbury, Brian (2012). "Deep Neural Networks for Acoustic Modeling in Speech Recognition". IEEE Signal Processing Magazine. 29 (6): 82–97. doi:10.1109/MSP.2012.2205597. S2CID 206485943.