Softmax function

K possible outcomes. It is a generalization of the logistic function to multiple dimensions and is used in multinomial logistic regression. The softmax function is often used as the last activation function of a neural network to normalize the output of a network to a probability distribution over predicted output classes.

K real numbers and normalizes it into a probability distribution consisting of K = 1, the sole component will always be exactly equal to 1.

2-dimensional softmax as function of ⁠ 1"}}'>K>1⁠, takes a tuple σ(𝐳)i=ezi∑j=1Kezj. In words, the softmax applies the standard exponential function to each element K real numbers) and normalizes these values by dividing by the sum of all these exponentials. The normalization ensures that the sum of the components of the output vector (0.001,0.002,0.997), which amounts to assigning almost all of the total unit weight in the result to the position of the tuple's maximal element (of 8).

In general, instead of b > 1, then larger input components will result in larger output probabilities, and increasing the value of b will create probability distributions that are more concentrated around the positions of the smallest input values. Writing β) yields the expressions β=1/kT, where β (or  max function: the function whose vector component values are all zero, except that it has a value of 1 corresponding to the location of the tuple's largest element. Because the term "softmax" can also be used for the closely related, but real-valued, LogSumExp function, which is a smooth maximum, some prefer to describe the one-hot version as "softargmax". The term "softmax" is conventional in machine learning.

Formally, instead of considering the arg max function with one-hot representation of the output (assuming there is a unique maximum arg): i is the argzi is the unique maximum value of  max values (multiple equal 1k⁠, where argmax⁡(1,5,5)=(0,12,12), since the second and third argument are both the maximum. In case all arguments are equal, this is simply  max values are singular points (or singularities, and form the singular set) – while points with a single arg⁠ max pointwise, meaning that for each fixed input β→∞⁠,  max is not continuous at the singular set where two coordinates are equal, while the uniform limit of continuous functions is continuous. The reason it fails to converge uniformly is that for inputs where two coordinates are almost equal (and one is the maximum), the argσβ(1,0.9999)→(1,0), and ⁠ min values. In the language of tropical analysis, the softmax is a deformation or "quantization" of arg max or arg⁠ max. For example, a difference of 10 is large relative to a temperature of 1: σ1/100(0,10)=(11+e1/10,e1/101+e1/10)≈(0.475,0.525). As T=1/β→0, so eventually all differences become large (relative to a shrinking temperature), which gives another interpretation for the limit behavior.

In statistical mechanics, the softmax function is known as the Boltzmann distribution (or Gibbs distribution): the index set Z; and the factor j-th class given a sample tuple P(y=j∣𝐱)=e𝐱T𝐰j∑k=1Ke𝐱T𝐰k. This can be seen as the composition of 𝐱T𝐰 denotes the inner product of 𝐰 to tuples ℝK.

The standard softmax function is often used in the final layer of a neural-network-based classifier. Such networks are commonly trained under a log loss (or cross-entropy) regime, giving a non-linear variant of multinomial logistic regression.

Since the function maps a tuple and a specific index i,k and thus may also be expressed as β.

See multinomial logit for a probability model which uses the softmax activation function.

In the field of reinforcement learning, a softmax function can be used to convert values into action probabilities. The function commonly used is τ is called a temperature parameter (in allusion to statistical mechanics). For high temperatures (K of possible outcomes is often large, e.g. in case of neural language models that predict the most likely outcome out of a vocabulary which might contain millions of possible words. This can make the calculations for the softmax layer (i.e. the matrix multiplications to determine the O(log2K). In practice, results depend on choosing a good strategy for clustering the outcomes into classes. A Huffman tree was used for this in Google's word2vec models (introduced in 2013) to achieve scalability.

A second kind of remedies is based on approximating the softmax (during training) with modified loss functions that avoid the calculation of the full normalization factor. These include methods that restrict the normalization sum to a sample of outcomes (e.g. importance sampling, target sampling).

σ(𝐳)i=eβ(zi−m)∑j=1Keβ(zj−m), where k1,…,kN, and a list of "value vectors" (K−1)-simplex, cutting the dimension by one (the range is a (x,x,…,x), softmax is just the uniform distribution on outputs, 𝐳 yields ezi+c=ezi⋅ec), so the ratios do not change: ∑zi=0, to the open simplex of positive values that sum to σ((0,1))=(11+e,e1+e), but z2=0), so ez1ez1+ez2=exex+1,ez2ez1+ez2=1ex+1. The 1-dimensional input could alternatively be expressed as the line ⁠⁠𝐳=(z1,…,zK)∈ℝK⁠.

The gradient of softmax is 1 if Vj(x)⁠, we can ensure both conditions by using a normalised exponential transformation:

1. ↑ Positive β as coldness.
2. ↑ The notation T=1/β.
3. ↑ For (1/n,…,1/n)⁠, corresponding to the discrete uniform distribution.
4. ↑ In statistical mechanics, fixing Bishop, Christopher M. (2006). Pattern Recognition and Machine Learning. Springer. ISBN 0-387-31073-8.
5. 1 2 LeCun, Yann; Chopra, Sumit; Hadsell, Raia; Ranzato, Marc’Aurelio; Huang, Fu Jie (2006). "A Tutorial on Energy-Based Learning" (PDF). In Gökhan Bakır; Thomas Hofmann; Bernhard Schölkopf; Alexander J. Smola; Ben Taskar; S. V. N. Vishwanathan (eds.). Predicting Structured Data. Neural Information Processing series. MIT Press. ISBN 978-0-26202617-8.
6. ↑ Onal, Kezban Dilek; Zhang, Ye; Altingovde, Ismail Sengor; Rahman, Md Mustafizur; Karagoz, Pinar; Braylan, Alex; Dang, Brandon; Chang, Heng-Lu; Kim, Henna; McNamara, Quinten; Angert, Aaron (2018-06-01). "Neural information retrieval: at the end of the early years". Information Retrieval Journal. 21 (2): 111–182. doi:10.1007/s10791-017-9321-y. hdl:11245.1/008d6e8f-df13-4abf-8ae9-6ff2e17377f3. ISSN 1573-7659. S2CID 21684923.
7. 1 2 3 4 5 6 Boltzmann, Ludwig (1868). "Studien über das Gleichgewicht der lebendigen Kraft zwischen bewegten materiellen Punkten" [Studies on the balance of living force between moving material points]. Wiener Berichte (in German). 58: 517–560.
8. ↑ Bridle, John S. (1990a). Soulié F. F.; Hérault J. (eds.). Probabilistic Interpretation of Feedforward Classification Network Outputs, with Relationships to Statistical Pattern Recognition. Neurocomputing: Algorithms, Architectures and Applications (1989). NATO ASI Series (Series F: Computer and Systems Sciences). Vol. 68. Berlin, Heidelberg: Springer. pp. 227–236. doi:10.1007/978-3-642-76153-9_28.
9. ↑