SIGMOID FUNCTION OPTIMIZATION

The definition of the sigmoid function is

$sigmoid\left ( x \right )=\frac{1}{1+e^{-x}}$

In Python code, it would be sth. like below shows.

import math
def sigmoid(x):
	return 1 / (1 + math.exp(-x))

Because it involves a division and a exponent operation, it will cost a certain amount of computing time during the training and prediction process of a neural network. However, since we often use the output of the sigmoid function in a comparison of a given threshold T like

if sigmoid(x) > T:
	... do something

we can rewrite the sigmoid formula like below.

$\begin{aligned} f\left ( x \right )&=y=\frac{1}{1+e^{-x}} \\ &\Rightarrow\frac{1}{y}=1+e^{-x} \\ &\Rightarrow\frac{1}{y}-1=e^{-x} \\ &\Rightarrow ln\left ( \frac{1}{y} -1 \right )=-x \\ &\Rightarrow-ln\left ( \frac{1}{y} -1 \right )=x \end{aligned}$

Now we get the inverse function of $f(x)$.

$f^{-1}\left ( x \right )=-ln\left ( \frac{1}{y} -1 \right )$

Therefore, in terms of the if statement above, we can take the inverse on the two sides of >. The left side would be x, and the right side would be a desigmoid threshold $-ln\left ( \frac{1}{T} -1 \right )$ which is a constant as well. So, in the end, the above if statement can be rewrited to

if x > desigmoid_T:
	... do something

This will significantly increase the performance of the if statement.

---

下一篇： CODE REVIEW OF LANGUAGE TRANSLATION WITH NN.TRANSFORMER AND TORCHTEXT - PART 2→