activation_func

fast_gelu(x)

Fast approximation of the gelu function. May slightly decrease accuracy.

gelu(x)

Gaussian Error Linear Unit (GELU)

relu(x)

Relu activation function

tanh(x)

Tanh activation function