oneflow.nn.QuickGELU¶
-
class
oneflow.nn.
QuickGELU
→ Tensor¶ Applies GELU approximation that is fast but somewhat inaccurate. See: https://github.com/hendrycks/GELUs
\[\text{QuickGELU}(x) = x * \sigma(1.702x) = x * \frac{1}{1 + \exp(-1.702x)}\]- Parameters
input (oneflow.Tensor) – Input Tensor
- Returns
A Tensor has same shape as the input.
- Return type
For example:
>>> import oneflow as flow >>> input = flow.Tensor([-0.5, 0, 0.5]) >>> gelu = flow.nn.QuickGELU() >>> out = gelu(input) >>> out tensor([-0.1496, 0.0000, 0.3504], dtype=oneflow.float32)