class oneflow.optim.Adagrad(params: Union[Iterator[oneflow.nn.Parameter], List[Dict]], lr: float = 0.001, lr_decay: float = 0.0, weight_decay: float = 0, initial_accumulator_value: float = 0.0, eps: float = 1e-10)

The formula is:

\begin{align}\begin{aligned}& S_{t} = S_{t-1} + grad \odot grad\\& decay\_lr = \frac{learning\_rate}{(1 + (train\_step - 1) * lr\_decay)}\\& X_{t} = X_{t-1} - \frac{decay\_lr}{\sqrt{S_{t} + \epsilon}} \odot grad\end{aligned}\end{align}
Parameters
• params (Union[Iterator[Parameter], List[Dict]]) – iterable of parameters to optimize or dicts defining

• groups (parameter) –

• lr (float, optional) – The learning rate. Defaults to 0.001.

• lr_decay (float, optional) – The decay factor of learning rate. Defaults to 0.0.

• weight_decay (float, optional) – The weight decay. Defaults to 0.

• initial_accumulator_value (float, optional) – The initial value of S. Defaults to 0.0.

• eps (float, optional) – A small constant terms added to the denominator to improve numerical stability. Defaults to 1e-10.

For example:

Example 1:

# Assume net is a custom model.

for epoch in range(epochs):
# Read data, Compute the loss and so on.
# ...
loss.backward()


Example 2:

# Assume net is a custom model.
[
{
"params": net.parameters(),
"lr": learning_rate,
}
],
)

for epoch in range(epochs):
# Read data, Compute the loss and so on.
# ...
loss.backward()


If you want to use clip_grad, you can refer this example.

For more details of clip_grad_max_norm and clip_grad_norm_type, you can refer to oneflow.nn.utils.clip_grad_norm_().

__init__(params: Union[Iterator[oneflow.nn.Parameter], List[Dict]], lr: float = 0.001, lr_decay: float = 0.0, weight_decay: float = 0, initial_accumulator_value: float = 0.0, eps: float = 1e-10)

Initialize self. See help(type(self)) for accurate signature.

Methods

 __delattr__(name, /) Implement delattr(self, name). __dir__() Default dir() implementation. __eq__(value, /) Return self==value. __format__(format_spec, /) Default object formatter. __ge__(value, /) Return self>=value. __getattribute__(name, /) Return getattr(self, name). __gt__(value, /) Return self>value. __hash__() Return hash(self). __init__(params[, lr, lr_decay, …]) Initialize self. __init_subclass__ This method is called when a class is subclassed. __le__(value, /) Return self<=value. __lt__(value, /) Return self

Attributes

 support_sparse Whether the Optimizer support sparse update.