oneflow.optim.lr_scheduler.CosineAnnealingLR

class oneflow.optim.lr_scheduler.CosineAnnealingLR(optimizer: oneflow.nn.optimizer.optimizer.Optimizer, T_max: int, eta_min: float = 0.0, last_step: int = - 1, verbose: bool = False)

Set the learning rate of each parameter group using a cosine annealing schedule, where \(\eta_{max}\) is set to the initial lr and \(T_{cur}\) is the number of epochs since the last restart in SGDR:

\[\begin{split}\begin{aligned} \eta_t & = \eta_{min} + \frac{1}{2}(\eta_{max} - \eta_{min})\left(1 + \cos\left(\frac{T_{cur}}{T_{max}}\pi\right)\right), & T_{cur} \neq (2k+1)T_{max}; \\ \eta_{t+1} & = \eta_{t} + \frac{1}{2}(\eta_{max} - \eta_{min}) \left(1 - \cos\left(\frac{1}{T_{max}}\pi\right)\right), & T_{cur} = (2k+1)T_{max}. \end{aligned}\end{split}\]

When last_step=-1, sets initial lr as lr. Notice that because the schedule is defined recursively, the learning rate can be simultaneously modified outside this scheduler by other operators. If the learning rate is set solely by this scheduler, the learning rate at each step becomes:

\[\eta_t = \eta_{min} + \frac{1}{2}(\eta_{max} - \eta_{min})\left(1 + \cos\left(\frac{T_{cur}}{T_{max}}\pi\right)\right)\]

It has been proposed in SGDR: Stochastic Gradient Descent with Warm Restarts. Note that this only implements the cosine annealing part of SGDR, and not the restarts.

The documentation is referenced from: https://pytorch.org/docs/1.10/generated/torch.optim.lr_scheduler.CosineAnnealingLR.html.

Parameters

• optimizer (Optimizer) – Wrapped optimizer.

• T_max (int) – Maximum number of iterations.

• eta_min (float) – Minimum learning rate. Default: 0.

• last_step (int) – The index of last epoch. Default: -1.

• verbose (bool) – If True, prints a message to stdout for each update. Default: False.

__init__(optimizer: oneflow.nn.optimizer.optimizer.Optimizer, T_max: int, eta_min: float = 0.0, last_step: int = - 1, verbose: bool = False)

Initialize self. See help(type(self)) for accurate signature.

Methods

__delattr__(name, /)	Implement delattr(self, name).
__dir__()	Default dir() implementation.
__eq__(value, /)	Return self==value.
__format__(format_spec, /)	Default object formatter.
__ge__(value, /)	Return self>=value.
__getattribute__(name, /)	Return getattr(self, name).
__gt__(value, /)	Return self>value.
__hash__()	Return hash(self).
__init__(optimizer, T_max[, eta_min, …])	Initialize self.
__init_subclass__	This method is called when a class is subclassed.
__le__(value, /)	Return self<=value.
__lt__(value, /)	Return self<value.
__ne__(value, /)	Return self!=value.
__new__(**kwargs)	Create and return a new object.
__reduce__()	Helper for pickle.
__reduce_ex__(protocol, /)	Helper for pickle.
__repr__()	Return repr(self).
__setattr__(name, value, /)	Implement setattr(self, name, value).
__sizeof__()	Size of object in memory, in bytes.
__str__()	Return str(self).
__subclasshook__	Abstract classes can override this to customize issubclass().
_generate_conf_for_graph(lr_conf)
_init_base_lrs()
get_last_lr()	Return last computed learning rate by current scheduler.
get_lr(base_lr, step)	Compute learning rate using chainable form of the scheduler
load_state_dict(state_dict)	Load the schedulers state.
print_lr(group, lr)	Display the current learning rate.
state_dict()	Return the state of the scheduler as a dict.
step()
update_lrs(lrs)