oneflow.nn.graph.graph_config.GraphConfig.enable_straighten_algorithm

GraphConfig.enable_straighten_algorithm(mode: str = 'MemoryFirst')

Whether enable the straighten algorithm.

straighten_algorithm_tag 1: Disable Disable the straighten algorithm in the task graph. Would use the original topography order for executing task nodes.

straighten_algorithm_tag 2: SpeedFirst Under the second configuration, the straighten algorithm would try to speed up the training as much as possible. If using nccl compute stream, setting the tag to 2 might not speed up the training. If not using nccl compute stream, setting the tag to 2 might speed up data parallelism by 0.6% and model parallelism by 6%. Considering memory, enabling the straighten algorithm is forbidden with one machine/device only, and not recommended under pipeline parallelism.

straighten_algorithm_tag 3: MemoryFirst Under the third configuration, the straighten algorithm would try to compress memory as much as possible. It might save up to 13% of the memory for some models. And might save nothing for some models.

straighten_algorithm_tag 4: OverlapCpuGpu Under the forth configuration, the straighten algorithm would try to run the cpu nodes and gpu nodes alternately. Such procedure would reduce the gaps of the execution on gpus. It might speed up the training by 2%. If no cpu nodes exist, the straighten_algorithm_tag would be switch to 3 automatically.

straighten_algorithm_tag 5: DelayShortGpu Under the fifth configuration, the straighten algorithm would try to delay the cpu nodes. Such procedure would reduce the gaps of the execution on gpus. It might speed up the validation (or training). If no cpu nodes exist, the straighten_algorithm_tag would be switch to 3 automatically.