Skip to content

如何设置learning rate decay?Annealing the learning rate  #1167

Closed

Description

@OleNet

在optimizers.py里面,我看到了设置learning rate decay的参数, 不过有两个,learning_rate_decay_a=0.,
learning_rate_decay_b=0.,
请问这两个参数有什么区别呢、分别代表什么含义呢?我应该用哪一个呢?
似乎并没有相关的wiki、文档记录呢。

Activity

  1. self-assigned this
    on Jan 17, 2017
  2. reyoung commented on Jan 17, 2017

    @reyoung
    Collaborator

    也对,这里应该加一下注释。。我下午加一下。。

  3. OleNet commented on Jan 17, 2017

    @OleNet
    Author

    多谢多谢

  4. reyoung commented on Jan 17, 2017

    @reyoung
    Collaborator

    See #1170

    不过里面加了一些强类型的Wrapper,如果手动的设置这几个参数,就参考下这里的XXXLRS怎么实现的吧。

  5. lcy-seso commented on Jan 18, 2017

    @lcy-seso
    Collaborator

    简单的贴一下之前的文档,供参考:

    • "learning_rate":学习率
    • "learning_rate_a"和"learning_rate_b":学习率衰减参数,具体衰减公式由learning_rate_schedule决定
      "learning_rate_schedule":配置不同的学习率递减模式,包括:
    • 1), "constant": lr = learning_rate
    • 2), "poly": lr = learning_rate * pow(1 + learning_rate_decay_a * num_samples_processed, -learning_rate_decay_b)
    • 3), "exp": lr = learning_rate * pow(learning_rate_decay_a, num_samples_processed / learning_rate_decay_b)
    • 4), "discexp": lr = learning_rate * pow(learning_rate_decay_a, floor(num_samples_processed / learning_rate_decay_b))
    • 5), "linear": lr = max(learning_rate - learning_rate_decay_a * num_sample_passed, learning_rate_decay_b)

    特别地,自适应学习率算法 adagrad, adadelta, rmsprop 不能通过learning_rate_schedule进行学习率衰减,momentum 可以通过learning_rate_schedule进行学习率衰减

  6. lcy-seso commented on Jan 18, 2017

    @lcy-seso
    Collaborator

    @qingqing01 提醒说,代码里面目前的逻辑 learning_rate_schedule 也会修改 adagrad, adadelta, rmsprop 的全局学习率。修正上一条的说法。

  7. THUHJ commented on Jul 11, 2017

    @THUHJ

    您好,请问num_samples_processed具体是训练了多少个sample嘛?还是多少个batch?

    然后第五个 "linear": lr = max(learning_rate - learning_rate_decay_a, learning_rate_decay_b) ,lr与num_samples_processed无关,不随着训练变化的嘛?

    另外有没有能从optimizer得到当前的learing rate的方法呢?我看了一下code,并没有找到接口。使用optimizer.dict["opt_conf_proto"].learning_rate 也只能获得最开始初始的lr.

  8. lcy-seso commented on Jul 19, 2017

    @lcy-seso
    Collaborator
    1. num_samples_processed 是到目前为止处理了多少训练样本。
    2. 线性decay时,检查了代码,你是正确的,以上的回复有误。计算策略如下:
      max(learning_rate - lr_decay_a * num_sample_passed, lr_decay_b);
    3. 目前无法查看当前学习率, 学习率是一个矩阵,v2 目前还没有支持输出学习率统计量的功能。0.9.0时,设置 show_parameter_stats_period参数,单机运行时会输出学习率的统计量,例如均值,最大值,最小值等。
  9. THUHJ commented on Jul 19, 2017

    @THUHJ

    好的~谢谢!

  10. added a commit that references this issue on Feb 23, 2024
  11. added a commit that references this issue on Feb 20, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions