python keras RAdam tutorial and load custom optimizer with CustomObjectScope
\<!--more-->

RAdam

usage

import keras
import numpy as np
from keras_radam import RAdam

# Build toy model with RAdam optimizer
model = keras.models.Sequential()
model.add(keras.layers.Dense(input_shape=(17,), units=3))
model.compile(RAdam(), loss='mse')

# Generate toy data
x = np.random.standard_normal((4096 * 30, 17))
w = np.random.standard_normal((17, 3))
y = np.dot(x, w)

# Fit
model.fit(x, y, epochs=5)

use warmup

from keras_radam import RAdam

RAdam(total_steps=10000, warmup_proportion=0.1, min_lr=1e-5)

load custom optimizer

keras load model with custom optimizer with CustomObjectScope

error

when load model with custom optimizer, eg RAdam()

model = load_model("resnet50_radam_model.h5")

output error

ValueError: Unknown optimizer: RAdam

solution

from keras_radam import RAdam
from keras.utils import CustomObjectScope

with CustomObjectScope({'RAdam': RAdam()}):
    best_model_filepath = "./checkpoint/best_model_efnb0.h5"
    model = load_model(best_model_filepath)
    model.save_weights("./checkpoint/weights_efnb0.h5")

Reference

• On the Variance of the Adaptive Learning Rate and Beyond
• RAdam official Pytorch Version
• https://github.com/CyberZHG/keras-radam

History

• 20190912: created.

Copyright

• Post author: kezunlin
• Post link: https://kezunlin.me/post/c691f02b/
• Copyright Notice: All articles in this blog are licensed under CC BY-NC-SA 3.0 unless stating additionally.