現在Kerasにてseq2seqモデルの構築を行なっているのですが、現在以下のエラーが発生しており学習が途中で止まってしまいます。
CancelledError: [_Derived_]RecvAsync is cancelled.
[[{{node RMSprop/RMSprop/update/ReadVariableOp_2/_33}}]]
[[concat/_30]] [Op:__inference_train_function_10864]
以下のコードでモデルを作成し、学習を行っています。
Model: "functional_1"
__________________________________________________________________________________________________
Layer (type) Output Shape Param # Connected to
==================================================================================================
decoder_input (InputLayer) [(None, 101)] 0
__________________________________________________________________________________________________
encoder_Input (InputLayer) [(None, 101)] 0
__________________________________________________________________________________________________
Embedding (Embedding) (None, 101, 800) 25238400 encoder_Input[0][0]
decoder_input[0][0]
__________________________________________________________________________________________________
encoder_lstm (LSTM) [(None, 256), (None, 1082368 Embedding[0][0]
__________________________________________________________________________________________________
decoder_lstm (LSTM) [(None, 101, 256), ( 1082368 Embedding[1][0]
encoder_lstm[0][1]
encoder_lstm[0][2]
__________________________________________________________________________________________________
dense (Dense) (None, 101, 31548) 8107836 decoder_lstm[0][0]
==================================================================================================
Total params: 35,510,972
Trainable params: 35,510,972
Non-trainable params: 0
__________________________________________________________________________________________________
embedding = Embedding(output_dim=vec_dim, input_dim=num_tokens, mask_zero=True, name='Embedding')
encoder_inputs = Input(shape=(max_seq_length,), name='encoder_Input')
e_i = embedding(encoder_inputs)
encoder = LSTM(latent_dim, return_state=True, name='encoder_lstm')
encoder_outputs, state_h, state_c = encoder(e_i)
encoder_states = [state_h, state_c]
decoder_inputs = Input(shape=(max_seq_length,), name='decoder_input')
d_i = embedding(decoder_inputs)
decoder_lstm = LSTM(latent_dim, return_sequences=True, return_state=True, name='decoder_lstm')
decoder_outputs, _, _ = decoder_lstm(d_i,
initial_state=encoder_states)
decoder_dense = Dense(num_tokens, activation='softmax')
decoder_outputs = decoder_dense(decoder_outputs)
model = Model([encoder_inputs, decoder_inputs], decoder_outputs)
model.compile(optimizer='rmsprop', loss='categorical_crossentropy', metrics=['acc'])
epochs = 100
for epoch in range(epochs):
print("=" * 50)
print(epoch, "/", epochs)
print("training start....")
batch = 10
count = 1
for encoder_input_data, decoder_input_data, decoder_target_data in get_batch(batch):
model.train_on_batch([encoder_input_data, decoder_input_data], decoder_target_data)
print(count, 'batch complete')
count += 1
現在これで14 batch completeまで出力されているのですが、それ以降上述のエラーが発生し学習が止まってしまっています。
まだ、初学者なので色々と見にくくなってしまっているかもしれません。
自分でも色々と調べてみたのですが、
TF_FORCE_GPU_ALLOW_GROWTH=true
を実行したところ変化がなく、
physical_devices = tf.config.list_physical_devices('GPU') tf.config.experimental.set_memory_growth(physical_devices[0], enable=True)
を実行すると以下のエラーが出てしまい、八方塞がりです...
RuntimeError: Physical devices cannot be modified after being initialized
何卒お力添えの程よろしくお願い致します。