3

I am training faster_rcnn_inception module for object detection on custom dataset. In training directory,we found folder called eval_0 and tensorflow generated events.out.tfevents.xxxxxx files.

Training Directory structure as follows

+training_dir
    +eval_0
     -events.out.tfevents.1542309785.instance-1  1.2GB
     -events.out.tfevents.1542367255.instance-1  5.3GB
     -events.out.tfevents.1542369886.instance-1  3.6GB
     -events.out.tfevents.1542624154.instance-1  31MB
     -events.out.tfevents.1543060258.instance-1  19MB
     -events.out.tfevents.1543066775.instance-2  1.6GB
 -events.out.tfevents.1542308099.instance-1  17MB
 -events.out.tfevents.1542308928.instance-1  17MB
 -events.out.tfevents.1542366369.instance-1  17MB
 -events.out.tfevents.1542369000.instance-1  17MB
 -events.out.tfevents.1542623262.instance-1  17MB
 -events.out.tfevents.1543064936.instance-2  17MB
 -events.out.tfevents.1543065796.instance-2  17MB
 -events.out.tfevents.1543065880.instance-2  17MB
 -model.ckpt-96004.data-00000-of-00001
 -model.ckpt-96004.data-00000-of-00001
 -model.ckpt-96004.index
 -model.ckpt-96004.meta
 -model.ckpt-96108.data-00000-of-00001
 -model.ckpt-96108.index
 -model.ckpt-96108.meta

As per my understanding, tfevents files in eval_0 folder are summery files of evaluation and tfevents files in training_dir are summery files of training.

I have interrupted training process several times and continued from recent checkpoint. I also understand restarting training process generates new tfevents files.

My Questions as follows:

  • Why training tfevents_files have same size, but in case if eval_0/tfevents_files size varies ?

  • Why interrupting training generates new tfevents_file in training folder, but same not observed in case of eval_0?

  • Can I delete all tfevents files in eval_0 except latest one? Does it affect on training or evolution history?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.