16.10 Tuning Deep Learning Models

• Testing accuracy and validation accuracy were significantly less than the training accuracy
  • This is usually due to overfitting, so we have room for improvement

    https://towardsdatascience.com/deep-learning-overfitting-846bf5b35e24
    https://hackernoon.com/memorizing-is-not-learning-6-tricks-to-prevent-overfitting-in-machine-learning-820b091dc42

• Each epoch's output shows training and validation accuracy increasing
  • Possible we have not yet trained enough

16.10 Tuning Deep Learning Models (cont.)

• Some variables that affect your model performance:
  • having more or less data to train with
  • having more or less data to test with
  • having more or less data to validate with
  • having more or fewer layers
  • the types of layers you use
  • the order of the layers

16.10 Tuning Deep Learning Models (cont.)

• Some things we could tune include:
  • trying different amounts of training data—we used only the top 10,000 words
  • different numbers of words per review—we used only 200
  • different numbers of neurons in our layers
  • more layers
  • loading pre-trained word vectors rather than learning them from scratch

16.10 Tuning Deep Learning Models (cont.)

• The compute time required to train models multiple times is significant so, in deep learning, you generally do not tune hyperparameters with techniques like k-fold cross-validation

  https://www.quora.com/Is-cross-validation-heavily-used-in-deep-learning-or-is-it-too-expensive-to-be-used

• One promising area for tuning is automated machine learning (AutoML)

  https://towardsdatascience.com/what-are-hyperparameters-and-how-to-tune-the-hyperparameters-in-a-deep-neural-network-d0604917584a
  https://medium.com/machine-learning-bites/deeplearning-series-deep-neural-networks-tuning-and-optimization-39250ff7786d
  https://flyyufelix.github.io/2016/10/03/fine-tuning-in-keras-part1.html
  https://flyyufelix.github.io/2016/10/08/fine-tuning-in-keras-part2.html
  https://towardsdatascience.com/a-comprehensive-guide-on-how-to-fine-tune-deep-neural-networks-using-keras-on-google-colab-free-daaaa0aced8f

• Auto-Keras is geared to automatically choosing the best Keras model configurations
• Others include Google’s Cloud AutoML and Baidu’s EZDL

---

©1992–2020 by Pearson Education, Inc. All Rights Reserved. This content is based on Chapter 5 of the book Intro to Python for Computer Science and Data Science: Learning to Program with AI, Big Data and the Cloud.

DISCLAIMER: The authors and publisher of this book have used their best efforts in preparing the book. These efforts include the development, research, and testing of the theories and programs to determine their effectiveness. The authors and publisher make no warranty of any kind, expressed or implied, with regard to these programs or to the documentation contained in these books. The authors and publisher shall not be liable in any event for incidental or consequential damages in connection with, or arising out of, the furnishing, performance, or use of these programs.