[Kaggle Study] 12. Early Stopping

• Early stopping is a technique used while training neural networks to prevent the model from overfitting.

reference: https://cyborgcodes.medium.com/what-is-early-stopping-in-deep-learning-eeb1e710a3cf

• It basically stops the training process before the model starts to overfit.
  • We monitor the model's performance on a validation set during the training process and stop the training process when the general performance starts to degrade, which indicates that the model is beginning to overfit the training data.
• That is, the goal is to stop the training where the model performs the best.
• Disadvantage of this technique:
  • We have 2 big steps during training ml models:
    • 1. optimize cost function J using various algorithms such as Gradient Descent
    • 2. Make sure the model doesn't overfit
      • We can do this by collecting more data, apply regularization, ...
  • It is much better to focus on each task one at a time and this is possible these days due to advanced ml techniques & algorithms.
  • However, early stopping techniques couples these 2 big tasks which means each steps greatly affects each other.
  • We can no longer work on each task independently.
• Advantage of this technique:
  • Other regularization techniques such as L2 regularization requires hyperparameter especially lambda.
    • This means that we should take the computational cost to find the best hyperparameter value.
  • However, by using early stopping, we can process the regularization by simply training the model and find the weights that makes the model perform best.

You can find further information about this issue(disadvantage of early stopping) from my previous post:

[Kaggle Study] 7. About Structuring ML Projects (1)

 

Reference

심층 신경망 개선하기: 하이퍼파라미터 튜닝, 정규화 및 최적화

---

My success isn't the result of arrogance, it's the result of belief.
- Conor Mcgregor -