the.com/dropout
The one regularization trick that works by making your network forget itself, on purpose.
means A person who leaves school or a program before finishing it — and in machine learning, a technique that randomly switches off neurons during training so the network can't lean too hard on any one of them.
from The human sense of 'dropout' emerged in mid-20th-century American education talk, built plainly from 'drop' + 'out' — someone who drops out of the race before the finish line. The machine-learning sense is a deliberate 2010s borrowing: researchers studying neural networks needed a name for randomly deactivating units during training, and reached for the everyday word for students vanishing from class rolls — a neuron, like a kid, just stops showing up.
- the mechanism
- randomly zeroes neurons each training step
- the logic
- forces redundancy so no neuron gets too comfortable
- the irony
- network trains worse to generalize better
- typical rate
- usually 20 to 50 percent of units dropped