the.com/activation
The moment a neuron decides your input was actually worth mentioning.
means The act of making something active or operational, or the moment a process switches from dormant to triggered.
from From Latin 'activus' (active, fit for action), from 'agere' meaning to do or drive — the same root that gives us 'act' and 'agent.' The '-ation' suffix turns it into the noun naming the process itself: not just being active, but the switching-on that gets you there.
- math role
- Adds nonlinearity so networks can learn curves, not just lines
- without it
- Stacked layers collapse into one boring linear function
- popular picks
- ReLU, sigmoid, GELU all argue over who fires best
- dead neuron risk
- ReLU units can permanently switch off and stop learning