Previously, I argued that emergent phenomena in machine learning mean that we canât rely on current trends to predict what the future of ML will be like. In this post, I will argue that despite this, empirical findings often do generalize very far, including across âphase transitionsâ caused by emergent behavior.
This might seem like a contradiction, but actually I think divergence from current trends and empirical generalization are consistent. Findings do often generalize, but you need to think to determine the right generalization, and also about what might stop any given generalization from holding.
I donât think many people would contest the claim that empirical investigation can uncover deep and generalizable truths. This is one of the big lessons of physics, and while some might attribute physicsâ success to math instead of empiricism, I think itâs clear that you need empirical data to point to the right mathematics.
However, just invoking physics isnât a good argument, because physical laws have fundamental symmetries that we shouldnât expect in machine learning. Moreover, we care specifically about findings that continue to hold up after some sort of emergent behavior (such as few-shot learning in the case of ML). So, to make my case, Iâll start by considering examples in deep learning that have held up in this way. Since âmodernâ deep learning hasnât been around that long, Iâll also look at examples from biology, a field that has been around for a relatively long time and where More Is Different is ubiquitous (see Appendix: More Is Different In Other Domains).
Source:
https://bounded-regret.ghost.io/empirical-findings-generalize-surprisingly-far/
