Researchers have developed a training procedure called Meta-learning for Compositionality (MLC) that teaches a standard transformer to combine known words in new ways, much as people do. In controlled tests, an MLC-trained network learned fresh vocabulary from only a handful of examples, then reused those words correctly in unfamiliar contexts. Humans reached about 81 percent perfect answers in the same tasks, and the MLC system matched or bettered that level, while GPT-4 lagged behind. Scientists say this “systematic generalization” addresses a classic criticism that neural networks merely mimic patterns without real understanding. Although far from a commercial product, the work hints that next-generation language models may move beyond today’s large chatbots by learning how to learn, rather than memorizing vast text.
Nature – Human-like systematic generalization through a meta-learning neural network – 25 Oct 2023
Brenden M. Lake and Marco Baroni introduced MLC, a curriculum of thousands of mini-tasks that forces a transformer to practice recombining words. After meta-training, the network shows both the flexibility of modern deep learning and the rule-like precision once thought unique to symbolic models, achieving human-level performance on several systematic-generalization benchmarks.


0 Comments