WEveryone wishes they had minds as retentive as a hard drive. Memory file created. Saved. Ready to access at any time. But don’t go wishing for the AI’s memory performance just yet.
Artificial neural networks are prone to a problematic failure known, evocatively, as catastrophic forgetting. These seemingly tireless networks can keep up with learning tasks day and night. But sometimes, once a new task is learned, any memory of a previous task fades. It’s like you learned to play tennis decently well, but after being taught to play water polo, you suddenly don’t remember how to swing a racket.
This apparent network overload put an idea in the head of Maxim Bazhenov, a professor who studies computational neuroscience and sleep at the University of California San Diego School of Medicine. Perhaps the spiky neural networks he was working with simply needed a break.
In natural sleep, he had seen that the same basic brain processes occur in humans and bees, working on information accumulated during waking moments. “That machinery was presumably doing something useful” to be conserved along evolutionary pathways, she says. So, she thought, why not try a similar state for machines?
The idea was simply to give the artificial neural networks a break from external stimuli, to signal them to enter a kind of resting state. Like the sleepy human brain, the networks were still active, but instead of receiving new information, they were reflecting on the old, consolidating, surfacing patterns.
The networks did not seem to bear any resemblance to what was actually going on in a human brain.
And it worked. In a pair of documents in late 2022, Bazhenov and colleagues showed that providing periods of neural networks in a sleep-like state mitigated the risk of catastrophic forgetting.1, 2 It turns out that the brains of the machines also need rest.
The cognitive and computer psychologist Geoffrey Hinton had proposed the idea of allow early neural networks to take a nap in the 1990s.3 But the newer work applies the concept to radically more complex networks — in this case, what’s called spiking neural networks, which mimic the pattern of neurons firing in our brains. The new work also demonstrates the use of restful sleep not only to prevent catastrophic forgetting, but also to improve generalizability, both of which have implications for the true usefulness of these networks, from cybersecurity to self-driving cars (remember: and apply Asimov and traffic rules wisely).
Sleep, of course, is also essential for our own memory and learning.4 Idle time appears to strengthen new task-related connections forged in the brain during wakefulness and help transfer them to areas of the brain for long-term storage. Researchers have known for decades that while we may not suffer a complete episode of catastrophic forgetfulness, sleep deprivation interferes with our ability to efficiently learn new skills and retain memories.5 More recent research even suggests that we don’t need to shut down completely to improve our procedural memory. just resting quietly although it doesn’t search for new inputs, or as the researchers put it, engage in “offline memory consolidation,” it also appears to work for human brains.6
However, Robert Stickgold cautions here. Stickgold is a professor of psychiatry at the Brain Sciences Initiative at Harvard Medical School, where he studies sleep and cognition. Sure, it’s useful to say that we’re letting a network “sleep”. But it is advisable not to take the vocabulary too far. For our sake, or for the sake of advancing network research.
Stickgold recalls a conversation he had decades ago with an MIT researcher who was building the first artificial intelligence algorithms to solve complex business problems. Stickgold commented that the networks did not seem to bear any resemblance to what was actually happening in a human brain. To which his engineer interlocutor replied: “What would you want?”
Katherine Harmon Courage is deputy editor of Nautilus.
Lead Image: Space Wind / Shutterstock
References
1. Golden, R., Delanois, JE, Sanda, P., and Bazhenov, M. Sleep prevents catastrophic forgetfulness in active neural networks by forming a synaptic weight joint representation. PLOS Computational Biology 18e1010628 (2022).
2. Tadros T, Krishnan GP, Ramyaa R & Bazhenov M. Unsupervised sleep-like replay reduces catastrophic forgetting in artificial neural networks. News-thread communications 137742 (2022).
3. Hinton GE, Dayan P, Frey BJ, and Neal RM The wake-sleep algorithm for unsupervised neural networks. Science 2681158-1161 (1995).
4. Walker, MP and Stickgold, R. Sleep-dependent learning and memory consolidation. neuron 44121-133 (2004).
5. Maquet, P. The role of sleep in learning and memory. Science 2941048-1052 (2001).
6. Wang, S.Y. et al. “Sleep Dependent” Memory Consolidation? Brief post-training rest and sleep periods provide an equivalent benefit for both declarative and procedural memory. learning and memory 28195-203 (2021).
Get the Nautilus Newsletter
The newest and most popular articles delivered directly to your inbox!
