Spiking neural network

Spiking neural networks (SNNs) are artificial neural networks (ANN) that mimic natural neural networks.^[1] These models leverage timing of discrete spikes as the main information carrier.^[2]

The insect is controlled by a spiking neural network to find a target in an unknown terrain.

In addition to neuronal and synaptic state, SNNs incorporate the concept of time into their operating model. The idea is that neurons in the SNN do not transmit information at each propagation cycle (as it happens with typical multi-layer perceptron networks), but rather transmit information only when a membrane potential—an intrinsic quality of the neuron related to its membrane electrical charge—reaches a specific value, called the threshold. When the membrane potential reaches the threshold, the neuron fires, and generates a signal that travels to other neurons which, in turn, increase or decrease their potentials in response to this signal. A neuron model that fires at the moment of threshold crossing is also called a spiking neuron model.^[3]

While spike rates can be considered the analogue of the variable output of a traditional ANN,^[4] neurobiology research indicated that high speed processing cannot be performed solely through a rate-based scheme. For example humans can perform an image recognition task requiring no more than 10ms of processing time per neuron through the successive layers (going from the retina to the temporal lobe). This time window is too short for rate-based encoding. The precise spike timings in a small set of spiking neurons also has a higher information coding capacity compared with a rate-based approach.^[5]

The most prominent spiking neuron model is the leaky integrate-and-fire model.^[6] In that model, the momentary activation level (modeled as a differential equation) is normally considered to be the neuron's state, with incoming spikes pushing this value higher or lower, until the state eventually either decays or—if the firing threshold is reached—the neuron fires. After firing, the state variable is reset to a lower value.

Various decoding methods exist for interpreting the outgoing spike train as a real-value number, relying on either the frequency of spikes (rate-code), the time-to-first-spike after stimulation, or the interval between spikes.

History

See also: Biological neuron model

Pulsed neuron model

Artificial synapses based on FTJs

Many multi-layer artificial neural networks are fully connected, receiving input from every neuron in the previous layer and signalling every neuron in the subsequent layer. Although these networks have achieved breakthroughs, they do not match biological networks and do not mimic neurons.^{[citation needed]}

The biology-inspired Hodgkin–Huxley model of a spiking neuron was proposed in 1952. This model described how action potentials are initiated and propagated. Communication between neurons, which requires the exchange of chemical neurotransmitters in the synaptic gap, is described in models such as the integrate-and-fire model, FitzHugh–Nagumo model (1961–1962), and Hindmarsh–Rose model (1984). The leaky integrate-and-fire model (or a derivative) is commonly used as it is easier to compute than Hodgkin–Huxley.^[7]

While the notion of an artificial spiking neural network became popular only in the twenty-first century, ^[8][9][10] studies between 1980 and 1995 supported the concept. The first models of this type of ANN appeared to simulate non-algorithmic intelligent information processing systems.^[11][12][13] However, the notion of the spiking neural network as a mathematical model was first worked on in the early 1970s.^[14]

As of 2019 SNNs lagged behind ANNs in accuracy, but the gap is decreasing, and has vanished on some tasks.^[15]

Underpinnings

Information in the brain is represented as action potentials (neuron spikes), which may group into spike trains or coordinated waves. A fundamental question of neuroscience is to determine whether neurons communicate by a rate or temporal code.^[16] Temporal coding implies that a single spiking neuron can replace hundreds of hidden units on a conventional neural net.^[1]

SNNs define a neuron's current state as its potential (possibly modeled as a differential equation).^[17] An input pulse causes the potential to rise and then gradually decline. Encoding schemes can interpret these pulse sequences as a number, considering pulse frequency and pulse interval.^[18] Using the precise time of pulse occurrence, a neural network can consider more information and offer better computing properties.^[19]

SNNs compute in the continuous domain. Such neurons test for activation only when their potentials reach a certain value. When a neuron is activated, it produces a signal that is passed to connected neurons, accordingly raising or lowering their potentials.

The SNN approach produces a continuous output instead of the binary output of traditional ANNs. Pulse trains are not easily interpretable, hence the need for encoding schemes. However, a pulse train representation may be more suited for processing spatiotemporal data (or real-world sensory data classification).^[20] SNNs connect neurons only to nearby neurons so that they process input blocks separately (similar to CNN using filters). They consider time by encoding information as pulse trains so as not to lose information. This avoids the complexity of a recurrent neural network (RNN). Impulse neurons are more powerful computational units than traditional artificial neurons.^[21]

SNNs are theoretically more powerful than so called "second-generation networks" defined as ANNs "based on computational units that apply activation function with a continuous set of possible output values to a weighted sum (or polynomial) of the inputs"; however, SNN training issues and hardware requirements limit their use. Although unsupervised biologically inspired learning methods are available such as Hebbian learning and STDP, no effective supervised training method is suitable for SNNs that can provide better performance than second-generation networks.^[21] Spike-based activation of SNNs is not differentiable, thus gradient descent-based backpropagation (BP) is not available.

SNNs have much larger computational costs for simulating realistic neural models than traditional ANNs.^[22]

Pulse-coupled neural networks (PCNN) are often confused with SNNs. A PCNN can be seen as a kind of SNN.

Researchers are actively working on various topics. The first concerns differentiability. The expressions for both the forward- and backward-learning methods contain the derivative of the neural activation function which is not differentiable because a neuron's output is either 1 when it spikes, and 0 otherwise. This all-or-nothing behavior disrupts gradients and makes these neurons unsuitable for gradient-based optimization. Approaches to resolving it include:

• resorting to entirely biologically inspired local learning rules for the hidden units
• translating conventionally trained “rate-based” NNs to SNNs
• smoothing the network model to be continuously differentiable
• defining an SG (Surrogate Gradient) as a continuous relaxation of the real gradients

The second concerns the optimization algorithm. Standard BP can be expensive in terms of computation, memory, and communication and may be poorly suited to the hardware that implements it (e.g., a computer, brain, or neuromorphic device).^[23]

Incorporating additional neuron dynamics such as Spike Frequency Adaptation (SFA) is a notable advance, enhancing efficiency and computational power.^[6][24] These neurons sit between biological complexity and computational complexity.^[25] Originating from biological insights, SFA offers significant computational benefits by reducing power usage,^[26] especially in cases of repetitive or intense stimuli. This adaptation improves signal/noise clarity and introduces an elementary short-term memory at the neuron level, which in turn, improves accuracy and efficiency.^[27] This was mostly achieved using compartmental neuron models. The simpler versions are of neuron models with adaptive thresholds, are an indirect way of achieving SFA. It equips SNNs with improved learning capabilities, even with constrained synaptic plasticity, and elevates computational efficiency.^[28][29] This feature lessens the demand on network layers by decreasing the need for spike processing, thus lowering computational load and memory access time—essential aspects of neural computation. Moreover, SNNs utilizing neurons capable of SFA achieve levels of accuracy that rival those of conventional ANNs,^[30][31] while also requiring fewer neurons for comparable tasks. This efficiency streamlines the computational workflow and conserves space and energy, while maintaining technical integrity. High-performance deep spiking neural networks can operate with 0.3 spikes per neuron.^[32]

Applications

SNNs can in principle be applied to the same applications as traditional ANNs.^[33] In addition, SNNs can model the central nervous system of biological organisms, such as an insect seeking food without prior knowledge of the environment.^[34] Due to their relative realism, they can be used to study biological neural circuits. Starting with a hypothesis about the topology of a biological neuronal circuit and its function, recordings of this circuit can be compared to the output of a corresponding SNN, evaluating the plausibility of the hypothesis. SNNs lack effective training mechanisms, which can complicate some applications, including computer vision.

When using SNNs for image based data, the images need to be converted into binary spike trains.^[35] Types of encodings include:^[36]

• Temporal coding; generating one spike per neuron, in which spike latency is inversely proportional to the pixel intensity.
• Rate coding: converting pixel intensity into a spike train, where the number of spikes is proportional to the pixel intensity.
• Direct coding; using a trainable layer to generate a floating-point value for each time step. The layer converts each pixel at a certain time step into a floating-point value, and then a threshold is used on the generated floating-point values to pick either zero or one.
• Phase coding; encoding temporal information into spike patterns based on a global oscillator.
• Burst coding; transmitting spikes in bursts, increasing communication reliability.

Software

A diverse range of application software can simulate SNNs. This software can be classified according to its uses:

SNN simulation

Unsupervised learning with ferroelectric synapses

These simulate complex neural models. Large networks usually require lengthy processing. Candidates include:^[37]

• Brian – developed by Romain Brette and Dan Goodman at the École Normale Supérieure;
• GENESIS (the GEneral NEural SImulation System^[38]) – developed in James Bower's laboratory at Caltech;
• NEST – developed by the NEST Initiative;
• NEURON – mainly developed by Michael Hines, John W. Moore and Ted Carnevale in Yale University and Duke University;
• RAVSim (Runtime Tool) ^[39] – mainly developed by Sanaullah in Bielefeld University of Applied Sciences and Arts;

Hardware

Predicting STDP learning with ferroelectric synapses

Neuron-to-neuron mesh routing model

Sutton and Barton proposed that future neuromorphic architectures^[40] will comprise billions of nanosynapses, which require a clear understanding of the accompanying physical mechanisms. Experimental systems based on ferroelectric tunnel junctions have been used to show that STDP can be harnessed from heterogeneous polarization switching. Through combined scanning probe imaging, electrical transport and atomic-scale molecular dynamics, conductance variations can be modelled by nucleation-dominated domain reversal. Simulations showed that arrays of ferroelectric nanosynapses can autonomously learn to recognize patterns in a predictable way, opening the path towards unsupervised learning.^[41]

Unsupervised learning with ferroelectric synapses

• Akida is a completely digital event-based neural processing device with 1.2 million artificial neurons and 10 billion artificial synapses developed by BrainChip. Utilizing event-based possessing, it analyzes essential inputs at specific points. Results are stored in on-chip memory units.
• AltAI-1^[42] is a Russian spike-based neuromorphic processor.
• Neurogrid is a board that can simulate spiking neural networks directly in hardware.
• SpiNNaker (Spiking Neural Network Architecture) uses ARM processors as the building blocks of a massively parallel computing platform based on a six-layer thalamocortical model.^[43] The system is based on numerical models running in real time on custom digital multicore chips. It uses custom chips, each with eighteen cores and a shared local 128 Mbyte RAM, with a total of over 1,000,000 cores.^[44] A single chip can simulate 16,000 neurons with eight million plastic synapses running in real time.^[45]
• Tianjic^[46] is a Chinese, hybrid, spike-based and continuous neural computation processor.
• TrueNorth is a processor that contains 5.4 billion transistors that consumes only 70 milliwatts; most processors contain about 1.4 billion transistors and require 35 watts or more. IBM refers to the design principle behind TrueNorth as neuromorphic computing.^[47]

Benchmarks

Classification capabilities of spiking networks trained according to unsupervised learning methods^[48] have been tested on benchmark datasets such as Iris, Wisconsin Breast Cancer or Statlog Landsat dataset.^[49][50] Various approaches to information encoding and network design have been used such as a 2-layer feedforward network for data clustering and classification. Based on Hopfield (1995) the authors implemented models of local receptive fields combining the properties of radial basis functions and spiking neurons to convert input signals having a floating-point representation into a spiking representation.^[51][52]

See also

• CoDi
• Cognitive architecture
• Cognitive map
• Cognitive computer
• Computational neuroscience
• Neural coding
• Neural correlate
• Neural decoding
• Neuroethology
• Neuroinformatics
• Models of neural computation
• Motion perception
• Systems neuroscience