LSTM and GRU: Control What the Network Remembers
A useful memory needs more than the ability to keep everything.
LSTM and GRU: Control What the Network Remembers explained visually
A useful memory needs more than the ability to keep everything. English captions are optional and start off. The complete transcript follows the written lesson.
What you will learn
A useful memory needs more than the ability to keep everything.
Use gates to regulate information
A useful memory needs more than the ability to keep everything. It also needs to decide what can be forgotten and what should influence the next step. Long short-term memory networks, or LSTMs, introduce gates that regulate information flow through recurrent computation. Gated recurrent units, or GRUs, use a related but simpler design.
Compare memory designs
An LSTM maintains a cell state and uses gates to control forgetting, writing new information, and exposing information to the output state. A GRU combines memory handling differently, commonly using update and reset gates. These mechanisms create paths that can make learning longer dependencies easier than in a basic recurrent network. They do not guarantee unlimited memory or eliminate every training difficulty.
Test the actual sequence length
Choose between architectures by testing the data and constraints. Sequence length, training size, latency, and parameter count all matter. Use masking or packed-sequence handling appropriately for variable-length inputs, and reset state at the correct boundaries. Accidentally carrying one subject's state into another subject's sequence can corrupt both learning and evaluation. Inspect failures at different time horizons instead of reporting only an average score.
Worked example
A machine may show a brief warning pattern many steps before a failure. A gated network can learn to retain information about that pattern while ignoring less relevant fluctuations. Whether it actually learns that relationship must be checked on unseen sequences containing realistic delays and interruptions.
Quick check: test your understanding
Try it yourself: Why should hidden state be reset between unrelated recording sessions?
Check the reasoning
The new session should not inherit information from a previous independent sequence unless the application explicitly defines such continuity.
Do LSTMs guarantee that any event can be remembered indefinitely?
No. Capacity, optimization, data, and sequence length still limit performance.
Further reading
Dive into Deep Learning — Aston Zhang, Zachary C. Lipton, Mu Li, and Alexander J. Smola.
Deep Learning — Ian Goodfellow, Yoshua Bengio, and Aaron Courville.
Original ToolsInVision teaching examples. Editorial approach · Suggest a correction
Accessible referenceLesson 75 transcript240 words · Video transcript · Matches the published narration and captionsOpen transcript
A warning light flashes briefly. Several minutes later, a machine stops. To connect those events, a model needs to retain useful information while other readings continue to arrive.
This is where gated recurrent networks can help. Long short-term memory networks, usually called L S T Ms, add controls that manage information flow. Gated recurrent units, or G R Us, use a related design with fewer components.
An L S T M maintains a cell state alongside its hidden state. A forget gate controls how much earlier information to retain. An input gate controls what new information is written. An output gate helps determine what information becomes available in the hidden state.
These gates are learned numerical controls. They are not manually written instructions saying, remember this warning. The training objective shapes their behavior through examples.
A G R U combines memory handling differently, commonly using update and reset gates. Both designs can make some longer dependencies easier to learn than in a basic recurrent network. Neither gives the model perfect or unlimited memory.
For our machine example, evaluate warnings followed by failures after different delays. Also include warnings that do not lead to failure. Otherwise, the model may learn an overly simple association.
Choose between these architectures using held-out performance, memory requirements, and prediction speed. Reset state at the correct boundaries, and handle padded inputs carefully. Useful memory comes from good data and evaluation as well as the network design.