Recurrent Networks: Carry Information Through a Sequence
The words dog bites person and person bites dog contain the same words but mean different things.
Recurrent Networks: Carry Information Through a Sequence explained visually
The words dog bites person and person bites dog contain the same words but mean different things. English captions are optional and start off. The complete transcript follows the written lesson.
What you will learn
The words dog bites person and person bites dog contain the same words but mean different things.
Order changes the task
The words dog bites person and person bites dog contain the same words but mean different things. Order matters. A recurrent neural network processes a sequence while carrying a hidden state from one step to the next. That state acts as a learned summary of information encountered so far.
Reuse a state update
At each step, the network combines the current input with the previous hidden state. The same update parameters are reused across the sequence. Outputs can be produced at every step or after processing the whole sequence, depending on the task. This makes recurrent models useful for discussing time series, language, and other ordered data, although modern applications may use other architectures.
Recognize long-range difficulty
Training unfolds the repeated computation through time and propagates gradients through those steps. Long sequences can produce vanishing or exploding gradients, making distant relationships difficult to learn. Sequence boundaries and padding also need correct handling. When predicting future values, do not accidentally use information from later time steps that would be unavailable in deployment. A sequence model still requires a carefully designed evaluation split.
Worked example
To predict the next sensor reading, a recurrent model might process the previous ten readings in order. Reversing those readings changes the history presented to the model, even though the set of numerical values stays the same. That order can distinguish a rising trend from a falling one.
Quick check: test your understanding
Try it yourself: Why is randomly shuffling time steps within each training sequence usually inappropriate for forecasting?
Check the reasoning
It destroys the temporal structure the model is supposed to learn and can change the meaning of the prediction task.
Does a hidden state preserve every earlier observation perfectly?
No. It is a learned, limited representation and may lose information.
Further reading
Dive into Deep Learning — Aston Zhang, Zachary C. Lipton, Mu Li, and Alexander J. Smola.
Deep Learning — Ian Goodfellow, Yoshua Bengio, and Aaron Courville.
Original ToolsInVision teaching examples. Editorial approach · Suggest a correction
Accessible referenceLesson 74 transcript230 words · Video transcript · Matches the published narration and captionsOpen transcript
The phrases “dog bites person” and “person bites dog” contain the same words. Yet their meanings are very different. The order carries information.
A recurrent neural network, often called an R N N, handles a sequence one step at a time. Along with the current input, it uses a hidden state from the previous step. That state is a learned summary of what came before.
At each step, the same update rule combines the new input with the previous state. The updated state then moves forward. Depending on the task, the network may produce an output at every step or one output after the sequence ends.
Consider ten temperature readings from a machine. Reading them in their original order might show a rising temperature. Reverse the order, and the same values now show a falling trend. The values alone do not tell the full story.
The hidden state is not a perfect recording of the past. It has limited capacity, and training across many steps can create vanishing or exploding gradients. That makes some long-range relationships difficult to learn.
We also need clean sequence boundaries. One machine's history should not accidentally become the starting state for another. And when predicting the future, do not include later readings that would be unavailable at prediction time. A sequence model is only useful when its training and evaluation respect how time actually works.