Recurrent neural network (RNN)

What is Recurrent Neural Network (RNN)?

Recurrent neural network (RNN) – A recurrent neural network is a class of artificial neural networks where connections between nodes form a directed graph along a temporal sequence. This allows it to exhibit temporal dynamic behavior. Unlike feedforward neural networks, RNNs can use their internal state to process sequences of inputs.

Recurrent neural networks (RNN) are the state-of-the-art algorithm for sequential data and are used by Apple’s Siri and Google’s voice search. It is the first algorithm that remembers its input, due to an internal memory, which makes it perfectly suited for machine learning problems that involve sequential data. It is one of the algorithms behind the scenes of the amazing achievements seen in deep learning over the past few years. In this post, we will cover the basic concepts of how recurrent neural networks work, what the biggest issues are, and how to solve them.

RNNs are a powerful and robust type of neural network and belong to the most promising algorithms in use because it is the only one with internal memory.

Like many other deep learning algorithms, recurrent neural networks are relatively old. They were initially created in the 1980s, but only in recent years have we seen their true potential. An increase in computational power along with the massive amounts of data that we now must work with, and the invention of long short-term memory (LSTM) in the 1990s, has really brought RNNs to the foreground.

Because of their internal memory, RNN’s can remember important things about the input they received, which allows them to be very precise in predicting what is coming next. Therefore, they are the preferred algorithm for sequential data like time series, speech, text, financial data, audio, video, weather and much more. Recurrent neural networks can form a much deeper understanding of a sequence and its context compared to other algorithms.

RNN produces predictive results in sequential data that other algorithms cannot.

But when do you need to use an RNN?

“Whenever there is a sequence of data and that temporal dynamics that connect the data is more important than the spatial content of each individual frame.” – Lex Fridman (MIT)

Since RNNs are being used in the software behind Siri and Google Translate, recurrent neural networks show up a lot in everyday life.

HOW RNN WORKS

To understand RNNs properly, you will need a working knowledge of “normal “feed-forward neural networks and sequential data.

Sequential data is basically just ordered data in which related things follow each other. Examples are financial data or the DNA sequence. The most popular type of sequential data is perhaps time-series data, which is just a series of data points that are listed in time order.

RNN VS. FEED-FORWARD NEURAL NETWORKS

RNN’s and feed-forward neural networks get their names from the way they channel information.

In a feed-forward neural network, the information only moves in one direction from the input layer, through the hidden layers, to the output layer. The information moves straight through the network and never touches a node twice.

Feed-forward neural networks have no memory of the input they receive and are bad at predicting what is coming next. Because a feed-forward network only considers the current input, it has no notion of order in time. It simply cannot remember anything about what happened in the past except its training.

In an RNN the information cycles through a loop. When it decides, it considers the current input and what it has learned from the inputs it received previously.

A usual RNN has a short-term memory. In combination with an LSTM, they also have a long-term memory (more on that later).

Another good way to illustrate the concept of a recurrent neural network’s memory is to explain it with an example:

Imagine you have a normal feed-forward neural network and give it the word “neuron” as an input, and it processes the word character by character. By the time it reaches the character “r,” it has already forgotten about “n,” “e” and “u,” which makes it almost impossible for this type of neural network to predict which character would come next.

A recurrent neural network, however, can remember those characters because of its internal memory. It produces output, copies that output, and loops it back into the network.

RNN adds the immediate past to the present.

Therefore, an RNN has two inputs: the present and the recent past. This is important because the sequence of data contains crucial information about what is coming next, which is why an RNN can do things other algorithms cannot.

A feed-forward neural network assigns, like all other deep learning algorithms, a weight matrix to its inputs and then produces the output. Note that RNNs apply weights to the current and to the previous input. Furthermore, a recurrent neural network will also tweak the weights for both through gradient descent and backpropagation through time (BPTT).

Also note that while feed-forward neural networks map one input to one output, RNNs can map one to many, many to many (translation) and many to one (classifying a voice).

Conclusion

Is your company in need of help? MV3 Marketing Agency has numerous Marketing experts ready to assist you with AI. Contact MV3 Marketing to jump-start your business.

 

« Back to Glossary Index