This article briefly discusses what transformer is, the architecture of transformer, the working mechanism of encoder and decoder. In the later part we will briefly discuss what is self-attention and its working mechanism with the help of an example.

2023_04_MicrosoftTeams-image-283.jpg

You know ChatGPT which is a gp transformer. Today we will cover what transformers are. To make it easier for you to understand, we will first take you through NLP, based on which we will explain the architecture and working of transformers. And then, the working mechanism of self-attention.

So, let’s start!!

Table of Content

  • What is NLP?
  • What is Transformer?
    • Architecture of Transformer
  • How do Transformer work?
    • Encoder
    • Decoder
  • Attention and Self-Attention
    • Mechanism of Self-Attention
  • Limitation of Transformer

What is NLP?

NLP (or Natural Language Processing) is a branch of Artificial Intelligence and Linguistics that focuses on enabling computers to understand and interpret the language of human beings. It focuses on understanding every single word individually as well as its context.

The goal of the NLP is not just limited to understanding the language of humans but also generating human language meaningfully. It helps to bridge the gap between human communication and computer understanding.

Must Explore – Artificial Intelligence Courses

NLP uses various techniques and algorithms from computer science, linguistics (parts of speech), and machine learning to train itself and generate results. Some of the common examples of NLP include: 

  • Sentiment Analysis
  • Spam Detection
  • Chatbots
  • Machine Translation
  • Voice Assistants
  • Text Summarization

ChatGPT and Google Bard are one of the most prominent examples of NLP, which work as chatbots, and generate texts.

What is a Transformer?

Transformers are a special type of neural network that was first introduced in 2017 by Vaswani et al. in the Research Paper “Attention is All You Need”. And after that, these transformer models became the foundation of many state-of-the-art NLP models. Unlike the traditional Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM), transformers use self-attention that processes the input sequence in parallel, resulting in better scalability and faster training time.

Mainly transformer is made up of two components: Encoder and Decoder. The basic architecture of a Transformer looks like:

2023_04_MicrosoftTeams-image-285.jpg

Use of Transformers:

  • Transformers can be trained to translate text from one language to another (Google Translate).
  • It can determine the sentiment or emotion behind the piece of text (Twitter Sentiment Analysis)
  • Transformers can generate human-like text on any given input prompt (ChatGPT and Google Bard).

Now, we will discuss the architecture of the Transformer.

Architecture of Transformers

The architecture of Transformers consists of three main components:

Encoder processes input sequences and consists of identical layers. Each layer contains:

  • Self-attention mechanism
  • Position-Wise Feed-Forward Network

Decoder: It generates the output sequence, and similar to the encoder, it is also composed of identical layers. Each layer contains:

  • Mult-Head Self Attention Mechanism
  • Positions-Wise Feed-Forward Network
  • Encode-Decoder Attention Layer

Position Encoding: It is used to inject information about the tokens. Since the transformers can’t sense the position of tokens in the input sequence.

2023_04_MicrosoftTeams-image-273.jpg

Image Source: Research Paper: Attention is All You Need