Skip to content

get_sinusoid_encoding_table

Module Name: get_sinusoid_encoding_table

def get_sinusoid_encoding_table(n_position, d_hid):

This module is designed to create a sinusoidal encoding table used to encode sequential time-specific information into the data input to a sequence-processing model, such as a Recurrent Neural Network (RNN) or a Transformer model.

The get_sinusoid_encoding_table function generates a sinusoidal encoding table. It uses a mathematical trick that constructs positional encodings as a sum of sine and cosine functions that can be computed in O(1) space and time, which allows the model to extrapolate to sequence lengths longer than the ones encountered during training.

Parameters
n_position (int) The number of positions for which the encoding is generated. It represents the maximum length of the sequence that can be handled by the model.
d_hid (int) The dimension of the hidden state of the model. This value denotes the size of the embeddings that will be supplied to the model.

For get_position_angle_vec function:

Argument Description
position (int) The current position for which the angles are being calculated.

Functionality and Usage

The function get_sinusoid_encoding_table generates an encoding table that uses sine and cosine functions. This encoding enables the model to identify the positional information of elements in a sequence.

The table is created by applying sine to even indices and cosine to odd indices in the array, and then calculating the positional and angle vectors for each position.

Here's an example of how this function can be used:

import numpy as np
import torch


def get_sinusoid_encoding_table(n_position, d_hid):
    def get_position_angle_vec(position):
        return [
            position / np.power(10000, 2 * (hid_j // 2) / d_hid)
            for hid_j in range(d_hid)
        ]

    sinusoid_table = np.array(
        [get_position_angle_vec(pos_i) for pos_i in range(n_position)]
    )
    sinusoid_table[:, 0::2] = np.sin(sinusoid_table[:, 0::2])  # dim 2i
    sinusoid_table[:, 1::2] = np.cos(sinusoid_table[:, 1::2])  # dim 2i+1

    return torch.FloatTensor(sinusoid_table).unsqueeze(0)


n_position = 10
d_hid = 64

print(get_sinusoid_encoding_table(n_position, d_hid))

In this example, we're creating a sinusoidal encoding table for a sequence length (n_position) of 10 and a hidden state size (d_hid) of 64. The output would be a sinusoidal table encoded as a torch tensor.

Additional information and tips

The sinusoidal encoding table is often used in attention-based models like the Transformer, where it helps the model understand relative positions of elements in the sequence. This trick is essential because in a Transformer model, unlike RNNs and CNNs, there’s no inherent notion of position.

References and resources