PositionInterpolationEmbeddings

Overview

PositionalEmbedding module that uses interpolation to generate positional embeddings.

Parameters

Parameter	Description	Default
`dim`	Dimension of the model.	`None`
`max_positions`	Maximum length of the input sequence.	`2048`
`base`	Base value for interpolation.	`10000`
`device`	Device to use.	`None`

Examples

import torch

from zeta.nn import PositionInterpolationEmbeddings

positional_embedding = PositionInterpolationEmbeddings(512, 1000)
x = torch.randn(32, 100, 512)
positions = torch.arange(100)
embedded_tensor = positional_embedding(x, positions)

Description

The PositionInterpolationEmbeddings class is used to generate positional embeddings for input sequences using interpolation. It is often used in neural network models for natural language processing tasks.

Parameters

dim (int, optional): Dimension of the model. This parameter specifies the dimension of the positional embeddings. Defaults to None.
max_positions (int, optional): Maximum length of the input sequence. This parameter determines the maximum number of positions for which positional embeddings will be generated. Defaults to 2048.
base (int, optional): Base value for interpolation. This parameter controls the interpolation behavior for generating positional embeddings. Defaults to 10000.
device (str or torch.device, optional): Device to use for computation. This parameter specifies the device on which the positional embeddings will be computed. Defaults to None.

Example

positional_embedding = PositionInterpolationEmbeddings(512, 1000)
x = torch.randn(32, 100, 512)
positions = torch.arange(100)
embedded_tensor = positional_embedding(x, positions)

In this example, a PositionInterpolationEmbeddings instance is created with a dimension of 512 and a maximum position of 1000. The x tensor represents input data of shape (32, 100, 512), and positions is a tensor containing position indices. The embedded_tensor will contain positional embeddings for the input data.

For more details on the usage of this module, refer to the example provided.

Methods

`forward(x, seq_len=None)`

Generate positional embeddings for the input data.

x (Tensor): Input data of shape (batch_size, sequence_length, dimension).
seq_len (int, optional): Length of the input sequence. This parameter can be used to specify the length of the sequence for which positional embeddings should be generated. If not provided, the maximum length specified during initialization is used.

Returns a tuple containing two tensors: (cosine_embeddings, sine_embeddings). These tensors represent the positional embeddings for the input sequence. ```