MultiModalMambaBlock¶

Table of Contents¶

Introduction
Fusion Method and Model Architecture
Usage and Examples
Further References

Introduction¶

The MultiModalMambaBlock is a PyTorch module designed to combine text and image embeddings using a multimodal fusion approach. It provides methods for attention-based fusion using a Mamba block, ViT encoder, and image/text embeddings. By using a variety of fusion methods, the MultiModalMambaBlock aims to facilitate the learning of joint representations from different modalities.

Fusion Method and Model Architecture¶

Args¶

Args	Description
`dim`	The dimension of the embeddings.
`depth`	The depth of the Mamba block.
`dropout`	The dropout rate.
`heads`	The number of attention heads.
`d_state`	The dimension of the state in the Mamba block.
`image_size`	The size of the input image.
`patch_size`	The size of the image patches.
`encoder_dim`	The dimension of the encoder embeddings.
`encoder_depth`	The depth of the encoder.
`encoder_heads`	The number of attention heads in the encoder.
`fusion_method`	The multimodal fusion method to use. Can be one of ["mlp", "concat", "add"].

Module Architecture¶

Mamba Block: Implements a transformer-like Mamba block for attention-based fusion of embeddings.
ViT Encoder: Utilizes a Vision Transformer encoder for image-based attention encoding.
Fusion Methods: Provides support for various fusion methods, including MLP fusion, concatenation, addition, and visual expert methods.

Usage and Examples¶

x = torch.randn(1, 16, 64)
y = torch.randn(1, 3, 64, 64)
model = MultiModalMambaBlock(
    dim=64,
    depth=5,
    dropout=0.1,
    heads=4,
    d_state=16,
    image_size=64,
    patch_size=16,
    encoder_dim=64,
    encoder_depth=5,
    encoder_heads=4,
    fusion_method="mlp",
)
out = model(x, y)
print(out.shape)

# Checking the current fusion method
model.check_fusion_method()

Further References¶

For additional information and detailed usage, please refer to the official documentation of the MultiModalMambaBlock module.

Note: The architecture and methods used in the MultiModalMambaBlock module are designed to address the specific challenge of joint attention-based multimodal representation learning. The selected fusion_method and fusion approach can significantly impact the model performance, and care should be taken when choosing the appropriate method for a particular use case.