Welcome to Modalities’ documentation!
We propose a novel training framework for Multimodal Large Language Models (LLMs) that prioritizes code readability and efficiency. The codebase adheres to the principles of “clean code,” minimizing Lines of Code (LoC) while maintaining extensibility. A single, comprehensive configuration file enables easy customization of various model and training parameters.
A key innovation is the adoption of a PyTorch-native training loop integrated with the Fully Sharded Data Parallelism (FSDP) technique. FSDP optimizes memory usage and training speed, enhancing scalability for large-scale multimodal models. By leveraging PyTorch’s native capabilities, our framework simplifies the development process and promotes ease of maintenance.
The framework’s modular design facilitates experimentation with different multimodal architectures and training strategies. Users can seamlessly integrate diverse datasets and model components, allowing for comprehensive exploration of multimodal learning tasks. The combination of clean code, minimal configuration, and PyTorch-native training with FSDP contributes to a user-friendly and efficient platform for developing state-of-the-art multimodal language models.
Note
This project is under active development.
Getting Started
Entrypoints
VSCode Setup
Future Work
API
- modalities
- modalities package
- Subpackages
- modalities.checkpointing package
- Subpackages
- Submodules
- modalities.checkpointing.checkpoint_conversion module
- modalities.checkpointing.checkpoint_loading module
- modalities.checkpointing.checkpoint_saving module
- modalities.checkpointing.checkpoint_saving_execution module
- modalities.checkpointing.checkpoint_saving_instruction module
- modalities.checkpointing.checkpoint_saving_strategies module
- Module contents
- modalities.config package
- modalities.dataloader package
- Submodules
- modalities.dataloader.create_index module
- modalities.dataloader.create_packed_data module
- modalities.dataloader.dataloader module
- modalities.dataloader.dataloader_factory module
- modalities.dataloader.dataset module
- modalities.dataloader.dataset_factory module
- modalities.dataloader.large_file_lines_reader module
- modalities.dataloader.samplers module
- Module contents
- modalities.inference package
- modalities.logging_broker package
- modalities.models package
- Subpackages
- Submodules
- modalities.models.model module
- modalities.models.model_factory module
- modalities.models.utils module
- Module contents
- modalities.nn package
- modalities.optimizers package
- modalities.registry package
- modalities.running_env package
- modalities.tokenization package
- modalities.training package
- Subpackages
- Submodules
- modalities.training.activation_checkpointing module
- modalities.training.training_progress module
TrainingProgressTrainingProgress.num_seen_steps_current_runTrainingProgress.num_seen_steps_previous_runTrainingProgress.num_seen_steps_totalTrainingProgress.num_seen_tokens_current_runTrainingProgress.num_seen_tokens_previous_runTrainingProgress.num_seen_tokens_totalTrainingProgress.num_target_stepsTrainingProgress.num_target_tokens
- Module contents
- modalities.checkpointing package
- Submodules
- modalities.api module
- modalities.batch module
- modalities.evaluator module
- modalities.exceptions module
- modalities.gym module
- modalities.loss_functions module
- modalities.trainer module
- modalities.util module
- Module contents
- Subpackages
- modalities package