modalities.training package

Subpackages

• modalities.training.activation_checkpointing package
  • Submodules
  • modalities.training.activation_checkpointing.activation_checkpointing module
    • ActivationCheckpointing
      • ActivationCheckpointing.SAVE_DICT
      • ActivationCheckpointing.apply_activation_checkpointing_()
    • apply_activation_checkpointing_fsdp1_inplace()
    • is_module_to_apply_activation_checkpointing()
  • modalities.training.activation_checkpointing.activation_checkpointing_variants module
    • ActivationCheckpointingVariants
      • ActivationCheckpointingVariants.FULL_ACTIVATION_CHECKPOINTING
      • ActivationCheckpointingVariants.SELECTIVE_LAYER_ACTIVATION_CHECKPOINTING
      • ActivationCheckpointingVariants.SELECTIVE_OP_ACTIVATION_CHECKPOINTING
  • Module contents
• modalities.training.gradient_clipping package
  • Submodules
  • modalities.training.gradient_clipping.fsdp_gradient_clipper module
    • DummyGradientClipper
      • DummyGradientClipper.clip_gradients()
    • FSDP1GradientClipper
      • FSDP1GradientClipper.clip_gradients()
    • FSDP1LoggingOnlyGradientClipper
      • FSDP1LoggingOnlyGradientClipper.clip_gradients()
    • FSDP2GradientClipper
      • FSDP2GradientClipper.clip_grad_norm_()
      • FSDP2GradientClipper.clip_gradients()
    • FSDP2LoggingOnlyGradientClipper
      • FSDP2LoggingOnlyGradientClipper.clip_gradients()
    • GradientClippingMode
      • GradientClippingMode.MAX_NORM
      • GradientClippingMode.P1_NORM
      • GradientClippingMode.P2_NORM
  • modalities.training.gradient_clipping.fsdp_gradient_clipper_config module
    • DummyGradientClipperConfig
      • DummyGradientClipperConfig.model_config
    • FSDPDummyGradientClipperConfig
      • FSDPDummyGradientClipperConfig.model_config
      • FSDPDummyGradientClipperConfig.norm_type
      • FSDPDummyGradientClipperConfig.wrapped_model
    • FSDPGradientClipperConfig
      • FSDPGradientClipperConfig.max_norm
      • FSDPGradientClipperConfig.model_config
      • FSDPGradientClipperConfig.norm_type
      • FSDPGradientClipperConfig.wrapped_model
  • modalities.training.gradient_clipping.gradient_clipper module
    • GradientClipperIF
      • GradientClipperIF.clip_gradients()
  • Module contents

Submodules

modalities.training.training_progress module

class modalities.training.training_progress.TrainingProgress(num_seen_steps_current_run, num_seen_tokens_current_run, num_target_steps, num_target_tokens, num_seen_steps_previous_run=0, num_seen_tokens_previous_run=0)

Bases: object

Dataclass to store the training progress.

Attributes:

num_seen_steps_current_run (int): Number of seen steps in the current run. num_seen_tokens_current_run (int): Number of seen tokens in the current run. num_target_steps (int): Target number of steps. num_target_tokens (int): Target number of tokens. num_seen_steps_previous_run (Optional[int]): Number of seen steps in the previous run. num_seen_tokens_previous_run (Optional[int]): Number of seen tokens in the previous run.

Parameters:

• num_seen_steps_current_run (int)

• num_seen_tokens_current_run (int)

• num_target_steps (int)

• num_target_tokens (int)

• num_seen_steps_previous_run (int | None)

• num_seen_tokens_previous_run (int | None)

num_seen_steps_current_run: int = <dataclasses._MISSING_TYPE object>

num_seen_steps_previous_run: Optional[int] = 0

property num_seen_steps_total: int

num_seen_tokens_current_run: int = <dataclasses._MISSING_TYPE object>

num_seen_tokens_previous_run: Optional[int] = 0

property num_seen_tokens_total: int

num_target_steps: int = <dataclasses._MISSING_TYPE object>

num_target_tokens: int = <dataclasses._MISSING_TYPE object>

Module contents