Welcome to dplutils documentation!

dplutils is a Python library for remote execution of compute intensive streaming task graphs.

Check out the Quick Start to get going immediately, Usage section for further information, and the API documentation for details the the classes and methods available.

Note

This project is under active development.

Contents

• Quick Start
  • Installation
  • Define Pipeline
  • Execute Pipeline
  • Learn More
• Usage
  • Tasks and Pipeline Graphs
    • Task units
    • Graph construction
    • Graph operations
    • Multiple edges
  • Configuring Pipelines
    • Task Defaults
    • Runtime Configuration
    • Context
  • Executing Pipelines
    • Example
    • Inputs
    • Outputs
  • Observing
    • Example Usage
    • Setting an Observer
    • Executor Compatibility
    • Custom observers
  • Executing via CLI
  • Extending Functionality
• API
  • Pipeline
    • Backbone
      • dplutils.pipeline.PipelineExecutor
        • PipelineExecutor
          • PipelineExecutor.__init__()
      • dplutils.pipeline.PipelineTask
        • PipelineTask
          • PipelineTask.__init__()
      • dplutils.pipeline.PipelineGraph
        • PipelineGraph
          • PipelineGraph.__init__()
    • Executors
      • dplutils.pipeline.stream.StreamingGraphExecutor
        • StreamingGraphExecutor
          • StreamingGraphExecutor.__init__()
    • Ray Execution
      • dplutils.pipeline.ray.RayDataPipelineExecutor
        • RayDataPipelineExecutor
          • RayDataPipelineExecutor.__init__()
      • dplutils.pipeline.ray.RayStreamGraphExecutor
        • RayStreamGraphExecutor
          • RayStreamGraphExecutor.__init__()
  • Observers
    • Observer interface
      • dplutils.observer.set_observer
        • set_observer()
      • dplutils.observer.get_observer
        • get_observer()
      • dplutils.observer.observe
        • observe()
      • dplutils.observer.increment
        • increment()
      • dplutils.observer.param
        • param()
      • dplutils.observer.timer
        • timer()
      • dplutils.observer.Observer
        • Observer
          • Observer.__init__()
      • dplutils.observer.Observer.observe
        • Observer.observe()
    • Ray implementations
      • dplutils.observer.ray.RayActorWrappedObserver
        • RayActorWrappedObserver
          • RayActorWrappedObserver.__init__()
      • dplutils.observer.ray.RayMetricsObserver
        • RayMetricsObserver
          • RayMetricsObserver.__init__()
    • Aim implementations
      • dplutils.observer.aim.AimObserver
        • AimObserver
          • AimObserver.__init__()
    • Mlflow implementations
      • dplutils.observer.mlflow.MlflowObserver
        • MlflowObserver
          • MlflowObserver.__init__()
  • CLI
    • CLI Execution Utilities
      • dplutils.cli.cli_run
        • cli_run()
      • dplutils.cli.get_argparser
        • get_argparser()
      • dplutils.cli.set_config_from_args
        • set_config_from_args()
      • dplutils.cli.add_generic_args
        • add_generic_args()