metaflow.org

Modeling: Use any Python libraries for models and business logic. Metaflow helps manage libraries locally and in the cloud.

Deployment: Deploy workflows to production with a single command and integrate with surrounding systems seamlessly.

Versioning: Metaflow tracks and stores variables inside the flow automatically for easy experiment tracking and debugging.

Orchestration: Create robust workflows in plain Python. Develop and debug them locally, deploy to production without changes.

Compute: Leverage the cloud to execute functions at scale. Use GPUs, multiple cores, and large amounts of memory as needed.

Data: Access data from data warehouses. Metaflow flows data across steps, versioning everything on the way.

Develop with Metaflow: Explore with notebooks, develop with Metaflow, and test and debug locally. Results are stored and tracked automatically for easy analysis.

Scale out to the cloud: Break out from the confines of a laptop or a single notebook. Scale out easily to the cloud, utilizing GPUs, multiple cores, and multiple instances in parallel. Metaflow organizes the work for easy collaboration on the way.

Deploy to production confidently: Deploy experiments to production with a single click without changing anything in the code. Make flows react to updating data and other events automatically.

Bring your own Cloud: Get started easily on a laptop. When you are ready to scale, deploy the Metaflow stack on your cloud account or on-premise Kubernetes cluster. Metaflow integrates seamlessly with your existing infrastructure, security, and data governance policies.

Battle-hardened at Netflix: Metaflow was originally developed at Netflix to address the needs of developers and data scientists who work on demanding real-life ML, AI, and data projects.

Real-Time, Dynamic Cards: Build observable ML/AI systems with cards that update in real-time.

Include any PyPI packages: Install dependencies from PyPI as well as Conda in your Metaflow steps.

Access Secrets Securely: Connect to external services securely using the new @secrets decorator.

Build reactive production systems with Metaflow: Metaflow 2.9 allows you to trigger workflows based on real-time events.

Load and process tabular data fast with Metaflow: Apache Arrow and Metaflow.S3 make it easy to process data fast.

Using Metaflow to train a large language model: Learn how to use Metaflow for demanding GPU tasks.

Metaflow supports Apache Airflow: Develop with Metaflow, deploy on your existing Apache Airflow servers.

Metaflow on Google Cloud: Deploy and operate Metaflow on GCP and all other major clouds.

Metaflow on Azure: Deploy and operate Metaflow on Microsoft Azure.