Include additional files or directories in a flow's code package.
Relative source paths are resolved from the directory containing the flow file, not from the current working directory. By default, each source is placed in the code package under its basename.
Parameters ---------- sources : path-like, (path-like, path-like), or iterable of these A source file or directory, a ``(source, arcname)`` pair, or multiple source specifications. Directories are traversed recursively.
A source may be absolute or relative to the flow file. The optional ``arcname`` in a pair specifies where that source is placed inside the code package.
Use a list to specify exactly two sources without archive names; a two-item tuple is interpreted as ``(source, arcname)``. arcname : path-like, optional Destination for a single source inside the code package. It must be a safe relative path and cannot be absolute, ``.``, or contain ``..``. For multiple sources, specify archive paths with ``(source, arcname)`` pairs instead. suffixes : iterable of str or comma-separated str, optional File suffixes to include. Leading dots are optional and matching is case-insensitive. The default is ``DEFAULT_PACKAGE_SUFFIXES`` (``.py,.R,.RDS`` by default).
Providing this argument replaces the default suffix set; it does not extend it.
Raises ------ MetaflowException If a source does not exist, an archive path is unsafe, or ``arcname`` is used with multiple sources.
Marks a method in a FlowSpec as a Metaflow Step. Note that this decorator needs to be placed as close to the method as possible (ie: before other decorators).
In other words, this is valid: ``` @batch @step def foo(self): pass ```
whereas this is not: ``` @step @batch def foo(self): pass ```
Parameters ---------- f : Union[Callable[[FlowSpecDerived], None], Callable[[FlowSpecDerived, Any], None]] Function to make into a Metaflow Step
Returns ------- Union[Callable[[FlowSpecDerived, StepFlag], None], Callable[[FlowSpecDerived, Any, StepFlag], None]] Function that is a Metaflow Step
Marks a method in a FlowSpec as a Metaflow Step. Note that this decorator needs to be placed as close to the method as possible (ie: before other decorators).
In other words, this is valid: ``` @batch @step def foo(self): pass ```
whereas this is not: ``` @step @batch def foo(self): pass ```
Parameters ---------- f : callable, optional Function to make into a Metaflow Step. When using keyword arguments (e.g. ``@step(start=True)``), this is ``None`` and a decorator function is returned instead. start : bool, default False Mark this step as the start (entry) step of the flow. end : bool, default False Mark this step as the end (terminal) step of the flow. node_info : dict, optional Extra metadata to attach to this step's DAGNode. Extensions can use this to store arbitrary information accessible via ``flow._graph`` (live references) and ``_graph_info`` (serialized via ``to_pod``).
Returns ------- callable The decorated function, or a decorator if keyword arguments were used.
\n",
"\n",
- "\n",
+ "\n",
"\n",
"\n",
"\n",
"\n",
"\n",
- "\t\n",
+ "\t\n",
+ "\t\n",
+ "\t\n",
+ "\t\n",
"\n",
"\n",
- "\t\n",
+ "\t\n",
"\n",
""
],
"text/plain": [
- ""
+ ""
]
},
"execution_count": 2,
@@ -104,7 +107,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
- "version": "3.11.0"
+ "version": "3.11.15"
}
},
"nbformat": 4,
diff --git a/docs/api/step-decorators/step.md b/docs/api/step-decorators/step.md
index 446064c0..2d42a676 100644
--- a/docs/api/step-decorators/step.md
+++ b/docs/api/step-decorators/step.md
@@ -7,16 +7,19 @@ Use `@step` to construct Metaflow workflows. For more information, see [Basics o
-
+
-
+
+
+
+
-
+
diff --git a/docs/metaflow/basics.md b/docs/metaflow/basics.md
index a8247ff8..46b68e0e 100644
--- a/docs/metaflow/basics.md
+++ b/docs/metaflow/basics.md
@@ -22,14 +22,15 @@ We call the graph of operations **a flow**. You define the operations, called **
which are nodes of the graph and contain transitions to the next steps, which serve as
edges.
-Metaflow sets some constraints on the structure of the graph. For starters, every flow
-needs a step called `start` and a step called `end`. An execution of the flow, which we
-call **a run**, starts at `start`. The run is successful if the final `end` step
-finishes successfully.
+Metaflow requires every flow to have an entry step and a terminal step. By default,
+Metaflow looks for steps named `start` and `end`, which is what most flows use. If you
+prefer different names, mark the steps explicitly with `@step(start=True)` and
+`@step(end=True)`. A single step may serve as both the entry and terminal step, as
+described in [Single-step flows](#single-step-flows) below.
-What happens between `start` and `end` is up to you. You can construct the graph in
-between using an arbitrary combination of the following three types of transitions
-supported by Metaflow:
+An execution of the flow, which we call **a run**, starts at the entry step. The run is
+successful if the terminal step finishes successfully. Multi-step flows can construct
+the graph between these steps using the transitions described below.
### Linear
@@ -75,6 +76,61 @@ python linear.py run
Whenever you see a flow like this in the documentation, just save it in a file and
execute it like above.
+### Custom step names
+
+By default, the entry step is named `start` and the terminal step is named `end`. You
+can use different names by marking the steps explicitly:
+
+```python
+from metaflow import FlowSpec, step
+
+
+class CustomStepsFlow(FlowSpec):
+
+ @step(start=True)
+ def ingest(self):
+ self.rows = list(range(10))
+ self.next(self.report)
+
+ @step(end=True)
+ def report(self):
+ print("processed %d rows" % len(self.rows))
+
+
+if __name__ == "__main__":
+ CustomStepsFlow()
+```
+
+Exactly one step in the flow must be the entry step and exactly one must be the
+terminal step.
+
+### Single-step flows
+
+A flow containing one step is valid. Metaflow automatically treats that step as both
+the entry and terminal step, so it does not need separate `start` and `end` methods.
+
+
+
+```python
+from metaflow import FlowSpec, step
+
+
+class SingleStepFlow(FlowSpec):
+
+ @step
+ def run(self):
+ self.result = 42
+ print("hello from a single-step flow")
+
+
+if __name__ == "__main__":
+ SingleStepFlow()
+```
+
+A single-step flow does not call `self.next(...)`. You may annotate the method with
+`@step(start=True, end=True)` if you prefer to make both roles explicit; the two forms
+are equivalent for a single-step flow.
+
### Artifacts
Besides executing the steps `start`, `a`, and `end` in order, this flow creates **a data
diff --git a/docs/metaflow/composing-flows/custom-decorators.md b/docs/metaflow/composing-flows/custom-decorators.md
index 8ba5069f..061e379b 100644
--- a/docs/metaflow/composing-flows/custom-decorators.md
+++ b/docs/metaflow/composing-flows/custom-decorators.md
@@ -114,7 +114,7 @@ even need to `import` the decorators if you add them with `--with`. Try it:
python waiterflow.py run --with myprofiler.my_profile --with kubernetes
```
or equally `--with batch`. Notably, the decorators don't have to exist in the same [directory
-hierarchy as your flow code](/scaling/dependencies/project-structure), nor you have to include them with `@pypi`.
+hierarchy as your flow code](/scaling/project-structure), nor you have to include them with `@pypi`.
If your custom decorator is part of a Python package with multiple modules, Metaflow will automatically package the entire package. This allows you to implement advanced decorators as well-structured Python packages, which can be distributed internally via your internal package repository or published to PyPI. If your decorator requires third-party dependencies, you can include them using a bundled `@pypi` decorator, as shown in
[this example](/metaflow/composing-flows/mutators#applying-multiple-decorators-with-a-step-mutator).
diff --git a/docs/metaflow/composing-flows/introduction.md b/docs/metaflow/composing-flows/introduction.md
index e936811e..1bce973a 100644
--- a/docs/metaflow/composing-flows/introduction.md
+++ b/docs/metaflow/composing-flows/introduction.md
@@ -18,7 +18,7 @@ steps and flows. For example, you might define shared, project-specific patterns
You can handle cases like these by developing a shared library that encapsulates
the logic and importing it in your steps. Metaflow will [package the
-library](/scaling/dependencies/project-structure) automatically for remote execution,
+library](/scaling/project-structure) automatically for remote execution,
ensuring the logic works seamlessly from local development to production deployments.
This section introduces a powerful Metaflow feature: **custom decorators and mutators**.
@@ -39,7 +39,7 @@ common logic in a decorator offers several key advantages:
- **Reusable and portable**: Distribute decorators as installable packages, whether private or public.
Metaflow packages them for remote execution automatically, even if they live outside
- [your project directory structure](/scaling/dependencies/project-structure).
+ [your project directory structure](/scaling/project-structure).
:::note
@@ -89,4 +89,4 @@ project’s best practices, ensuring that all relevant decorators are applied au
without requiring users to remember to add them manually. [Read more about
the `BaseFlow` pattern](/metaflow/composing-flows/baseflow).
-
\ No newline at end of file
+
diff --git a/docs/scaling/dependencies/README.md b/docs/scaling/dependencies/README.md
index 198975d9..5ac133bc 100644
--- a/docs/scaling/dependencies/README.md
+++ b/docs/scaling/dependencies/README.md
@@ -5,7 +5,7 @@
:::tip
If you are in a hurry:
- Want to use your own modules and packages in a flow?
- See [Structuring Projects](/scaling/dependencies/project-structure).
+ See [Structuring Projects](/scaling/project-structure).
- Want to use a Python library in a flow? See [Managing Libraries](/scaling/dependencies/libraries).
- Want to use `uv` with Metaflow? See [using `uv`](/scaling/dependencies/uv)
- Want to use or build a specific Docker image?
@@ -60,9 +60,11 @@ file may contain arbitrary user-defined code, making it easy to get started
just with a single file.
2. As the project grows, it is convenient to [structure the project as multiple
-modules and packages](/scaling/dependencies/project-structure), instead of
+modules and packages](/scaling/project-structure), instead of
including hundreds of lines in a single file. Metaflow packages local Python
-dependencies like this automatically.
+dependencies under the flow file directory automatically. Use
+[`@package_sources`](/api/flow-decorators/package_sources) for source roots
+outside that directory, such as sibling packages shared by multiple flows.
3. Crucially, Metaflow packages Metaflow itself for remote execution so that you
don't have to install it manually when
@@ -100,9 +102,11 @@ package by executing
python myflow.py package list
```
-By the default, the package only includes local Python files, not libraries you
-have installed e.g. with `pip install` manually. To include external libraries,
-you need to include them either in [a custom Docker
+By default, the package only includes local Python files under the flow file
+directory, not libraries you have installed e.g. with `pip install` manually.
+Use [`@package_sources`](/api/flow-decorators/package_sources) for additional
+local source roots. To include external libraries, you need to include them
+either in [a custom Docker
image](/scaling/dependencies/containers), [specify them in a `@pypi` or `@conda`
decorator](/scaling/dependencies/conda-vs-pypi), or [use `uv`](/scaling/dependencies/uv).
@@ -116,7 +120,7 @@ Let's say your team has a common module `special_module.py` that is used by
many flows. You could include it in any of the three layers. You could
1. Include it next to `flow.py` as a local dependency which Metaflow packages
-automatically.
+automatically, or include a sibling source root with `@package_sources`.
2. Publish it as a Python package and include it with `@pypi`.
3. Include it in a custom Docker image.
diff --git a/docs/scaling/dependencies/internals.md b/docs/scaling/dependencies/internals.md
index d452132d..ffcbae3e 100644
--- a/docs/scaling/dependencies/internals.md
+++ b/docs/scaling/dependencies/internals.md
@@ -34,7 +34,7 @@ this diagram:
Let's go over the operation following the blue steps, starting at the top:
1. Local `.py` files and others specified by `--package-suffixes` are [packaged
- in a code package as usual](/scaling/dependencies/project-structure).
+ in a code package as usual](/scaling/project-structure).
2. When using `@pypi` or `@conda`, every step gets its own virtual environment.
@@ -70,4 +70,3 @@ The code package is unpacked to make local dependencies available.
After this, the task is executed in an environment that contains
exactly the specified packages. Nothing more, nothing less.
-
diff --git a/docs/scaling/dependencies/libraries.md b/docs/scaling/dependencies/libraries.md
index 23017cef..02ee1c25 100644
--- a/docs/scaling/dependencies/libraries.md
+++ b/docs/scaling/dependencies/libraries.md
@@ -2,7 +2,7 @@
# Managing Libraries
Whereas the previous page covered [packaging of your own Python modules and
-packages](/scaling/dependencies/project-structure), this page covers handling
+packages](/scaling/project-structure), this page covers handling
of 3rd party dependencies that are published as installable Python packages.
Metaflow supports installation of external packages from two Python package
@@ -27,7 +27,7 @@ use `uv` instead, see [using `uv`](/scaling/dependencies/uv).
The `@pypi` and `@conda` decorators allow you to make arbitrary packages
available to Metaflow steps, as if you were installing them manually with
`pip install` or `conda install`. This functionality works in conjuction
-with [local code packaging](/scaling/dependencies/project-structure), so
+with [local code packaging](/scaling/project-structure), so
steps can execute in safely isolated, remote-execution friendly environments
that contain all dependencies they need.
@@ -321,4 +321,4 @@ Run the flow as usual:
$ python peekabooflow.py --environment=conda run
```
Notice how the path is the same in the `start` and `end` steps but different in the
-`peekaboo` step which uses a system-wide Python installation.
\ No newline at end of file
+`peekaboo` step which uses a system-wide Python installation.
diff --git a/docs/scaling/dependencies/project-structure.md b/docs/scaling/project-structure.md
similarity index 50%
rename from docs/scaling/dependencies/project-structure.md
rename to docs/scaling/project-structure.md
index face1b26..14f0c97d 100644
--- a/docs/scaling/dependencies/project-structure.md
+++ b/docs/scaling/project-structure.md
@@ -133,58 +133,184 @@ As a project grows, it may become desirable to separate each flow in a
subdirectory of its own, so each person or a team can manage their
files independently. All of them may share one or more common packages.
-For instance, we can have two flows, `crumpetflow` and `teatimeflow` as
-independent subdirectories with their own READMEs, as well as a shared
-`crumpet` package:
+For instance, we can have two flows, `crumpetflow` and `teatimeflow`, in
+independent subdirectories with their own READMEs. Both flows import a shared
+package named `crumpet` that lives next to the flow directories:
+
+```text
+my-repo/
+|-- crumpetflow/
+| |-- flow.py
+| `-- README.md
+|-- teatimeflow/
+| |-- flow.py
+| `-- README.md
+`-- crumpet/
+ |-- __init__.py
+ |-- teatime.py
+ `-- raisin.py
+```
+
+Metaflow packages files under the flow file directory by default. In this
+layout, a run of `crumpetflow/flow.py` would include files under
+`crumpetflow/`, but it would not include the sibling `crumpet/` package unless
+you ask for it explicitly.
+
+Add `@package_sources("../crumpet")` to each flow that needs the shared
+package:
+
+```python
+from metaflow import FlowSpec, package_sources, step
+
+
+@package_sources("../crumpet")
+class CrumpetFlow(FlowSpec):
+
+ @step
+ def start(self):
+ from crumpet import raisin, teatime
+ self.tea_time = teatime.is_tea_time()
+ self.is_dry = raisin.is_dry()
+ self.next(self.end)
+
+ @step
+ def end(self):
+ pass
```
-crumpetflow/flow.py
-crumpetflow/README.md
-teatimeflow/flow.py
-teatimeflow/README.md
-crumpet/__init__.py
-crumpet/teatime.py
-crumpet/raisin.py
+
+Use the same decorator in `teatimeflow/flow.py`; from that file,
+`../crumpet` still points at the shared package. Relative source paths are
+resolved from the directory containing the flow file, not from the current
+working directory.
+
+By default, the source is stored in the code package under its basename. In
+this case, `../crumpet` becomes `crumpet/`, so remote tasks can import it as
+`from crumpet import teatime`.
+
+The decorator controls the code package used for remote execution. It does not
+modify Python's local import path. If you launch these flows from the repository
+root, make the root importable locally too:
+
+```bash
+PYTHONPATH=. python crumpetflow/flow.py run
+PYTHONPATH=. python teatimeflow/flow.py run
```
-Unfortunately this wouldn't work out of the box since Metaflow packages
-only the files under the `crumpetflow` and `teatimeflow` directory, ignoring
-`crumpet` by default.
+You can verify that the shared package is included before launching a run:
-The solution is to including a symbolic link (*symlink*) in each flow directory, pointing
-at the common package that should be included. You can create a symlink as follows
+```bash
+PYTHONPATH=. python crumpetflow/flow.py package list
```
-cd crumpetflow
-ln -s ../crumpet .
+
+### Projects using a `src` layout
+
+A common project layout keeps flow definitions and reusable Python packages in
+separate directories:
+
+```text
+my-project/
+|-- flows/
+| |-- train.py
+| `-- score.py
+|-- src/
+| |-- forecasting/
+| | |-- __init__.py
+| | |-- features.py
+| | `-- model.py
+| `-- common/
+| `-- __init__.py
+`-- tests/
```
-With symlinks included the hierarchy looks like this:
+To preserve a top-level `import forecasting` during remote execution, select
+the importable package directory itself:
+
+```python
+@package_sources("../src/forecasting")
+class TrainFlow(FlowSpec):
+ ...
```
-crumpetflow/flow.py
-crumpetflow/README.md
-crumpetflow/crumpet -> ../crumpet
-teatimeflow/flow.py
-teatimeflow/README.md
-teatimeflow/crumpet -> ../crumpet
-crumpet/__init__.py
-crumpet/teatime.py
-crumpet/raisin.py
+
+This places `forecasting/` at the root of the code package. Selecting
+`../src` instead would place the package under `src/forecasting/`, which does
+not provide a top-level `forecasting` import.
+
+For multiple packages under `src/`, pass each package explicitly:
+
+```python
+@package_sources([
+ "../src/forecasting",
+ "../src/common",
+])
+class TrainFlow(FlowSpec):
+ ...
+```
+
+When launching from the project root, make `src/` importable locally:
+
+```bash
+PYTHONPATH=src python flows/train.py run
+```
+
+### Other `@package_sources` forms
+
+Package more than one shared root by passing a list:
+
+```python
+@package_sources(["../crumpet", "../shared_models"])
+class CrumpetFlow(FlowSpec):
+ ...
+```
+
+Use `arcname` when a single source should appear under a different path inside
+the code package:
+
+```python
+@package_sources("../crumpet", arcname="vendor/crumpet")
+class CrumpetFlow(FlowSpec):
+ ...
```
+The archive path affects imports: the example above packages the module as
+`vendor/crumpet`, rather than as a top-level `crumpet` package.
+
+For multiple sources with explicit archive paths, use `(source, arcname)`
+pairs:
+
+```python
+@package_sources([
+ ("../crumpet", "vendor/crumpet"),
+ ("../shared_models", "models"),
+])
+class CrumpetFlow(FlowSpec):
+ ...
+```
+
+Single files can be packaged too:
+
+```python
+@package_sources("../config/settings.py", arcname="config/settings.py")
+class CrumpetFlow(FlowSpec):
+ ...
+```
+
+Archive paths must be relative paths inside the code package. For non-Python
+files under an extra source root, set `suffixes` as shown in the
+[Non-Python dependencies](#non-python-dependencies) section below.
+
+For the full decorator contract, see
+[`@package_sources`](/api/flow-decorators/package_sources).
+
### Using common packages from other Git repositories
-The above hierarchy works well when everything is stored in a single Git
-repository. Technically, you could have `crumpetflow`, `teatimeflow`, and
-`crumpet` as separate repositories as well, but you would need to ensure
-that symlinks stay valid between repositories. This can be fragile.
+`@package_sources` packages files that are already available locally; it does
+not clone or download another repository.
If you want to include a package from a separate repository, a better approach
-is to use [the `git subtree`
-command](https://www.atlassian.com/git/tutorials/git-subtree), which is an enhanced
-version of Git submodules. With `git subtree` you can nest a repository as
-a subdirectory of another repository. For instance, the `crumpet` package
-could be a repository of its own, included as a subtree in every flow project
-that wants to use it.
+is to vendor it into the project with a mechanism such as [Git
+subtree](https://www.atlassian.com/git/tutorials/git-subtree) or a Git
+submodule, then point `@package_sources` at the resulting local directory.
Alternatively, you can publish the package as a private Python package which
you can [include with `@pypi` or `@conda`](/scaling/dependencies/libraries).
@@ -192,9 +318,9 @@ you can [include with `@pypi` or `@conda`](/scaling/dependencies/libraries).
## Non-Python dependencies
-By default, Metaflow packages `.py` files in the flow's directory hierarchy.
-You can also include arbitrary files in the package for remote execution by
-including their file suffices in the `--package-suffixes` option.
+By default, Metaflow packages Python files in the flow's directory hierarchy.
+You can include other file types from the same hierarchy with the
+`--package-suffixes` option.
For instance, the example below shows how to include SQL files but you could
also include custom binaries or configuration files.
@@ -248,14 +374,38 @@ Execute the code as follows:
```
python moviesqlflow.py --package-suffixes .sql run
```
-Locally, it would work without `--package-suffixes` but when running remotely
-`--with batch` or `--with kubernetes`, it would complain about missing `.sql` files
-unless `--package-suffixes` is specified.
+Locally, it would work without `--package-suffixes` because the files are
+already on disk. When running remotely with `--with batch` or
+`--with kubernetes`, the task only sees the code package, so it would complain
+about missing `.sql` files unless `--package-suffixes` is specified. Metaflow
+still includes the default Python and R suffixes when you add extra suffixes
+this way.
You can confirm that all dependencies are included properly by executing
```
python moviesqlflow.py --package-suffixes .sql package list
```
+If the non-Python files live outside the flow directory, package that source
+root with `@package_sources` and set `suffixes` on the decorator:
+
+```python
+from metaflow import FlowSpec, package_sources
+
+
+@package_sources("../shared_assets", suffixes=[".py", ".sql", ".json"])
+class MovieSQLFlow(FlowSpec):
+ ...
+```
+
+When `suffixes` is provided to `@package_sources`, that list replaces the
+default suffix list for the extra source root. Include `.py` explicitly if the
+source root also contains Python modules. You can inspect the result with:
+
+```bash
+python moviesqlflow.py package list
+```
+For the full decorator contract, see
+[`@package_sources`](/api/flow-decorators/package_sources).
diff --git a/docusaurus.config.js b/docusaurus.config.js
index 8e727023..f3fc8d4d 100644
--- a/docusaurus.config.js
+++ b/docusaurus.config.js
@@ -5,7 +5,7 @@ const lightCodeTheme = require("prism-react-renderer/themes/github");
const darkCodeTheme = require("prism-react-renderer/themes/dracula");
// produced by redirect-config.py based on redirect-urls.csv
-const REDIRECTS = [{"to": "/production/coordinating-larger-metaflow-projects", "from": "/going-to-production-with-metaflow/coordinating-larger-metaflow-projects"}, {"to": "/production/introduction", "from": "/going-to-production-with-metaflow/scheduling-metaflow-flows"}, {"to": "/production/scheduling-metaflow-flows/scheduling-with-argo-workflows", "from": "/going-to-production-with-metaflow/scheduling-metaflow-flows/scheduling-with-argo-workflows"}, {"to": "/production/scheduling-metaflow-flows/scheduling-with-aws-step-functions", "from": "/going-to-production-with-metaflow/scheduling-metaflow-flows/scheduling-with-aws-step-functions"}, {"to": "/internals/technical-overview", "from": "/internals-of-metaflow/technical-overview"}, {"to": "/internals/testing-philosophy", "from": "/internals-of-metaflow/testing-philosophy"}, {"to": "/internals/contributing", "from": "/introduction/contributing-to-metaflow"}, {"to": "/introduction/metaflow-resources", "from": "/introduction/getting-in-touch"}, {"to": "/internals/release-notes", "from": "/introduction/release-notes"}, {"to": "/introduction/metaflow-resources", "from": "/introduction/roadmap"}, {"to": "/getting-started/infrastructure", "from": "/metaflow-on-aws"}, {"to": "/getting-started/infrastructure", "from": "/metaflow-on-aws/deploy-to-aws"}, {"to": "/getting-started/infrastructure", "from": "/metaflow-on-aws/metaflow-sandbox"}, {"to": "/scaling/data", "from": "/metaflow/data"}, {"to": "/scaling/dependencies", "from": "/metaflow/dependencies"}, {"to": "/scaling/failures", "from": "/metaflow/failures"}, {"to": "/scaling/introduction", "from": "/metaflow/scaling-out-and-up"}, {"to": "/scaling/introduction", "from": "/metaflow/scaling-out-and-up/effortless-scaling-with-aws-batch"}, {"to": "/scaling/introduction", "from": "/metaflow/scaling-out-and-up/effortless-scaling-with-kubernetes"}, {"to": "/scaling/tagging", "from": "/metaflow/tagging"}]
+const REDIRECTS = [{"to": "/production/coordinating-larger-metaflow-projects", "from": "/going-to-production-with-metaflow/coordinating-larger-metaflow-projects"}, {"to": "/production/introduction", "from": "/going-to-production-with-metaflow/scheduling-metaflow-flows"}, {"to": "/production/scheduling-metaflow-flows/scheduling-with-argo-workflows", "from": "/going-to-production-with-metaflow/scheduling-metaflow-flows/scheduling-with-argo-workflows"}, {"to": "/production/scheduling-metaflow-flows/scheduling-with-aws-step-functions", "from": "/going-to-production-with-metaflow/scheduling-metaflow-flows/scheduling-with-aws-step-functions"}, {"to": "/internals/technical-overview", "from": "/internals-of-metaflow/technical-overview"}, {"to": "/internals/testing-philosophy", "from": "/internals-of-metaflow/testing-philosophy"}, {"to": "/internals/contributing", "from": "/introduction/contributing-to-metaflow"}, {"to": "/introduction/metaflow-resources", "from": "/introduction/getting-in-touch"}, {"to": "/internals/release-notes", "from": "/introduction/release-notes"}, {"to": "/introduction/metaflow-resources", "from": "/introduction/roadmap"}, {"to": "/getting-started/infrastructure", "from": "/metaflow-on-aws"}, {"to": "/getting-started/infrastructure", "from": "/metaflow-on-aws/deploy-to-aws"}, {"to": "/getting-started/infrastructure", "from": "/metaflow-on-aws/metaflow-sandbox"}, {"to": "/scaling/data", "from": "/metaflow/data"}, {"to": "/scaling/dependencies", "from": "/metaflow/dependencies"}, {"to": "/scaling/failures", "from": "/metaflow/failures"}, {"to": "/scaling/introduction", "from": "/metaflow/scaling-out-and-up"}, {"to": "/scaling/introduction", "from": "/metaflow/scaling-out-and-up/effortless-scaling-with-aws-batch"}, {"to": "/scaling/introduction", "from": "/metaflow/scaling-out-and-up/effortless-scaling-with-kubernetes"}, {"to": "/scaling/tagging", "from": "/metaflow/tagging"}, {"to": "/scaling/project-structure", "from": "/scaling/dependencies/project-structure"}, {"to": "/production/scheduling-metaflow-flows/scheduling-with-argo-workflows", "from": "/production/scheduling-metaflow-flows/introduction/scheduling-with-argo-workflows"}, {"to": "/scaling/remote-tasks/kubernetes", "from": "/scaling/introduction/effortless-scaling-with-kubernetes"}];
/** @type {import('@docusaurus/types').Config} */
const config = {
diff --git a/redirect-urls.csv b/redirect-urls.csv
index fc7a5f6c..6f736bb9 100644
--- a/redirect-urls.csv
+++ b/redirect-urls.csv
@@ -18,6 +18,6 @@
/scaling/introduction /metaflow/scaling-out-and-up/effortless-scaling-with-aws-batch
/scaling/introduction /metaflow/scaling-out-and-up/effortless-scaling-with-kubernetes
/scaling/tagging /metaflow/tagging
+/scaling/project-structure /scaling/dependencies/project-structure
/production/scheduling-metaflow-flows/scheduling-with-argo-workflows /production/scheduling-metaflow-flows/introduction/scheduling-with-argo-workflows
/scaling/remote-tasks/kubernetes /scaling/introduction/effortless-scaling-with-kubernetes
-https://metaflow.org/sandbox/ /v/r/metaflow-on-aws/metaflow-sandbox
diff --git a/sidebars.js b/sidebars.js
index 4473eb5a..9d340ed4 100644
--- a/sidebars.js
+++ b/sidebars.js
@@ -160,6 +160,7 @@ const sidebars = {
id: "scaling/introduction",
},
items: [
+ "scaling/project-structure",
{
type: "category",
label: "Computing at Scale",
@@ -187,7 +188,6 @@ const sidebars = {
id: "scaling/dependencies/README",
},
items: [
- "scaling/dependencies/project-structure",
"scaling/dependencies/libraries",
"scaling/dependencies/uv",
"scaling/dependencies/conda-vs-pypi",
@@ -303,6 +303,7 @@ const sidebars = {
},
items: [
"api/flow-decorators/conda_base",
+ "api/flow-decorators/package_sources",
"api/flow-decorators/project",
"api/flow-decorators/schedule",
"api/flow-decorators/trigger",
diff --git a/static/assets/dag-single-step.svg b/static/assets/dag-single-step.svg
new file mode 100644
index 00000000..f05b92a7
--- /dev/null
+++ b/static/assets/dag-single-step.svg
@@ -0,0 +1,13 @@
+