| Title | Fabric Metadata-Driven (FMD) Framework Overview |
|---|---|
| Description | Discover the architecture, components, and deployment guidance for the Fabric Metadata-Driven (FMD) Framework on Fabric SQL Database. |
| Date | 07/2025 |
| Author | edkreuk |
The Fabric Metadata-Driven (FMD) Framework is a scalable, extensible solution for managing, integrating, and governing data using a metadata-driven approach on Fabric SQL Database. This article provides an overview of the FMD Framework, including its architecture, core components, workspace structure, supported data sources, and deployment guidance.
The FMD Framework enables organizations to streamline data operations by leveraging metadata to drive dynamic data pipelines and parameterized notebooks. Built on Fabric SQL Database, the framework supports secure, flexible, and modern data management at scale.
Tip
The FMD Framework is designed for rapid deployment and extensibility. You can use it out-of-the-box or customize it to meet your organization's evolving data needs.
- Comprehensive data governance: Maintain detailed metadata for improved data quality, consistency, and compliance.
- Scalability and flexibility: Seamlessly scale with organizational growth and adapt to changing data requirements.
- Streamlined data integration: Integrate diverse data sources for a unified data landscape.
- Cost efficiency: Optimize data processes and reduce redundancy to achieve cost savings.
The FMD Framework uses a modular architecture that separates data, code, and orchestration for enhanced security and manageability.
The recommended workspace structure ensures clear separation of responsibilities and access control.
| Workspace type | Purpose | Examples |
|---|---|---|
| Data workspaces | Manage and store data | Data Landing Zone, Bronze, Silver |
| Code workspaces | Develop and manage code artifacts | Data Pipelines, Notebooks, Spark |
| Gold workspaces | Manage and store data | Gold and Semantic Model |
| Reporting workspaces | Create reports for end users | Reports |
| Orchestration & logging | Orchestrate and monitor data operations | Fabric SQL Database, Auditing and logging |
Note
For organizations with a Gold Layer, create a separate workspace for reports to restrict access appropriately.
The framework implements the Medallion Architecture using Lakehouses:
- Data Landing Zone: Stores raw, unstructured, and structured data with incremental loads and no enforced schema.
- Bronze layer: Deduplicates and structures data, applying schema for consistency.
- Silver layer: Maintains historical, validated data with enforced quality rules.
- Gold layer: Maintains Dimensions and Facts
The Variable Library centralizes the management of variables used throughout the FMD Framework. It is integrated with the PL_LOAD_ALL process, allowing Variables(parameters) to be passed dynamically during execution.
Parameters are added for the ConnectionString and Database name for the SQL_FMD_FRAMEWORK. These parameters are necassary for the build in logging. Parameters are updated according your environment during the execution on the Setup Notebook.
Currently supported variables:
key_vault_nameLakehouse_schema_enabled
Additional variables will be supported in future releases.
To deploy the default taskflow:
- Import
FMD_FABRIC_TASKFLOW.jsonfrom the Taskflow folder into your workspace. - Assign the correct artifacts to the taskflows as shown in the documentation.
The FMD Framework supports integration with the following sources:
- SQL Server (on-premises or cloud)
- Azure Data Lake Gen2
- SFTP / FTP
- Azure Data Factory
- Onelake tables and files
To get started:
- Review the FMD Framework Deployment Guide.
- Import the taskflow and configure your workspaces as recommended.
- Refer to additional resources for data model, pipelines, and logging.
- FMD Data Model reference
- FMD Framework Data Pipelines
- Configure and load demo data
- Auditing and logging
- Data cleansing
-
Fabric SQL Database limitations:
If deployment fails, verify that you haven't exceeded the allowed number of Fabric Databases in your tenant. Trial capacities are limited to three databases. -
Error handling:
If you encounter errors during deployment (for example, database creation failures), re-run the notebook or manually create the database to diagnose the issue.
We welcome contributions! To suggest improvements, open an issue or submit a pull request.
If opening a pull request, please follow these steps:
- Fork the repository.
- Create a feature branch.
- Commit your changes.
- Push to the feature branch.
- Create a pull request and add documentation on what you have changed.
This project is licensed under the GNU GENERAL PUBLIC LICENSE.
Contributors:
Erwin de Kreuk
Marco Hansma