Microsoft Fabric Data Engineering artefacts in short

-

 

⭐ Microsoft Fabric – Ingestion Options

These are the mechanisms that actually ingest or load data into Fabric (Lakehouse, Warehouse, KQL DB, etc.).

1. Data Pipelines – Copy Activity

The primary batch ingestion mechanism in Fabric. Moves data from supported connectors into Fabric.
[interloopdata.com]

2. Dataflows Gen2 (Power Query–based ingestion)

No‑code ingestion + transformation tool for analysts.
[interloopdata.com]

3. Notebooks (PySpark / Python / Spark SQL)

Code-based ingestion, supports APIs, files, cloud storage, databases.
[Document | Word]

4. Eventstream (real-time ingestion)

Low-latency, streaming ingestion from event sources.
[interloopdata.com]

5. Copy Jobs (standalone quick ingestion)

Quick setup ingestion without a full pipeline.
[withum.com]

6. Database Mirroring

Federated live sync from SQL sources into Fabric (Fabric native feature).
[interloopdata.com]

⭐ Microsoft Fabric – Orchestration Options

These are tools that schedule, coordinate, automate, and trigger ingestion jobs.

1. Fabric Data Pipelines (Data Factory Experience)

The successor to Azure Data Factory inside Fabric.
It orchestrates workflows, schedules activities, triggers notebooks, dataflows, SQL, etc.
[keyrus.com]

2. Pipeline Triggers (Schedule + File/Reflex events)

Fabric includes native scheduling and file-event triggers.
Equivalent to ADF triggers but integrated into the Fabric Activator framework.
[keyrus.com]

3. Notebooks Scheduled via Pipelines

Notebooks can be automated and triggered through Data Pipelines.
[interloopdata.com]

4. Eventstream Routing

Though primarily ingestion, Eventstream can route data to different Fabric destinations, acting as a real-time orchestrator.
[interloopdata.com]

5. External orchestration (non-Fabric but commonly used)

Even though inside Fabric these are “outside”, enterprises often use them:

  • Azure Logic Apps
  • Azure Functions
  • GitHub Actions
  • Azure DevOps pipelines
    These can call Fabric APIs or pipelines, but they’re not built into Fabric.


⭐ MS Fabric DE artefacts | (Ingestion Tools & Orchestration Tools)

Ingestion Tools

Orchestration Tools

Comments

Post a Comment

Popular posts from this blog

Refresh Settings & Failure Management for Large Semantic Models in Microsoft Fabric (2026 Edition)

-
Image
  Large semantic models—those exceeding 1 GB—require special handling in Microsoft Fabric and Power BI Premium due to their refresh cost, memory requirements, and impact on shared capacity. Refreshing these models safely is critical to operational stability, data reliability, and user experience. This blog post provides a complete guide to refresh settings, refresh orchestration, and failure recovery for large models, supported by diagrams, best‑practice configurations, Fabric‑aligned techniques, and citations to Microsoft Learn and Fabric resources. 🔷 1. Why Large Models Need Special Refresh Handling Large semantic models: Consume significantly more memory during refresh (often 2.5× the model size ). Cause capacity evictions or query delays if unmanaged. Require incremental refresh and partition-aware refresh to remain stable. Require scale-out replicas for uninterrupted user experience during refresh. Microsoft confirms that Import mode semantic models are the only ones ...
Read more

Migrating from Power BI Premium to Microsoft Fabric: A Step-by-Step Guide

-
Image
Why This Matters? Microsoft has announced the retirement of Power BI Premium per capacity SKUs (P-SKUs) in favor of the Microsoft Fabric F-SKU capacity model as part of its unified analytics platform strategy. Fabric includes all Power BI capabilities plus additional workloads like data engineering, data warehousing, Synapse-style analytics, governance, and AI. If you currently run on Power BI Premium capacity and want to continue without disruption, you need to transition your content to Microsoft Fabric capacity before your next renewal or end of subscription. 📅 Key Licensing and Transition Details Before we dive into steps, here’s the high-level context you need: Purchase of Fabric capacity (F-SKUs) is done via Azure; P-SKUs are no longer purchasable after February 1, 2025. You can continue using your existing P-SKU until your current agreement expires . After the P-SKU ends, Microsoft gives 30 days of free Fabric capacity equal to your previous SKU size to tran...
Read more

Incremental Refresh in Dataflows Gen2 (2026 Edition)

-
Image
  Incremental Refresh (IR) in Power BI Dataflows Gen2 is a powerful, often misunderstood capability—especially now that Microsoft Fabric positions OneLake as the unified storage layer for ingestion, transformation, and analytics. This post explains how Dataflow Incremental Refresh works, how it differs from dataset IR, when to use it, and how to design it safely for enterprise‑grade workloads. 🔷 1. What Dataflow Incremental Refresh Is (and Isn’t) Dataflow IR performs incremental ingestion at the ETL layer. It controls what data is extracted from source systems and loaded into OneLake , not how semantic models partition their imported data. Key characteristics: ✔ Dataflow IR manages data ingestion , not semantic model partitions Dataflows apply Store/Refresh windows during data extraction and materialization. Unlike dataset IR, they do not create semantic model partitions. 1 ✔ Query folding is mandatory If folding breaks, IR is ignored and the entire dataflow refreshes fully. M...
Read more