Understanding the data consumption capability¶

Data consumption can refer to almost any access to or sharing of, data after data processing in the data lifecycle. This area is considered the value delivery area of the data lifecycle, where consumers get access to the data. Generally, Data Consumption capabilities vary based on the type of data consumers and the type of data systems. For example, an analytical system with internal data consumers has different requirements compared to a transactional system with external data consumers. See Understanding data consumption characteristics below for working data consumption code.

Understanding data consumption characteristics¶

This section describes some of the key characteristics/functions seen in the data consumption area.

Implementing data virtualization¶

Data virtualization is one approach to managing data. It provides a unified representation of data. It also allows for data retrieval without requiring technical details about the data's format or physical storage location. Other benefits are unified governance of enterprise data sources, abstraction from the complexity of data storage, and simplified maintenance without data duplication. Common use cases for data virtualization include decentralized data management and data sharing/data marketplace scenarios. Data virtualization enables users to access data in place, allowing for real-time analytics on operational data stores. Azure provides several options for data virtualization, including:

• Azure Synapse Analytics with Azure Synapse Link and External Tables.
• Azure SQL Database with OPENROWSET and EXTERNAL TABLE statements.
• Azure Data Explorer with External Tables.

Defining data marketplace¶

The data marketplace concept allows your organization to securely share your data with internal and external parties. It combines a well-defined process and technical capabilities to define what, how, when, and with whom data is shared. Organizations that implement the right strategy for a data marketplace benefit from new revenue streams, increased efficiencies, scalability, and improved data processing performance. Common scenarios in data marketplace include data consolidation, data distribution, data brokerage, and data escrow. Azure provides several technology options for data marketplace, such as Azure Data Share, Microsoft Purview in-place data sharing, and Microsoft Purview. These services enable your organization to securely share and manage your data.

Defining data privacy and security¶

Data privacy and security are critical aspects of data management your organization must prioritize to protect sensitive information and comply with regulations. Data privacy refers to protection of personal and sensitive data, ensuring that its collected, stored, and processed in a secure and responsible manner. Data security, on the other hand, focuses on safeguarding data from unauthorized access, breaches, and malicious activities. Implementing robust data privacy and security measures helps your organization build trust with your customers, partners, and stakeholders, and avoid legal and reputational risks. Azure provides various tools and services to support data privacy and security. These tools include Azure Purview for data discovery and classification, Azure Key Vault for secure key management, and Azure Sentinel for threat detection and response.

Defining data versioning and sharing¶

Data versioning and sharing is a crucial aspect of data management that allows your organization to effectively track and manage changes to your data over time. It involves creating and maintaining different versions of datasets, enabling users to access and analyze historical data while ensuring data integrity and traceability. Data versioning provides benefits such as improved collaboration, reproducibility of analyses, and the ability to roll back to previous versions if needed. Your Organization can implement data versioning and sharing through various technologies and platforms, such as Azure Data Share, Azure Purview, and Azure Synapse Analytics. These solutions offer features like continuous export from operational data sources, centralized security and governance, and the ability to explore and query data in place.

Learn more about data consumption in Microsoft Fabric¶

In Microsoft Fabric, once data is ingested and processed it can be made available for user consumption in multiple ways by:

• Storing data in Lakehouse and Sharing the Lakehouse.
• Using Shortcuts to reference data without copying it.
• Using Mirroring to access and ingest data continuously and seamlessly from any database or data warehouse into the Data Warehousing experience in Microsoft Fabric.

See the decision guide for a comparison of data stores, which can be sources of data consumption supported by Microsoft Fabric.

Examples¶

• Single tech samples, data share automation - Part of the Modern Data Warehouse repository

For more information¶

• Data marketplace
• In-place data sharing
• Data engineering in Microsoft Fabric