Comparing Azure Databricks and Azure Synapse Analytics¶
Azure Synapse Analytics is an integrated analytics service provided by Microsoft. Azure Synapse Analytics combines enterprise data warehousing, big data processing, and data integration into a single platform. It has deep integration with other Azure services such as Microsoft Power BI, Azure Cosmos DB, and Azure ML.
Azure Databricks is a fast, scalable, and collaborative analytics platform provided by Microsoft in collaboration with Databricks. Azure Databricks is built on Apache Spark, an open-source analytics engine. It provides a fully managed and optimized environment designed for processing and analyzing large volumes of big data.
Review considerations¶
This section lists some considerations that are helpful to evaluate Azure Databricks and Azure Synapse Analytics for data processing. Note that this is not a detailed comparison of each feature.
Using Structured Streaming¶
For near-real time data processing, Databricks: Structured Streaming is a great choice. With tight integration with Delta Lake and "Auto Loader" functionality, it offers end-to-end fault tolerance with exactly-once processing guarantees.
As a data warehouse, near-real time data can be ingested into Azure Synapse Analytics using Azure Stream analytics but it currently doesn't support Delta format. As a developer platform, Synapse doesn’t fully focus on real-time transformations yet.
Using enterprise data warehouse capabilities¶
Many times, there is a requirement of having a serving layer in your data lake that has traditional data warehousing capabilities. For such cases, Azure Synapse Analytics is a great choice.
Azure Synapse Analytics brings together enterprise data warehousing and Big Data analytics. Dedicated SQL pool refers to the enterprise data warehousing features that are available in Azure Synapse Analytics.
Dedicated SQL pool represents a collection of analytic resources that are provisioned when using Synapse SQL. The data is stored in relational tables with columnar storage. This format significantly reduces the data storage costs, and improves query performance. Once data is stored, you can run analytics at massive scale.
In Azure Databricks, a Delta Lake based data warehouse is possible, but it won't have the full width of SQL and data warehousing capabilities.
Understand the serverless SQL capabilities¶
Generally, a running cluster is required to query data using Spark that results in extra cost and underutilized resources. Having serverless capabilities to run ad-hoc queries and reports is a much-desired feature.
Azure Synapse Analytics has this built-in capability in the form of Serverless SQL pool. Serverless SQL pool is a distributed data processing system, built for large-scale data and computational functions. Here are some of the characteristics of Serverless SQL pool:
- It's serverless, hence there's no infrastructure to setup or clusters to maintain.
- A default endpoint for this service is provided within every Azure Synapse workspace.
- The external Spark tables can be queried directly from serverless SQL pool.
- Based on pay-per-use model, there is no charge for resources reserved. Users are only charged for the data processed by queries run.
On Azure Databricks, Serverless SQL provides instant compute to users for their BI and SQL workloads, with minimal management requirements. Similar to Synapse, users only pay for Serverless SQL when they start running reports or queries.
Admins can create serverless SQL warehouses (formerly SQL endpoints) that enable instant compute and are managed by Azure Databricks. Serverless SQL warehouses use compute clusters in the Azure subscription of Azure Databricks.
Understand the support for Git Providers¶
Companies often use specific Git providers across the organization. In such cases, it becomes important to check that the data service supports integration with each such git repository.
Azure Databricks supports the following Git providers:
- GitHub and GitHub AE
- Bitbucket Cloud
- GitLab
- Azure DevOps Git
Azure Databricks also offers support for enterprise git platforms like GitHub Enterprise Server, Bitbucket Server, GitLab self-managed integration. However, it is important to note that for integration to work, the git server needs to be accessible over the internet.
Azure Synapse Analytics only supports the following Git providers:
- Azure DevOps Git
- GitHub and GitHub Enterprise
Check for integration with Microsoft Dataverse¶
If you are using Power Platforms, seamless and managed integration between Azure Synapse with Dataverse might be an important consideration. With Azure Synapse Link, Microsoft Dataverse data can be connected to Azure Synapse Analytics to get near real-time insights about the data.
This kind of integration of Microsoft Dataverse is not currently available for Azure Databricks. It can be achieved by using third-party connectors or writing custom integration code based on Dataverse Web APIs.
Learn about supported languages¶
Here is a quick summary of supported languages:
| Language | Azure Synapse | Azure Databricks |
|---|---|---|
| PySpark (Python) | Yes | Yes |
| Spark (Scala) | Yes | Yes |
| Spark SQL | Yes | Yes |
| .NET Spark (C#) | Yes | No** |
| SparkR (R) | Yes* | Yes |
*Support for R is currently under public preview as of October 18, 2022.
**Though not supported out-of-box, the .NET for Apache Spark jobs can still be run on Databricks clusters. Check Microsoft Documentation for the details.
Learn about specific features¶
There are many features that are unique to each product. For instance, Azure Databricks has Z-Ordering a technique to collocate related information in the same set of files. Z-Ordering can dramatically reduce the amount of data that Delta Lake on Databricks needs to read, and thus improve overall query performance.
Auto Loader is another unique capability of Azure Databricks that allows incremental processing of new data files as they arrive in cloud storage efficiently.
For workloads that process a significant amount of data (100+ GB) and include aggregations and joins, Azure Databricks has a native vectorized query engine called Photon. It is directly compatible with Apache Spark APIs; Photon is developed in C++ to take advantage of modern hardware. It uses the latest techniques available for vectorized query processing to capitalize on data, and instruction-level parallelism in CPUs. It does have certain limitations though such as it doesn't support UDFs or RDD APIs. Refer to the documentation link above for details.
Check for data governance support¶
Microsoft Purview is a data governance service provided by Microsoft. It helps organizations discover, understand, and manage their data assets across various sources and locations.
Purview has connectors to authenticate and interact with both Azure Synapse and Azure Databricks. For Azure Databricks, the Azure Databricks to Purview Lineage Connector can transfer lineage metadata from Spark operations to Microsoft Purview.
In addition, a Microsoft Purview account can be registered to an Azure Synapse workspace. It allows you to discover Microsoft Purview assets, interact with them through Synapse capabilities, and push lineage information to Microsoft Purview.
For Azure Databricks, Unity Catalog provides an alternate data governance option for data and AI assets in the Lakehouse. Unity Catalog offers a single place to administer data access policies that apply across all workspaces and personas. It has built-in auditing and lineage capabilities and supports data discovery.
In summary, if Microsoft Purview is used as the data governance platform, Azure Synapse generally has seamless integration and more flexibility. But if you are planning to use Unity Catalog, Azure Databricks is a natural (and only) choice.
Check for network isolation support¶
Both Azure Databricks and Azure Synapse Analytics provides capabilities of secure network deployment but the way it gets deployed is different. While Azure Synapse can be deployed using Managed workspace Virtual Networks to provide network isolation, Azure Databricks uses VNet injections with Secure cluster connectivity to achieve the same.
Having said that, the "Managed VNet" deployment of Azure Synapse Analytics makes it much easier. Also, the public access to Synapse studio can be easily blocked using the Azure Synapse Analytics IP firewall rules. On Azure Databricks, the same can be achieved by using IP Access List API that enables Databricks admins to configure IP allow lists and block lists for a workspace.
Check Azure Databricks VNet recipe and Azure Synapse VNet recipe for more details and code samples.
Check for data pipeline orchestration support¶
Most big data solutions consist of repeated data processing operations, encapsulated in workflows. A pipeline orchestrator is a tool that helps to automate these workflows. An orchestrator can schedule jobs, execute workflows, and coordinate dependencies among tasks.
Azure Data Factory (ADF) is a managed service that can be used as a pipeline orchestrator. Within Azure Synapse, these pipelines are in-built (based on ADF) and are called Synapse pipelines. These pipelines can be triggered manually or configured based on a scheduled, tumbling window, storage event, or custom event. It can trigger both Databricks and Synapse jobs/notebooks.
Databricks also has a new capability called Delta Live Tables (DLT). DLT is a framework for building reliable, maintainable, and testable data processing pipelines. Users define transformations to perform on data, and DLT manages task orchestration, cluster management, monitoring, data quality, and error handling. You can also enforce data quality with DLT expectations. DLT on Azure Databricks requires "Premium Plan".
For more information, see Choosing Pipeline Orchestrator and Data: Data Orchestration.
Choosing between Azure Databricks and Azure Synapse Analytics¶
Both Azure Databricks and Azure Synapse Analytics are first-party services on Azure. These services can be easily created using the Azure portal (and Azure CLI, PowerShell, and SDK) and have full enterprise support. In addition, the technological differences between two platforms are either trivial or can be addressed by using alternate options and/or workarounds.
There are various factors that can affect a customer's decision in choosing the data processing tool of choice. Some of the factors are:
- Customer is already using a specific service for other projects.
- Customer has a preference for open-source products.
- Customer wants portability across cloud platforms.
- There are organizational guidelines for using (or not using) particular technology.
- The team is proficient in certain service/programming language.