What's new in Azure Synapse Analytics?
This page is continuously updated with a recent review of what's new in Azure Synapse Analytics, and also what features are currently in preview. To follow the latest in Azure Synapse news and features, see the Azure Synapse Analytics Blog and companion videos on YouTube.
For older updates, review past Azure Synapse Analytics Blog posts or previous updates in Azure Synapse Analytics.
Important
Microsoft Fabric has been announced!
- Learn about this exciting new preview and discover What is Microsoft Fabric?
- Get started with end-to-end tutorials in Microsoft Fabric.
- See What's new in Microsoft Fabric?
Features currently in preview
The following table lists the features of Azure Synapse Analytics that are currently in preview. Preview features are sorted alphabetically.
Note
Features currently in preview are available under supplemental terms of use, review for legal terms that apply to Azure features that are in beta, preview, or otherwise not yet released into general availability. Azure Synapse Analytics provides previews to give you a chance to evaluate and share feedback with the product group on features before they become generally available (GA).
| Feature | Learn more |
|---|---|
| Apache Spark Delta Lake tables in serverless SQL pools | The ability to for serverless SQL pools to access Delta Lake tables created in Spark databases is in preview. For more information, see Azure Synapse Analytics shared metadata tables. |
| Apache Spark elastic pool storage | Azure Synapse Analytics Spark pools now support elastic pool storage in preview. Elastic pool storage allows the Spark engine to monitor worker node temporary storage and attach more disks if needed. No action is required, and you should see fewer job failures as a result. For more information, see Azure Synapse Analytics Spark elastic pool storage. |
| Apache Spark R language support | Built-in R support for Apache Spark is now in preview. |
| Azure Synapse Data Explorer | The Azure Synapse Data Explorer provides an interactive query experience to unlock insights from log and telemetry data. Connectors for Azure Data Explorer are available for Synapse Data Explorer. For more news, see Azure Synapse Data Explorer (preview). |
| Browse ADLS Gen2 folders in the Azure Synapse Analytics workspace | You can now browse an Azure Data Lake Storage Gen2 (ADLS Gen2) container or folder in your Azure Synapse Analytics workspace in Synapse Studio. To learn more, see Browse an ADLS Gen2 folder with ACLs in Azure Synapse Analytics. |
| Capture changed data from Cosmos DB analytical store | Azure Cosmos DB analytical store now supports change data capture (CDC) for Azure Cosmos DB API for NoSQL and Azure Cosmos DB API for MongoDB. For more information, see Capture Changed Data from your Cosmos DB analytical store and DevBlog: Change Data Capture (CDC) with Azure Cosmos DB analytical store. |
| Distribution Advisor | The Distribution Advisor is a new preview feature in Azure Synapse dedicated SQL pools Gen2 that analyzes queries and recommends the best distribution strategies for tables to improve query performance. For more information, see Distribution Advisor in Azure Synapse SQL. |
| Distributed Deep Neural Network Training | Learn more about new distributed training libraries like Horovod, Petastorm, TensorFlow, and PyTorch in Deep learning tutorials. |
| Embed ADX dashboards | Azure Data Explorer dashboards be embedded in an IFrame and hosted in third party apps. |
| Reject options for delimited text files | Reject options for CREATE EXTERNAL TABLE on delimited files is in preview. |
| Spark Advisor for Azure Synapse Notebook | The Spark Advisor for Azure Synapse Notebook analyzes code run by Spark and displays real-time advice for Notebooks. The Spark advisor offers recommendations for code optimization based on built-in common patterns, performs error analysis, and locates the root cause of failures. |
| Time-To-Live in managed virtual network (VNet) | Reserve compute for the time-to-live (TTL) in managed virtual network TTL period, saving time and improving efficiency. For more information on this preview, see Announcing public preview of Time-To-Live (TTL) in managed virtual network. |
| User-Assigned managed identities | Now you can use user-assigned managed identities in linked services for authentication in Synapse Pipelines and Dataflows. To learn more, see Credentials in Azure Data Factory and Azure Synapse. |
Generally available features
The following table lists the features of Azure Synapse Analytics that have transitioned from preview to general availability (GA) within the last 12 months.
| Month | Feature | Learn more |
|---|---|---|
| April 2023 | Apache Spark Optimized Write | Optimize Write is a Delta Lake on Azure Synapse feature reduces the number of files written by Apache Spark 3 (3.1 and 3.2) and aims to increase individual file size of the written data. |
| March 2023 | Cosmos DB Synapse Link for Azure Data Explorer GA | Azure Data Explorer supports fully managed data ingestion from Azure Cosmos DB using a change feed. We now support Cosmos DB accounts behind a Managed Private Endpoint or Service Endpoint. For more information, see Ingest data from Azure Cosmos DB into Azure Data Explorer. |
| March 2023 | Multi-column distribution in dedicated SQL pools | You can now Hash Distribute tables on multiple columns for a more even distribution of the base table, reducing data skew over time and improving query performance. For more information on this generally available feature, see the three options: CREATE MATERIALIZED VIEW, CREATE TABLE distribution options, or CREATE TABLE AS SELECT distribution options. |
| March 2023 | Deploying Synapse SQL serverless using SSDT | SqlPackage's long-awaited support for Azure Synapse Analytics serverless SQL pools is now available starting with the 161.8089.0 SqlPackage. Serverless SQL pools are supported for both the extract and publish actions. |
| February 2023 | ADX Dashboards GA | Now generally available, Azure Data Explorer dashboards using the Azure Data Explorer web UI allow you to explore your data from end-to-end, starting with data ingestion, running queries, and ultimately building dashboards. |
| February 2023 | UTF-8 and Japanese collations support for dedicated SQL pools | Both UTF-8 support and Japanese collations are now generally available for dedicated SQL pools. |
| February 2023 | Azure Synapse Runtime for Apache Spark 3.3 | The Azure Synapse Runtime for Apache Spark 3.3 is now generally available. Based on our testing using the 1TB TPC-H industry benchmark, you're likely to see up to 77% increased performance. |
| December 2022 | SSIS IR Express virtual network injection | Both the standard and express methods to inject your SSIS Integration Runtime (IR) into a VNet are generally available now. For more information, see General Availability of Express Virtual Network injection for SSIS in Azure Data Factory. |
| November 2022 | Ingest data from Azure Stream Analytics into Synapse Data Explorer | The ability to use a Streaming Analytics job to collect data from an event hub and send it to your Azure Data Explorer cluster is now generally available. For more information, see Ingest data from Azure Stream Analytics into Azure Data Explorer and ADX output from Azure Stream Analytics. |
| November 2022 | Azure Synapse Link for SQL | Azure Synapse Link for SQL is now generally available for both SQL Server 2022 and Azure SQL Database. The Azure Synapse Link for SQL feature provides low- and no-code, near real-time data replication from your SQL-based operational stores into Azure Synapse Analytics. Provide BI reporting on operational data in near real-time, with minimal impact on your operational store. To learn more, visit What is Azure Synapse Link for SQL? |
| October 2022 | SAP CDC connector GA | The data connector for SAP Change Data Capture (CDC) is now GA. For more information, see Announcing Public Preview of the SAP CDC solution in Azure Data Factory and Azure Synapse Analytics and SAP CDC solution in Azure Data Factory. |
| September 2022 | MERGE T-SQL syntax | MERGE T-SQL syntax has been a highly requested addition to the Synapse T-SQL library. As in SQL Server, the MERGE syntax encapsulates INSERTs/UPDATEs/DELETEs into a single high-performance statement. Available in dedicated SQL pools in version 10.0.17829 and above. For more, see the MERGE T-SQL announcement blog. |
| July 2022 | Apache Spark™ 3.2 for Synapse Analytics | Apache Spark™ 3.2 for Synapse Analytics is now generally available. Review the official release notes and migration guidelines between Spark 3.1 and 3.2 to assess potential changes to your applications. For more details, read Apache Spark version support and Azure Synapse Runtime for Apache Spark 3.2. Highlights of what got better in Spark 3.2 in the Azure Synapse Analytics July Update 2022. |
| July 2022 | Apache Spark in Azure Synapse Intelligent Cache feature | Intelligent Cache for Spark automatically stores each read within the allocated cache storage space, detecting underlying file changes and refreshing the files to provide the most recent data. To learn more, see how to Enable/Disable the cache for your Apache Spark pool. |
| June 2022 | Map Data tool | The Map Data tool is a guided process to help you create ETL mappings and mapping data flows from your source data to Synapse without writing code. To learn more about the Map Data tool, read Map Data in Azure Synapse Analytics. |
| June 2022 | User Defined Functions | User defined functions (UDFs) are now generally available. To learn more, read User defined functions in mapping data flows. |
Community
This section summarizes new Azure Synapse Analytics community opportunities and the Azure Synapse Influencer program from Microsoft.
| Month | Feature | Learn more |
|---|---|---|
| April 2023 | Azure Synapse MVP Corner | March highlights from the Microsoft Azure Synapse MVP blog series Azure Synapse MVP Corner. |
| March 2023 | Azure Synapse MVP Corner | February highlights from the Microsoft Azure Synapse MVP blog series Azure Synapse MVP Corner. |
| February 2023 | Azure Synapse MVP Corner | January highlights from the Microsoft Azure Synapse MVP blog series Azure Synapse MVP Corner. |
| January 2023 | Azure Synapse MVP Corner | December highlights from the Microsoft Azure Synapse MVP blog series Azure Synapse MVP Corner. |
| December 2022 | Azure Synapse MVP Corner | November highlights from the Microsoft Azure Synapse MVP blog series in this month's Azure Synapse MVP Corner. |
| November 2022 | Azure Synapse Influencer program | The Azure Synapse Influencer program provides exclusive events and Q&A sessions like Ask the Experts with the Microsoft product team, where members can interact directly with product experts by asking any questions on various rotating topics. Get feedback from members of Azure Synapse Analytics influencer community. |
| October 2022 | Azure Synapse MVP Corner | October highlights from the Microsoft Azure Synapse MVP blog series in this month's Azure Synapse MVP Corner. |
| September 2022 | Azure Synapse MVP Corner | September highlights from the Microsoft Azure Synapse MVP blog series in this month's Azure Synapse MVP Corner. |
| May 2022 | Azure Synapse Influencer program | Sign up for our free Azure Synapse Influencer program and get connected with a community of Synapse-users who are dedicated to helping others achieve more with cloud analytics. Register now for our next Synapse Influencer Ask the Experts session. It's free to attend and everyone is welcome to participate and join the discussion on Synapse-related topics. You can watch past recorded Ask the Experts events on the Azure Synapse YouTube channel. |
Apache Spark for Azure Synapse Analytics
This section summarizes recent new features and capabilities of Apache Spark for Azure Synapse Analytics.
| Month | Feature | Learn more |
|---|---|---|
| April 2023 | Delta Lake - Low Shuffle Merge | Low Shuffle Merge optimization for Delta tables is now available in Apache Spark 3.2 and 3.3 pools. You can now update a Delta table with advanced conditions using the Delta Lake MERGE command. |
| March 2023 | Library management new ability: in-line installation | %pip and %conda are now available in Apache Spark for Synapse! %pip and %conda are commands that can be used on Notebooks to install Python packages. For more information, see Manage session-scoped Python packages through %pip and %conda commands. |
| March 2023 | Increasing Azure Synapse Analytics Spark performance up to 77% | More regions are receiving the performance increase for Azure Synapse Spark workloads, including most recently Korea Central, Central India, and Australia Southeast. |
| March 2023 | Azure Synapse Spark Notebook – Unit Testing | Learn how to test and create unit test cases for Spark jobs developed using Synapse Notebook. |
| March 2023 | Apache Spark 2.4 and 3.1 retirement cycle | The Azure Synapse runtime for Apache Spark 2.4 and 3.1 have entered the retirement cycle. Apache Spark 2.4 will be retired September 29, 2023, and Apache Spark 3.1 will be retired as of January 26, 2024. You should relocate your workloads to a newer Apache Spark runtime within this period. Read more at Apache Spark runtimes in Azure Synapse and view the Spark migration guide. |
| February 2023 | Azure Synapse Runtime for Apache Spark 3.3 | The Azure Synapse Runtime for Apache Spark 3.3 is now generally available. Based on our testing using the 1TB TPC-H industry benchmark, you're likely to see up to 77% increased performance. |
| January 2023 | Spark Advisor for Azure Synapse Notebook | The Spark Advisor for Azure Synapse Notebook analyzes code run by Spark and displays real-time advice for Notebooks. The Spark advisor offers recommendations for code optimization based on built-in common patterns, performs error analysis, and locates the root cause of failures. |
| January 2023 | Improve Spark pool utilization with Synapse Genie | The Synapse Genie Framework improves Spark pool utilization by executing multiple Synapse notebooks on the same Spark pool instance. Read more about this metadata-driven utility written in Python. |
| November 2022 | Azure Synapse Runtime for Apache Spark 3.3 | The Azure Synapse Runtime for Apache Spark 3.3 is currently in preview. For more information, see the Apache Spark 3.3 preview blog post. Based on our testing using the 1TB TPC-H industry benchmark, you're likely to see up to 77% increased performance. |
| September 2022 | New informative Livy error codes | More precise error codes describe the cause of failure and replaces the previous generic error codes. Previously, all errors in failing Spark jobs surfaced with a generic error code displaying LIVY_JOB_STATE_DEAD. |
| September 2022 | New query optimization techniques in Apache Spark for Azure Synapse Analytics | Read the findings from Microsoft's work to gain considerable performance benefits across the board on the reference TPC-DS workload as well as a significant reduction in query plan generation time. |
| August 2022 | Apache Spark elastic pool storage | Azure Synapse Analytics Spark pools now support elastic pool storage in preview. Elastic pool storage allows the Spark engine to monitor worker nodes temporary storage and attach additional disks if needed. No action is required, and you should see fewer job failures as a result. For more information, see Blog: Azure Synapse Analytics Spark elastic pool storage is available for public preview. |
| August 2022 | Apache Spark Optimized Write | Optimize Write is a Delta Lake on Synapse preview feature that reduces the number of files written by Apache Spark 3 (3.1 and 3.2) and aims to increase individual file size of the written data. To learn more, see The need for optimize write on Apache Spark. |
Data integration
This section summarizes recent new features and capabilities of Azure Synapse Analytics data integration. Learn how to Load data into Azure Synapse Analytics using Azure Data Factory (ADF) or a Synapse pipeline.
| Month | Feature | Learn more |
|---|---|---|
| April 2023 | Capture changed data from Cosmos DB analytical store (Public Preview) | Azure Cosmos DB analytical store now supports change data capture (CDC) for Azure Cosmos DB API for NoSQL and Azure Cosmos DB API for MongoDB. For more information, see Capture Changed Data from your Cosmos DB analytical store and DevBlog: Change Data Capture (CDC) with Azure Cosmos DB analytical store. |
| March 2023 | Deep dive: Synapse pipelines storage event trigger security | This Customer Success Engineering blog post is a deep dive into Azure Synapse pipelines storage event trigger security. ADF and Synapse Pipelines offer a feature that allows pipeline execution to be triggered based on various events, such as storage blob creation or deletion. This can be used by customers to implement event-driven pipeline orchestration. |
| January 2023 | SQL CDC incremental extract now supports numeric columns | Enabling incremental extract from SQL Server CDC in dataflows allows you to only process rows that have changed since the last time that pipeline was executed. Supported incremental column types now include date/time and numeric columns. |
| December 2022 | Express virtual network injection | Both the standard and express methods to inject your SSIS Integration Runtime (IR) into a VNet are generally available now. For more information, see General Availability of Express Virtual Network injection for SSIS in Azure Data Factory. |
| October 2022 | SAP CDC connector GA | The data connector for SAP Change Data Capture (CDC) is now GA. For more information, see Announcing Public Preview of the SAP CDC solution in Azure Data Factory and Azure Synapse Analytics and SAP CDC solution in Azure Data Factory. |
| September 2022 | Gantt chart view | You can now view your activity runs with a Gantt chart in Azure Data Factory Integration Runtime monitoring. |
| September 2022 | Monitoring improvements | We've released a new bundle of improvements to the monitoring experience based on community feedback. |
| September 2022 | Maximum column optimization in mapping dataflow | For delimited text data sources such as CSVs, a new maximum columns setting allows you to set the maximum number of columns. |
| September 2022 | NUMBER to integer conversion in Oracle data source connector | New property to convert Oracle NUMBER type to a corresponding integer type in source via the new property convertDecimalToInteger. For more information, see the Oracle source connector. |
| September 2022 | Support for sending a body with HTTP request DELETE method in Web activity | New support for sending a body (optional) when using the DELETE method in Web activity. For more information, see the available Type properties for the Web activity. |
| August 2022 | Mapping data flows now support visual Cast transformation | You can use the cast transformation to easily modify the data types of individual columns in a data flow. |
| August 2022 | Default activity timeout changed to 12 hours | The default activity timeout is now 12 hours. |
| August 2022 | Pipeline expression builder ease-of-use enhancements | We've updated our expression builder UI to make pipeline designing easier. |
| August 2022 | New UI for mapping dataflow inline dataset types | We've updated our data flow source UI to make it easier to find your inline dataset type. |
| July 2022 | Time-To-Live in managed virtual network (VNet) | Reserve compute for the time-to-live (TTL) in managed virtual network TTL period, saving time and improving efficiency. For more information on this preview, see Announcing public preview of Time-To-Live (TTL) in managed virtual network. |
| June 2022 | SAP CDC connector preview | A new data connector for SAP Change Data Capture (CDC) is now available in preview. For more information, see Announcing Public Preview of the SAP CDC solution in Azure Data Factory and Azure Synapse Analytics and SAP CDC solution in Azure Data Factory. |
| June 2022 | Fuzzy join option in Join Transformation | Use fuzzy matching with a similarity threshold score slider has been added to the Join transformation in Mapping Data Flows. |
| June 2022 | Map Data tool GA | We're excited to announce that the Map Data tool is now Generally Available. The Map Data tool is a guided process to help you create ETL mappings and mapping data flows from your source data to Synapse without writing code. |
| June 2022 | Rerun pipeline with new parameters | You can now change pipeline parameters when rerunning a pipeline from the Monitoring page without having to return to the pipeline editor. To learn more, read Rerun pipelines and activities. |
| June 2022 | User Defined Functions GA | User defined functions (UDFs) in mapping data flows are now generally available (GA). |
Database Templates & Database Designer
This section summarizes recent new features and capabilities of database templates and the database designer.
| Month | Feature | Learn more |
|---|---|---|
| July 2022 | Browse industry templates | Browse industry templates and add tables to create your own lake database. Learn more about ways you can browse industry templates and get started with Quickstart: Create a new lake database leveraging database templates. |
Developer experience
This section summarizes recent new quality of life and feature improvements for developers in Azure Synapse Analytics.
| Month | Feature | Learn more |
|---|---|---|
| May 2023 | Using Azure DevOps with Synapse Workspaces to create hot fixes in production environments | Blog post on how to deploy a fix from your development Synapse Workspace into a production Synapse Workspace without adversely affecting ongoing development projects. |
| December 2022 | MSSparkUtils is the Swiss Army knife inside Synapse Spark | MSSparkUtils is a built-in package to help you easily perform common tasks called Microsoft Spark utilities, including the ability to share results between notebooks. |
| September 2022 | Synapse CICD for publishing workspace artifacts | Integrating Synapse Studio with a Source Control System such as Azure DevOps Git or GitHub has been shown as one of Synapse Studio's preferred features to collaborate and provide source control for Azure Synapse. The Visual Studio marketplace has a Synapse workspace deployment task to automate publishing. |
| July 2022 | Synapse Notebooks compatibility with IPython | The official kernel for Jupyter notebooks is IPython and it's now supported in Synapse Notebooks. For more information, see Synapse Notebooks is now fully compatible with IPython. |
| July 2022 | Mssparkutils now has spark.stop() method | A new API mssparkutils.session.stop() has been added to the mssparkutils package. This feature becomes handy when there are multiple sessions running against the same Spark pool. The new API is available for Scala and Python. To learn more, see Stop an interactive session. |
Machine Learning
This section summarizes recent new features and improvements to machine learning models in Azure Synapse Analytics.
| Month | Feature | Learn more |
|---|---|---|
| March 2023 | Using OpenAI GPT in Synapse Analytics | Microsoft offers Azure OpenAI as an Azure Cognitive Service, and you can access Azure OpenAI's GPT models from within Synapse Spark. |
| November 2022 | R Support (preview) | Azure Synapse Analytics now provides built-in R support for Apache Spark, currently in preview. For an example, install an R library from CRAN and CRAN snapshots. |
| August 2022 | SynapseML v.0.10.0 | New release of SynapseML v0.10.0 (previously MMLSpark), an open-source library that aims to simplify the creation of massively scalable machine learning pipelines. Learn more about the latest additions to SynapseML and get started with SynapseML. |
| August 2022 | .NET support | SynapseML v0.10 adds full support for .NET languages like C# and F#. For a .NET SynapseML example, see .NET Example with LightGBMClassifier. |
| August 2022 | Azure OpenAI Service support | SynapseML now allows users to tap into 175-Billion parameter language models (GPT-3) from OpenAI that can generate and complete text and code near human parity. For more information, see Azure OpenAI for Big Data. |
| August 2022 | MLflow platform support | SynapseML models now integrate with MLflow with full support for saving, loading, deployment, and autologging. |
| August 2022 | SynapseML in Binder | We know that Spark can be intimidating for first users but fear not because with the technology Binder, you can explore and experiment with SynapseML in Binder with zero setup, install, infrastructure, or Azure account required. |
| June 2022 | Distributed Deep Neural Network Training (preview) | The Azure Synapse runtime also includes supporting libraries like Petastorm and Horovod, which are commonly used for distributed training. This feature is currently available in preview. The Azure Synapse Analytics runtime for Apache Spark 3.1 and 3.2 also now includes support for the most common deep learning libraries like TensorFlow and PyTorch. To learn more about how to leverage these libraries within your Azure Synapse Analytics GPU-accelerated pools, read the Deep learning tutorials. |
Samples and guidance
This section summarizes new guidance and sample project resources for Azure Synapse Analytics.
| Month | Feature | Learn more |
|---|---|---|
| May 2023 | Implementing Slow Change Dimension with Synapse | Demonstrate how to use a serverless SQL Pool to implement Slow Change Dimension type 2 on top of a data lake. |
| May 2023 | CI & CD With Azure Synapse Dedicated SQL Pool | Use version control, Continuous Integration & Deployment, and best practices to manage ALM lifecycle of an Azure Synapse Data Warehouse with this blog article. |
| March 2023 | Create a Data Solution on Azure Synapse Analytics with Snapshot Serengeti | This is a four-part series on building an end-to-end data analytics and machine learning solution on Azure Synapse Analytics. The dataset used in this solution is the Snapshot Serengeti dataset, which consists of a large-scale collection of camera trap images. |
| March 2023 | Introduction to Kusto Query Language (KQL) | This Customer Success Engineering blog post provides an introduction to Kusto Query Language (KQL), a powerful query language to analyze large volumes of structured, semi structured and unstructured (Free Text) data. |
| March 2023 | Creating a custom disaster recovery plan for your Synapse workspace | A multi-part blog series on creating a disaster recovery plan for their Synapse Workspace. |
| March 2023 | Azure Synapse connectivity: public endpoints, private endpoints, managed VNet and managed private endpoints | A three-part expert-written blog series on Azure Synapse connectivity for the various networking options, including inbound dedicated pool public endpoint connectivity, Azure Synapse private endpoints, and managed VNet and managed private endpoints. |
| February 2023 | Historical monitoring dashboards for Azure Synapse dedicated SQL pools | A walkthrough of the steps to enable historical monitoring using Azure Monitor Workbook templates on top of Azure Metrics and Azure Log Analytics. |
| January 2023 | Read Data Lake with Synapse Serverless pools | A two-part guide on how to use OPENROWSET to query a path within the lake or use an external table to query a path within the lake. |
| January 2023 | Structured streaming in Synapse Spark | A detailed example of streaming IoT temperature data from IoT devices into Synapse Spark. |
| January 2023 | Create DNS alias for dedicated SQL pool in Synapse workspace for disaster recovery | A custom DNS for dedicated SQL pools (formerly SQL DW) can provide redirect to client programs during a disaster. |
| December 2022 | Azure Synapse - Data Lake vs. Delta Lake vs. Data Lakehouse | Read a new Success Engineering blog post demystifying the terms Data Lake, Delta Lake, and Data Lakehouse. |
| November 2022 | How Data Exfiltration Protection (DEP) impacts Azure Synapse Analytics Pipelines | Data Exfiltration Protection (DEP) is a feature that enables additional restrictions on the ability of Azure Synapse Analytics to connect to other services. |
| November 2022 | Getting started with REST APIs for Azure Synapse Analytics - Apache Spark Pool | We provide instructions on how to setup and use Synapse REST endpoints and describe the Apache Spark Pool operations supported by REST APIs. |
| November 2022 | Demystifying Azure Synapse Data Explorer | A two-part explainer demystify Data Explorer in Azure Synapse and data ingestion with Azure Synapse Data Explorer. |
| November 2022 | Synapse Spark Delta Time Travel | Delta Lake time travel enables point-in-time query snapshots or even rolls back erroneous updates. |
| September 2022 | What is the difference between Synapse dedicated SQL pool (formerly SQL DW) and Serverless SQL pool? | Understand dedicated vs serverless pools and their concurrency. Read more at basic concepts of dedicated SQL pools and serverless SQL pools. |
| September 2022 | Reading Delta Lake in dedicated SQL Pool | Sample script to import Delta Lake files directly into the dedicated SQL Pool and support features like time-travel. For an explanation, see Reading Delta Lake in dedicated SQL Pool. |
| September 2022 | Azure Synapse Customer Success Engineering blog series | The new Azure Synapse Customer Success Engineering blog series launches with a detailed introduction to Building the Lakehouse - Implementing a Data Lake Strategy with Azure Synapse. |
| June 2022 | Azure Orbital analytics with Synapse Analytics | We now offer an Azure Orbital analytics sample solution showing an end-to-end implementation of extracting, loading, transforming, and analyzing spaceborne data by using geospatial libraries and AI models with Azure Synapse Analytics. The sample solution also demonstrates how to integrate geospatial-specific Azure AI services models, AI models from partners, and bring-your-own-data models. |
| June 2022 | Migration guides for Oracle | A new Microsoft-authored migration guide for Oracle to Azure Synapse Analytics is now available. Design and performance for Oracle migrations. |
| June 2022 | Azure Synapse success by design | The Azure Synapse proof of concept playbook provides a guide to scope, design, execute, and evaluate a proof of concept for SQL or Spark workloads. |
| June 2022 | Migration guides for Teradata | A new Microsoft-authored migration guide for Teradata to Azure Synapse Analytics is now available. Design and performance for Teradata migrations. |
| June 2022 | Migration guides for IBM Netezza | A new Microsoft-authored migration guide for IBM Netezza to Azure Synapse Analytics is now available. Design and performance for IBM Netezza migrations. |
Security
This section summarizes recent new security features and settings in Azure Synapse Analytics.
| Month | Feature | Learn more |
|---|---|---|
| December 2022 | How Data Exfiltration Protection (DEP) impacts Azure Synapse Analytics Pipelines | Data Exfiltration Protection (DEP) is a feature that enables additional restrictions on the ability of Azure Synapse Analytics to connect to other services. |
| August 2022 | Execute Azure Synapse Spark Notebooks with system-assigned managed identity | You can now execute Spark Notebooks with the system-assigned managed identity (or workspace managed identity) by enabling Run as managed identity from the Configure session menu. With this feature, you are able to validate that your notebook works as expected when using the system-assigned managed identity, before using the notebook in a pipeline. For more information, see Managed identity for Azure Synapse. |
| July 2022 | Changes to permissions needed for publishing to Git | Now, only Git permissions and the Synapse Artifact Publisher (Synapse RBAC) role are needed to commit changes in Git-mode. For more information, see Access control enforcement in Synapse Studio. |
Azure Synapse Data Explorer (preview)
Azure Data Explorer (ADX) is a fast and highly scalable data exploration service for log and telemetry data. It offers ingestion from Event Hubs, IoT Hubs, blobs written to blob containers, and Azure Stream Analytics jobs. This section summarizes recent new features and capabilities of the Azure Synapse Data Explorer and the Kusto Query Language (KQL). Read more about What is the difference between Azure Synapse Data Explorer and Azure Data Explorer?
| Month | Feature | Learn more |
|---|---|---|
| April 2023 | ARM template to deploy Azure Data Explorer DB with Cosmos DB connection | An ARM template is now available to quickly deploy an Azure Data Explorer cluster with System Assigned Identity, a database, an Azure Cosmos DB account (NoSql), an Azure Cosmos DB database, an Azure Cosmos DB container, and a data connection between the Cosmos DB container and the Kusto database (using the system assigned identity). |
| April 2023 | Ingest data from Azure Events Hub to ADX free tier | Azure Data Explorer now supports integration with Events Hub in ADX free tier. For more information, see Free Event Hub data analysis with Azure Data Explorer. |
| March 2023 | View cluster history in Kusto Data Explorer | It is now easier to track the history of queries and commands run on a Kusto cluster using .show queries and .show commands-and-queries. |
| March 2023 | Amazon S3 support in Kusto Web Explorer | You can now ingest data from Amazon S3 seamlessly via the Ingestion Hub in Kusto Web Explorer (KWE). |
| March 2023 | Plotly visuals support | Use the Plotly graphing library to create visualizations for a KQL query using 'render' operator or interactively when building ADX dashboards. |
| February 2023 | ADX Dashboards GA | Now generally available, Azure Data Explorer dashboards using the Azure Data Explorer web UI allow you to explore your data from end-to-end, starting with data ingestion, running queries, and ultimately building dashboards. |
| February 2023 | ADX file ingestion supports up to 1000 files | The ADX ingestion wizard now supports up to 1000 files (previously 10) at once. |
| January 2023 | Apache Log4j 2 connector for Azure Data Explorer | The Apache Log4J 2 sink for Azure Data Explorer was developed to easily stream your Log4j 2 log data to Azure Data Explorer, where you can analyze, visualize, and alert on your logs in real-time. For more information, see Getting started with Apache Log4j and Azure Data Explorer. |
| January 2023 | Ingest preexisting Event Hub events to ADX | ADX can now ingest Event Hubs data that existed before the creation of an Event Hubs data connection in your ADX cluster via the Event retrieval start date. |
| January 2023 | Multivariate Anomaly Detection | ADX contains native support for detecting anomalies over multiple time series by using the function series_decompose_anomalies(). For more information, see Multivariate Anomaly Detection. |
| January 2023 | Improved conditional formatting in dashboard | Conditional formatting helps in surfacing anomalies or outlier data points visually. Now you can either format a visual by using conditions or by applying themes to numeric columns or discrete values to non-numeric ones. |
| January 2023 | New display options for pie chart displays | Focus on the data you care about with new display options for pie chart visualizations in Dashboards. |
| December 2022 | ADX Kusto Web Explorer (KWE) JPath viewer | JPath notation describes the path to one or more elements in a JSON document. Use the new expanded view to quickly get a specific element of a JSON text, and easily copy its path expression. For an example, see JPath viewer. |
| December 2022 | Demystifying data consumption using Azure Synapse Data Explorer | A guide to the various ways of retrieving, consuming and visualizing data from Azure Synapse Data Explorer. |
| November 2022 | Table Level Sharing support via Azure Data Share | We have now added Table level sharing support via the Azure Data Share interface where you can share specific tables in the database. This allows you to easily and securely share your data with people in your company or external partners. |
| November 2022 | Ingest data from Azure Stream Analytics into Synapse Data Explorer | The ability to use a Streaming Analytics job to collect data from an event hub and send it to your Azure Data Explorer cluster is now generally available. For more information, see Ingest data from Azure Stream Analytics into Azure Data Explorer and ADX output from Azure Stream Analytics. |
| November 2022 | Parse-kv operator | The new parse-kv operator extracts structured information from a string expression and represents the information in a key/value form. You can use a specified delimeter, a non-specified delimeter, or Regex via a RE2 regular expression. |
| October 2022 | Leaders and followers in ADX clusters | Use the database page in the Azure portal to easily identify all the follower databases following a leader, and the leader for a given follower. |
| October 2022 | Aliasing follower databases | The follower database feature allows you to attach a database located in a different cluster to your Azure Data Explorer cluster. Now you can override the database name while establishing a follower relationship. |
| October 2022 | Ingest data from OpenTelemetry | OpenTelemetry (OTel) is a vendor-neutral open-source application observability framework. The OpenTelemetry exporter supports ingestion of data from many receivers into Azure Data Explorer. |
| October 2022 | Ingest data from Telegraf | Telegraf is an open source, lightweight, minimal memory footprint agent for collecting, processing, and writing telemetry data including logs, metrics, and IoT data. The Azure Data Explorer output plugin serves as the connector from Telegraf and supports ingestion of data from many types of input plugins into Azure Data Explorer. |
| September 2022 | Azure Data Explorer Kusto emulator | The ADX Emulator is a Docker Image exposing an ADX Query Engine endpoint. You can use it to create databases and ingest and query data. The emulator understands Kusto Query Language (KQL) the same way the Azure Service does. |
| September 2022 | Logstash connector proxy configuration | The Azure Data Explorer (ADX) Logstash plugin enables you to process events from Logstash into an ADX database for analysis. Version 1.0.5 now supports HTTP/HTTPS proxies. |
| September 2022 | Kafka support for Protobuf format | The ADX Kafka sink connector leverages the Kafka Connect framework and provides an adapter to ingest data from Kafka in JSON, Avro, String, and now the Protobuf format in the latest update. Read more about Ingesting Protobuf data from Kafka to Azure Data Explorer. |
| September 2022 | Funnel visuals | Funnel is the latest visual we added to Azure Data Explorer dashboards following the feedback we received from customers. |
| September 2022 | .NET and Node.js support in Sample App Generator | The Azure Data Explorer (ADX) sample app generator wizard is a tool that allows you to create a working app to ingest and query your data in your preferred programming language. Now, generating sample apps in .NET and Node.js is supported along with the previously available options Java and Python. |
| August 2022 | Protobuf support in Kafka sink | Azure Data Explorer Kafka sink - a gold certified Confluent connector - helps ingest data from Kafka to Azure Data Explorer. We have added Protobuf support in the connector to help customers bring Protobuf data into ADX. |
| August 2022 | Native support for Amazon S3 | The .ingest into ADX command ingests data into a table by "pulling" the data from one or more cloud storage files. The command now supports Amazon S3 URLs. For an example, read the blog post announcing Continuous data ingestion from S3. |
| August 2022 | Embed ADX dashboards | The ADX web UI and dashboards be embedded in an IFrame and hosted in third party apps. |
| August 2022 | Free cluster upgrade option | You can now upgrade your Azure Data Explorer free cluster to a full cluster that removes the storage limitation allowing you more capacity to grow your data. |
| August 2022 | Analyze fresh ADX data from Excel pivot table | Now you can Use fresh and unlimited volume of ADX data (Kusto) from your favorite analytic tool, Excel pivot tables. MDX queries generated by the Pivot code, will find their way to the Kusto backend as KQL statements that aggregate the data as needed by the pivot and back to Excel. |
| August 2022 | Query results - color by value | Highlight unique data at-a-glance in query results to visually group rows that share identical values for a specific column. Use Explore results and Color by value to apply color to rows based on the selected column. |
| August 2022 | Web explorer - crosshair support for charts | The ysplit property now supports the crosshair visual (vertical lines that move along the mouse pointer) for many charts. |
| July 2022 | Scan operator | The powerful scan operator enables efficient and scalable process mining and sequence analytics and user analytics in ADX. Common scenarios for using scan include preventive maintenance for IoT devices, funnel analysis, recursive calculation, security scenarios looking for known attack steps, and more. |
| July 2022 | Ingest data from Azure Stream Analytics into Synapse Data Explorer (Preview) | You can now use a Streaming Analytics job to collect data from an event hub and send it to your Azure Data Explorer cluster using the Azure portal or an ARM template. For more information, see Ingest data from Azure Stream Analytics into Azure Data Explorer. |
| July 2022 | Render charts for each y column | Synapse Web Data Explorer now supports rendering charts for each y column. For an example, see the Azure Synapse Analytics July Update 2022. |
| June 2022 | Web Explorer new homepage | The new Azure Synapse Web Explorer homepage makes it even easier to get started with Synapse Web Explorer. |
| June 2022 | Web Explorer sample gallery | The Web Explorer sample gallery provides end-to-end samples of how customers leverage Synapse Data Explorer popular use cases such as Logs Data, Metrics Data, IoT data and Basic big data examples. |
| June 2022 | Web Explorer dashboards drill through capabilities | You can now use drillthroughs as parameters in your Synapse Web Explorer dashboards. |
| June 2022 | Time Zone settings for Web Explorer | The Time Zone settings of the Web Explorer now apply to both the Query results and to the Dashboard. By changing the time zone, the dashboards are automatically refreshed to present the data with the selected time zone. |
Azure Synapse Link
Azure Synapse Link is an automated system for replicating data from SQL Server or Azure SQL Database, Azure Cosmos DB, or Dataverse into Azure Synapse Analytics. This section summarizes recent news about the Azure Synapse Link feature.
| Month | Feature | Learn more |
|---|---|---|
| March 2023 | Cosmos DB Synapse Link for Azure Data Explorer GA | Azure Data Explorer supports fully managed data ingestion from Azure Cosmos DB using a change feed. We now support Cosmos DB accounts behind a Managed Private Endpoint or Service Endpoint. For more information, see Ingest data from Azure Cosmos DB into Azure Data Explorer. |
| January 2023 | Cosmos DB Synapse Link for Azure Data Explorer preview | Azure Data Explorer supports fully managed data ingestion from Azure Cosmos DB using a change feed. For more information, see Ingest data from Azure Cosmos DB into Azure Data Explorer (Preview). |
| November 2022 | Azure Synapse Link for SQL | Azure Synapse Link for SQL is now generally available for both SQL Server 2022 and Azure SQL Database. The Azure Synapse Link for SQL feature provides low- and no-code, near real-time data replication from your SQL-based operational stores into Azure Synapse Analytics. Provide BI reporting on operational data in near real-time, with minimal impact on your operational store. For more information, see What is Azure Synapse Link for SQL? |
| July 2022 | Batch mode | Decide between cost and latency in Azure Synapse Link for SQL by selecting continuous or batch mode to replicate your data. Batch mode allows you to save even more on costs by only paying for ingestion service during the batch loads instead of it being continuously on. You can select between 20 and 60 minutes for batch processing. |
Synapse SQL
This section summarizes recent improvements and features in SQL pools in Azure Synapse Analytics.
| Month | Feature | Learn more |
|---|---|---|
| June 2023 | Updated diagnostic settings fields | Nine fields have been added to the dedicated SQL pool diagnostic settings logs. |
| March 2023 | Create alerts for your Azure Synapse dedicated SQL pool | This Customer Success Engineering blog post provides steps to configure alerts for your Azure Synapse dedicated SQL pool and provide recommended alerts to get you started. |
| March 2023 | Performance Tuning Synapse Dedicated Pools - Understanding the Query Lifecycle | This Customer Success Engineering blog post is a deep dive into Understanding Query Lifecycle to Maximize Performance. |
| March 2023 | GREATEST and LEAST T-SQL syntax support | GREATEST and LEAST functions are now available in both serverless and dedicated SQL pools. These scalar-valued functions and return the maximum and minimum value out of a list of one or more expressions. |
| March 2023 | Multi-column distribution in dedicated SQL pools GA | You can now Hash Distribute tables on multiple columns for a more even distribution of the base table, reducing data skew over time and improving query performance. For more information on this generally available feature, see the three options: CREATE MATERIALIZED VIEW, CREATE TABLE distribution options, or CREATE TABLE AS SELECT distribution options. |
| March 2023 | Deploying Synapse SQL serverless using SSDT | SqlPackage's long-awaited support for Azure Synapse Analytics serverless SQL pools is now available starting with the 161.8089.0 SqlPackage. Serverless SQL pools are supported for both the extract and publish actions. |
| February 2023 | UTF-8 and Japanese collations support for dedicated SQL pools | Both UTF-8 support and Japanese collations are now generally available for dedicated SQL pools. |
| September 2022 | Auto-statistics for OPENROWSET in CSV datasets | Serverless SQL pool will automatically create statistics for CSV datasets when needed to ensure an optimal query execution plan for OPENROWSET queries. |
| September 2022 | MERGE T-SQL syntax | T-SQL MERGE syntax has been a highly requested addition to the Synapse T-SQL library. MERGE encapsulates INSERTs/UPDATEs/DELETEs into a single statement. Available in dedicated SQL pools in version 10.0.17829 and above. For more, see the MERGE T-SQL announcement blog. |
| August 2022 | Apache Spark Delta Lake tables in serverless SQL pools | The ability to for serverless SQL pools to access Delta Lake tables created in Spark databases is in preview. For more information, see Azure Synapse Analytics shared metadata tables. |
| August 2022 | Multi-column distribution in dedicated SQL pools | You can now Hash Distribute tables on multiple columns for a more even distribution of the base table, reducing data skew over time and improving query performance. For more information on opting-in to the preview, see CREATE TABLE distribution options or CREATE TABLE AS SELECT distribution options. |
| August 2022 | Distribution Advisor | The Distribution Advisor is a new preview feature in Azure Synapse dedicated SQL pools Gen2 that analyzes queries and recommends the best distribution strategies for tables to improve query performance. For more information, see Distribution Advisor in Azure Synapse SQL. |
| August 2022 | Add SQL objects and users in Lake databases | New capabilities announced for lake databases in serverless SQL pools: create schemas, views, procedures, inline table-valued functions. You can also database users from your Azure Active Directory domain and assign them to the db_datareader role. For more information, see Access lake databases using serverless SQL pool in Azure Synapse Analytics and Create and use native external tables using SQL pools in Azure Synapse Analytics. |
Learn more
For older updates, review past Azure Synapse Analytics Blog posts or previous updates in Azure Synapse Analytics.
- Get started with Azure Synapse Analytics
- Introduction to Azure Synapse Analytics
- Realize Integrated Analytical Solutions with Azure Synapse Analytics
- Data integration at scale with Azure Data Factory or Azure Synapse Pipeline
- Microsoft Training Learning Paths for Azure Synapse
- Azure Synapse Analytics in Microsoft Q&A
Next steps
Feedback
Coming soon: Throughout 2024 we will be phasing out GitHub Issues as the feedback mechanism for content and replacing it with a new feedback system. For more information see: https://aka.ms/ContentUserFeedback.
Submit and view feedback for