Acceldata Data Observability Cloud v2.9.0

Acceldata has released our latest version of Acceldata Data Observability Cloud (ADOC). Version 2.9.0 is now generally available, and it provides enhancements to compute, data reliability, monitoring, alerting, as well as Azure integrations.

The following video offers a snapshot of the enhancements, and more details on new capabilities and features are explained below (along with links to ADOC documentation):

Data Reliability

• Data Cadence and Freshness Policy: ADOC provides the capability to activate a data freshness policy for an asset, which effectively identifies anomalies and breaches in SLAs. For more information, see Asset Detail.
• Dynamic Filters Enhancement for Bulk Policies and DQ Policies: Added a spark SQL filter panel to selectively target specific data records that need to be evaluated for data quality issues. For more information, see Data Quality Policy.
• Integration with GCS Secret Manager: ADOC can now use Google Cloud Secret Manager to store your secrets. For more information, see GCS Secret Manager
• Introduced BigQuery Lineage Data Support: ADOC has been enhanced to offer the functionality of showcasing lineage details for assets originating from BigQuery data sources. For more information, see Lineage.
• Enhanced Profiling of Data Assets: ADOC has enhanced its profiling capabilities to encompass the profiling of semi-structured data assets as well.
• Crawling a Single Asset to a Data Source: ADOC provides you with an option to crawl either a single asset or select specific assets to add to an existing Snowflake, Redshift, or Databricks data source. For more information, see Managing Data Sources.

Monitors and Alerts

• Added New Databricks Monitor: Introduced the following new monitors in Databricks Workspace: Running Clusters after Work Hours.
• Import Azure Data Factory Monitors: Added the following ADF monitors that can be imported to the monitors list: Factory Cost, Pipeline Cost, and Failed Activities.

Data Source and Data Plane

• Added Support for Integration of Azure Data Factory: Azure Data Factory can now be integrated with ADOC as a data source. Our platform includes a comprehensive dashboard that meticulously tracks various costs and resource utilization across factories, pipelines, and activities within Azure Data Factory. For more information, see Azure Data Factory.

Compute

• Azure Actual Cost Retrieval for Databricks Workspace: Introduced a new capability in ADOC that empowers Databricks users to access real-time cost information for their Databricks resources on Azure cloud platform.
• Query Studio Enhancements: Introduced a new dashboard that allows a provision to see user specific details such as the queries executed by the user and the corresponding costs incurred by them. For more information, see Query Studio.
• Snowflake Warehouse Utilization: ADOC has enhanced its Snowflake warehouse monitoring capabilities, enabling you to gain deeper insights into warehouse utilization, including productivity and wastage metrics. For more information, see Snowflake Warehouse.
• Create External Stage for Snowflake Storage Integration: ADOC has enhanced the Compute Observability Setup page by providing an option to enable the creation of an external stage for storage integration.
• Use Service Principal for Databricks Connection : ADOC now allows you to use service principal identity to give access to your AWS or Azure Databricks resources.

‍

Photo by Bruno Figueiredo on Unsplash