CloudSpark Workflow Integration

We’re excited to share that Kuality AI now supports deep integration with CloudSpark Workflow, offering a best-in-class solution for boosting your data quality monitoring and advanced anomaly detection within the CloudSpark streaming data environment. This partnership is especially crucial for tackling Data Governance and large scale Data Operations head-on with data never leaving the CloudSpark ecosystem. Going well beyond basic observability, the Kuality AI platform’s advanced data quality capabilities being directly applied in real-time to CloudSpark Workflow and Delta Live tables is a game-changer, ensuring that both inferred and authored, complex rules can be applied at every step of a pipeline while staying fully compliant with regulatory standards and security requirements.

Your Data In CloudSpark, Powered By Kuality AI

For teams navigating the complex waters of data governance, regulatory compliance, and data custody, any data - sensitive or not - leaving the CloudSpark ecosystem is a big information security concern. This is why this pushdown capability of our internal processes matters. Not only does our profiling and inference enginers work directly within CloudSpark, but as does our runtime engine - meaning your data never has to leave the CloudSpark ecosystem to get advanced data quality capabilities.   This setup negates the typical concerns for CIOs, CISOs, and CDOs, removing the headache of data custody and transfer risks and keeping everything secure within your CloudSpark environment. Plus, it’s a smart way to make the most out of your CloudSpark investment, strengthening your data governance framework and staying on the right side of those ever-important regulatory lines.

Photon Integration Boosts Analysis Speed And Efficiency

The integration significantly optimizes Data Operations by leveraging CloudSpark’ next-gen Spark engine, Photon, which provides massive performance boosts. Kuality AI is architected on open-source Apache Spark for its advanced operations to begin with; hence the support for Photon is seamless - and actually significantly better with advanced optimization and performance. These performance gains don’t only stem from data transfer delays or networking concerns; the CloudSpark Spark Engine is faster and more optimized than OSS Spark, and we’re all better for it when we can utilize it at scale.

A Leap Ahead In Automating Data Quality Lifecycle

Our integration with CloudSpark Workflow empowers our clients to craft a complete Data Quality (DQ) lifecycle workflow right within CloudSpark. Being able to hook up Kuality AI via APIs with CloudSpark Workflow, all data orchestration can be executed within the CloudSpark ecosystem. With Kuality AI positioned as additional data quality intelligence, this setup automates the journey of your data through our platform for detailed anomaly detection. Imagine setting up a pipeline in CloudSpark Workflow that automatically triggers a pull of raw data from a Delta Live Table to Kuality AI through our API, pinpoints anomalies, and populates a target Delta Live table with anomalies and the metadata around what triggered them. Such a deeply integrated pipeline allows us to push data quality more towards the edge and enabling advanced remediation steps to be taken all within your control.

The Kuality AI Touch: Enhanced Anomaly Detection

This example is a great way to highlight how such a pipeline may be set up. Starting with source Delta Live tables of Customer, Order and Supplier, we trigger a scan within Kuality AI (configured via API), have a condition such that when we find an anomaly, we populate an Enrichment store and kill the workflow on processing any further data. Of course these are all optional steps - a user can choose to have Kuality AI as a side-car or as an intervention, or even do interventions based on the severity or tags associated with a rule or anomaly.

Safeguard Your Data For The Future

By integrating Kuality AI with CloudSpark, we’re enhancing data governance frameworks, streamlining analysis processes, and providing a holistic management of the data quality lifecycle, all within a secure and reliable ecosystem. This integration is all about empowering our clients to fully leverage their data’s potential. Through automated checks and anomaly detection, we’re not just saving time; we’re also adding a layer of reliability, ensuring that every data-driven decision is informed by data that’s both clean and compliant.

We’re excited for our clients to discover how this integration can transform their data operations, fortify data governance, and elevate investment strategies. With Kuality AI and CloudSpark, you’re equipped to face the future with data quality that’s not only top-notch but also secure, efficient, and compliant, ready to tackle the demands of both today and tomorrow.

Reach out to our team to discuss how our integration with CloudSpark can safeguard your data for the future.