Qualytics makes it easier than ever to take action against anomalous data. What’s the point of finding anomalies if you can’t do anything about them?!?
Operational Data Quality can be broken down into 3 main steps:
- Establish Fit: whether it’s automated or manual, technical or business rules – this is when you define what expectations of the data is established, alas what does good look like?
- Detect Anomalies: use your rules to find anomalies in your data
- Remediate: do something about the anomalies you catch, day in and day out
No one step is more important than the other, and you can’t manage a comprehensive DQ program if you don’t do all three.
Remediate can mean a lot of things to a lot of people. You may want to get a Teams or Slack message every time you find an anomaly associated with a specific business rule. You may want to create a JIRA or ServiceNow ticket every time a specific group of data assets have an anomaly. Or, you may want to orchestrate a series of operations based on a specific behavior observed.
Reality is that in most organizations, analysts and data stewards closest to the problem at hand have the hardest time supporting remediation efforts – they not only spot issues manually, but then have to use muscle memory and non-contextual tickets to communicate around an issue.
This is why we built Flows. Technical and non-technical stakeholders need a way to construct automated pipelines that can chain actions together and configure triggers based on events & specific behaviors.
What are Flows
At its core, Flows allows users to define sequences of actions that are initiated by specific triggers. These triggers can be configured based on various events, filters, and operational conditions, providing a high degree of flexibility.
For instance, when an anomaly tagged with “CDE” is detected, a Flow can be set up to:
- Send a notification via Microsoft Teams
- Initiate a new data profile
- Await the completion of the profile
- Launch a subsequent data scan
- Concurrently dispatch an email to a designated user group
- When an anomaly in a subsequent data scan is detected, send a notification to an external system like Pager Duty
This level of automation ensures that data quality issues are promptly addressed, minimizing manual intervention and reducing the risk of oversight.
Getting Started with Flows
Qualytics has prioritized user experience in the design of Flows. To create a new Flow, navigate to Flows from the left pane and click the “Add Flow” button. From there, it’s WYSIWYG – drag and drop your desired triggers and corresponding actions, and build your own workflow within minutes. Managing existing Flows is equally straightforward; users can deactivate, delete, or edit Flows through intuitive controls, as well as easily navigate what’s happening in real-time, ensuring that the system remains adaptable to evolving business needs.
Enhanced Triggering and Action Options
The versatility of Flows is further exemplified by its diverse triggering options. Triggers such as “Operations Complete,” “Anomalous Table/File Detected,” and “Anomaly Detected” provide flexible, event-driven automation. Users can fine-tune these triggers using filters like tags, rule types, or anomaly weights, allowing for precise control over when and how actions are initiated.
On the action side, Flows supports a wide array of notifications and operations. Notifications can be sent through various channels, including in-app messages, Emails, Slack, Microsoft Teams, Pager
Duty, and custom HTTP actions. Operational tasks such as cataloging, profiling, and s
canning can also be triggered, enabling comprehensive data quality manageme
nt workflows to be executed seamlessly.
Real-Time Monitoring and Activity Tracking
To ensure transparency and control, Qualytics has incorporated real-time monitoring capabilities within Flows – users can view execution history and track real-time progress in the Flow Executions section where ongoing Operations, displaying step-by-step status associated with each step to easily navigate and address any issues associated with a specific Flow.
tl;dr
The introduction of Flows marks a significant advancement in Qualytics’ platform, offering users an intuitive and powerful tool to automate and manage data quality processes. By enabling the creation of customized pipelines that respond to specific events, Flows not only enhances operational efficiency but also ensures that anomalies are addressed effectively and timely. As data environments continue to grow in complexity, capabilities such as Flows will be indispensable in maintaining the integrity and reliability of organizational data.