Sync with Confidence: Stay ahead of data surprises w/ warehouse-centric observability for rETL | Census
Katy YuanMay 12, 2022
Katy is a Product Marketing Manager at Census who loves diving into startups, SaaS technology, and modern data platforms. When she's not working, you can find her playing pickleball or Ultimate Frisbee.
Data teams desperately want to support the requests of business teams. But just when data engineers settle into impactful, career-building project work, they’re often interrupted by a data quality bug or a broken pipeline out of nowhere…👇 (Hint: Make sure you have your sound on 🔉)
Luckily, Census helps you level up your visibility and bug-fighting skills. 🐛
Today, we’re announcing an entire set of ✨ warehouse-centric observability ✨ capabilities that ensure data quality for reverse ETL, so data teams can sync data into downstream business tools with confidence (and without all the bug smashing).
🥇 Census: The most observable reverse ETL platform
Census has always had the fastest and most reliable reverse ETL connectors. However, shipping all your data everywhere isn’t the goal – it’s shipping high-quality data everywhere your ops teams need it.
With the best observability, logging, and debugging capabilities centered around your warehouse, Census provides a single place for the data team to track where data flows to downstream tools and the health of those pipelines.
Now, you can Sync with Confidence when sending and updating large volumes of data in downstream tools (meaning you can move faster when delivering data to ops teams).
Why “warehouse-centric” matters for observability
We believe all data and business logic should be centralized in the warehouse, the central hub for your data team.
Unlike other reverse ETL platforms, Census’s advanced observability tools work directly on your warehouse. No wrestling with S3 buckets, external files, or additional tools to get visibility into your data pipelines.
Native integration with your warehouse is a big deal:
All your logs are in the warehouse so you can analyze and combine them with all your other data.
All operations are compatible with SQL. You can even incorporate sync logs into your Census SQL models.
Use the tools you already have: BI tools, SQL clients, and data testing frameworks like Great Expectations or dbt.
What’s included in our observability suite?
🗄️ Sync Logs
Sync Logs provide detailed logs of each data point you’ve synced so you can audit, troubleshoot, and create alerts using the most granular information. We store logs directly in your warehouse so you can analyze and combine them with all your other data. You can also query them with SQL models in Census!
Logging in the warehouse enables transparency for all data flows 💪
Easily reference and understand what records failed and why, as well as which were successfully synced, so you can feel confident your data has synced correctly or quickly troubleshoot any issues that arise.
Sync Logs are compatible with SQL clients, BI tools, dbt, and more
🕵️ API Inspector
The API Inspector helps you play Data Detective with real-time transparency around API requests and responses. See the API calls Census makes to destinations, down to the individual row, so you can track and fix errors as they happen.
Drill down into requests & responses in real time while your sync is running
🔮 Sync Dry Runs New!
Sync Dry Runs provide a detailed summary report of how your data downstream will change – before you make any changes. The report shows the expected sync time, source record errors, and destination changes (number of records created, updated, and deleted). A dry run serves as a sanity check so both data and ops teams are aware of the scope of their updates and can make final approvals.
Sync Dry Runs are available today on Salesforce, our most popular destination, and will be available on other connectors very soon.
Run a test sync without changing any records to preview how the destination will change
⚠️ Custom Alerting
With custom alerting, Census alerts you of invalid and rejected records and general errors to help you take proactive action to fix issues before they cause major breaks. Your alerts aren’t just limited to sync failures either. You can configure alerts per individual sync and trigger them based on a percentage threshold of rejected records. Alerts are available quickly via email, as well as Slack or other tools, when something goes wrong.
Configure thresholds for alerts, per individual sync, to avoid alert fatigue
⛔️ Invalid / Rejected Records
In the Sync History tab of any sync, you can click into a specific failed run to see a sample of invalid or rejected records (up to 100), and the reasons why they failed. Census automatically generates this diagnostic log in the app UI for all failed syncs.
Every plan, including our free tier, has access to alerting and invalid/rejected records. Sync Dry Run and the API Inspector are enabled for any paid Census plan. Sync Logs are only available on Business plans and above.
We’re dedicated to helping data folks stand out 🦩 and drive more impact in the business, instead of spending their days overwhelmed by manual data tasks.
In the near future, we plan to expand our existing observability suite and build even more capabilities to improve visibility and data quality – so even more teams can sync with confidence. Our roadmap includes integration with dedicated observability platforms, field-level details for Sync Dry Runs beyond record-level, and more collaboration tools for ops users (like Salesforce admins) to get involved with upstream data.
Ultimately, we’ve aligned our mission around enabling both data and ops teams to achieve Operational Analytics, which means action, automation, and trust with data for everyone, regardless of technical skill level.
🥳 If you’d like to help us build this vision, we’re hiring!
