I’m excited to announce that we’ve added a new integration for dbt projects in Census. With our new native dbt support, you can take advantage of shared query logic across all your models, and create “last-mile” models that are automatically synced into apps like Salesforce, Marketo, etc. This is the first data modeling integration we’ve ever built in Census and we think it’s the first of its kind for dbt too.
We’ve mentioned dbt as an emerging piece of technology in the modern data stack. For those of you who aren’t familiar, dbt is a tool that helps you write & run SQL transforms in your data warehouse. The key benefit of using dbt is that your SQL is stored in version control, which means there’s a history of all changes. This makes it much easier for teams to collaborate. The other key feature of dbt is that you can create shared logic for your SQL code (e.g. you can build intermediate views on your data or you can write macros for repetitive parts of your queries). All of this makes it an ideal tool for the modern data stack where you can maintain all the code that cleans & prepares your data for action.
Census can now materialize select dbt models directly into your external tools like Salesforce, Marketo, Customer.io, etc.
As of today in Census, you can connect to a dbt project stored on GitHub, select models whose output you’d like to make available for publishing, and take advantage of our sync engine to get the data into the hands of your business teams. Let’s walk through what this looks like in practice.
- Connect to GitHub. When you go to the Models tab, you’ll be prompted with a choice to either store & edit SQL directly in Census or connect to GitHub and select a repository where you keep your dbt data models. You don’t need to create a new project for Census – you can keep all your transform logic in one repository.
- Select models for Census. By default when you connect a new dbt project, Census will only look for models that have been tagged with the keyword census. This way you can decide which models or groups of models should be available for Census to sync into other applications (most of us don’t want to sync our internal staging transforms). You can configure this selector for your project (e.g. path:/models/publish).
- Sync models to destinations. The best part about our integration is that you don’t have to think about managing yet another materialization schedule. We automatically respect your existing setup, whether your dbt runner is in-house or in dbtCloud. The models you expose to Census are treated as ephemeral for our syncs so they can reference your other models without having to think about their materialization. In other words, you can think of Census as responsible for materializing these models (incrementally) into your external tools like Salesforce, Customer.io, etc.
As you can see in this video, it's easy to connect existing dbt projects with Census & to take advantage of our integration. If you would like to migrate your existing Census models to a dbt project, reach out to our team or your dedicated CSM.
PS: We plan to do a lot more with dbt in the months to come so stay tuned :-)