Data-driven businesses in the private sector collect so much data they have the answer to every question right at their fingertips. The tricky challenge then has morphed into deciding which questions to ask in the first place. You can think of it like the well-stocked-fridge dilemma: You have so many options in your fridge or pantry that you don’t know what to cook for dinner (aka choice paralysis).
Data-driven decisions in the public sector, however, come from the other end of the spectrum. For nonprofit and government agencies, data questions have to be asked first, intentionally, and then data is collected to answer it. It’s more streamlined – like choosing a dinner recipe and then going to the store to buy what you need. 🚘
“Nonprofits and government agencies don’t start with data. They don’t have it, to begin with,” explained Marc Stone in a recent episode of The Sequel Show with Census Co-founder and CEO Boris Jabes. Marc is the head of analytics at ClickUp, but before that he spent nearly eight years with Clear Impact, a business consultant for the public sector.
“Businesses take their data for granted because they have transactions with all their customers,” he said. “If you’re running a literacy nonprofit, you don’t have user IDs; you don’t have individuals’ names and birth dates. Even the government doesn’t necessarily have access to giant repositories of constituent data because of personal data privacy concerns.”
Without the infrastructure to collect massive swaths of data, public sector organizations must select data projects more carefully. They have to start with detailed KPIs (vs figuring out the best KPIs as you go) because building data collections from scratch means you can’t waste resources on metrics you can’t use. ❌
Each data point comes with an expense, so local governments and agencies are especially judicious about what data they choose to collect to ensure it fits within the budget. Data collection means putting manual work into opening envelopes and knocking on doors, so most public sector data projects don’t have a pipeline of collection triggers.
“At least 25 percent of the projects I worked on had some human pipeline,” Marc said. “To know if you had an impact on people’s lives, you have to go ask them, whether that’s by mailing out random sampling surveys or checking in with someone. Any time you see governments publishing numbers, rest assured, there was a ton of legwork involved in getting those numbers.”
According to Marc, businesses should pay attention to public sector data governance, especially when it comes to starting with KPIs and big-picture thinking. At Clear Impact, Marc worked on data projects that spanned years, not weeks.
At ClickUp, that background has helped him to focus on quick wins within a long-term framework. ClickUp focused on the methodology first: They architected the data warehouse to support machine learning from the start which forced the team to follow best practices and think long term as it built its data stack and pipeline.
“My previous experience had led me to always look two, three, five years down the road,” Marc said. “Then I paired that with the Clickup culture of ‘What can we deliver today?’”
Applying nonprofit thinking to a for-profit company
Speed is essential in a competitive SaaS arena like the project management space where ClickUp plays. However, “move fast and break things” isn’t a terribly efficient way to run data operations. Instead, the ClickUp team takes out a page out of the nonprofit data playbook Marc helped write by treating fast data as a privilege. This means finding the middle ground between a resource scarcity mindset and an agile one.
On the speed side of the spectrum, the ClickUp team values the ability to pivot over designing “perfect” architecture. In practice, this means they build processes, scripts, and tools that adapt in near real-time, so changes in the system architecture don’t bring everything crashing down.
Many data people might consider one of ClickUp’s key processes a design mistake. Rather than writing incremental update code for small tables, the team does daily wholesale refreshes of tables throughout the pipeline.
“I think it has negated the issue of a fast-changing engineering environment, realizing engineering added a column, and what that does to the data,” Marc explained.
Those engineering changes, often subtle shifts in semantics, can, when changes have unanticipated consequences, create data headaches and downtime. As data teams adapt structure from engineering disciplines for their own use, schema changes are one luxury software engineers have that data engineers just don’t.
“People need to embrace the idea that it’s always going to be a little bit broken,” Boris said during the recent episode. “Getting the thing right isn’t as important as getting the direction right. We can improve the thing as we go.”
Getting the thing right isn't as important as getting the direction right. We can improve the thing as we go.
To channel the intentionality of nonprofit data workflows, ClickUp has taken a novel approach to improvements that hinges on embedding smaller teams of engineers and analysts throughout the pipeline. Like water moving through finer and finer sieves, the data is cleaned repeatedly as it makes its way through filters of data engineers, data analytics engineers, and machine learning engineers. 🔁
This data infrastructure improves overall data management: When a person downstream identifies an error, they know right where to go to find the correct information. It gives engineers and analysts the luxury of staying in a lane and focusing on specific types of errors (and streamlines decision-making across the ecosystem).
“Everyone can overlap by about 50 percent into the layer next to them, but whatever level you’re at, you only have to worry about specific types of errors,” Marc explained. “The layers look tedious if it’s your first day at ClickUp. By your fiftieth day, you’re like, ‘Thank God these layers are here.’” 🙌
Every Friday, the entire data team gathers for data sharing, covering both what they did that week and what’s on the horizon. Because of how quickly the company moves, this regular communication cadence is crucial.
This weekly roundup is when the team discusses new analytics requests. And yes – there are always new analytics requests. Going back to the lessons he learned in the public sector about making judicious use of resources, Marc stressed the importance for every team to have a good gatekeeper reviewing these requests and deciding the ROI of every project.
Is it a good use of data resources?
Or can the question be answered by something the team already has?
After all, just because a business has more resources than a nonprofit doesn’t mean it makes sense to squander them. As businesses start to recognize the benefits of the public sector’s approach to data, we may start to see more quality data and less quantity of data.
Want to learn more? You can catch the full conversation between Boris and Marc below, or on your favorite streaming platforms. 🎧
Got thoughts and opinions about this topic? Join the conversation around this, and many other data best practices, in ✨ The Operational Analytics Club. ✨