Best Practices

Why you need reverse ETL right now | Census

Allie Beazell
Allie Beazell December 02, 2022

Allie Beazell is director of developer marketing @ Census. She loves getting to connect with data practitioners about their day-to-day work, helping technical folks communicate their expertise through writing, and bringing people together to learn from each other. Los Angeles Metropolitan Area, California, United States

Welcome to part one of our series on evaluating reverse ETL tools. This was updated December 2, 2022 to better capture the current state of reverse ETL tooling. We recommend grabbing our new-and-improved companion guide, The definitive guide for evaluating reverse ETL tools, to complete the set. 📚

In this installment, you’ll learn why you should invest in reverse ETL, and, more importantly, why you should invest it now. We’ll focus on:

  • Why reverse ETL now ⏱️, including:
  • How business teams have contributed to this need 👩‍💼
  • How data engineers have contributed to this need 📊
  • How a dedicated reverse ETL tool can streamline and simplify your data architecture 🧰
  • How reverse ETL can help you adapt at the flip of a switch 💡
  • How the push toward more personalized contributed to this need ❤️
  • The new era of data for business and data teams made possible by reverse ETL right now 🙌

Every day, your customers get up in the morning, make some coffee, go to their computer, and start a conversation with you.

No, they aren’t using words. They’re communicating with data.

If you’re like most modern, data-driven companies, you’ve invested a ton of money to figure out how to listen to these conversation points (about $467 billion globally in 2020, according to Gartner) with new data pipelines. But you still haven’t cracked that final piece: How do you turn all that data into a two-way conversation? After all, while every solid relationship should be mostly listening, you need to show you understand your partner’s needs, too.

Reverse ETL can drive personalization and growth like never before while unlocking the data team’s ability to do their best and brightest work to operationalize analytics across the company. This means happier data teams, and stronger, more communicative dialogue with customers. But only if the new generation of data leaders and RevOps experts know how to evaluate and implement these tools successfully.

That’s why we’ve created our reverse ETL evaluation content series. This collection of resources (starting with the installment you’re reading right now and culminating in The definitive guide to evaluating reverse ETL tools) are designed to help you parse why reverse ETL is so important, examine the trade-offs of build vs. buy, and understand the key things to look for when partnering with a reverse ETL vendor.

Why reverse ETL (and why reverse ETL now)? ⏱️

Reverse ETL is a doorway that allows you and your company to step into the new era of Operational Analytics. Instead of just using business intelligence to look in the rearview mirror and identify trends for long-term strategy, you can leverage your insights to take action in your day-to-day operations.

But why reverse ETL now? Well, that comes down to five things:

1. Business teams are using more SaaS apps than ever 👩💼

According to the 2020 SaaS reports, companies have an average of 137 unique apps! 😱 Every day, teams need to buy more and more specialized software to stay competitive and serve the ultimate customer experience. But they can’t get the lifetime value out of all these specialized (read: expensive) apps without trustworthy, real-time data access. Instead, operations teams end up tangling integrations together or data teams get saddled with building custom connections to and from these apps that cause more issues than they fix.

2. Data engineers are too valuable to be pipeline babysitters 🧑🍼

The increasing number of applications used by business teams has caused the number of support tickets and custom requests from Ops teams to surge, leaving in-house engineers and analysts scrambling to keep up. Or even worse, they fail to manage your data product, making the data less valuable (and reducing everyone’s trust in data along the way).

Building an integration once is easy. Maintaining it and the corresponding SLAs is awful. And without a dedicated reverse ETL solution, data teams waste their skills buried under error messages and integration maintenance. Companies should instead be leveraging their high-value data folks on transforming data to unlock new value for the business, whilst simultaneously enabling business teams to be more self-service.

3. Companies need better personalization and engagement to be competitive

Customer journeys are longer and the relationships we build with each user in our CRMs are more complex than ever before. Sales, marketing, and customer success teams need a way to action customer data from each customer touchpoint to ensure they convert. Reverse ETL enables you to automate personalization at scale and ensures the data you’re collecting and ingesting makes it back to the front-line teams that need it most.

4. Dedicated services do a better job than other tools in your data stack doing things “half-well”

The plug-and-play nature of point-to-point platforms like Workato, Zapier, or Tray often entices teams with quick fixes. But relying too heavily on these duct tape solutions can quickly get messy. Fully integrating point-to-point solutions requires exponentially more connectors as your data stack grows, adding more complexity to your tools and increasing your costs. In fact, the number of connections grows by the square of the number of applications, meaning just eight apps could require as many as 64 distinct connections to keep your entire modern data stack in sync.

So, instead of a messy, spaghetti pile of point-to-point integrations, you can use reverse ETL to architect your data infrastructure as a series of orderly spokes around your central hub (data warehouse). This creates a single source of truth informing each application and workflow within your stack, eliminating silos, and making you truly data-informed. Plus, by centralizing, you reduce the overall complexity and number of connections, thereby reducing your bills.

5. You don’t need to build new for a different use case. Adapt with just the flip of a switch.

Connecting teams throughout your organization to the warehouse using reverse ETL empowers them with data-enriched, valuable context about what your customers are doing in real-time. By quickly and easily tapping into powerful data insights, reverse ETL is often the difference between a missed connection and a life-long customer bond.

Marketing, for example, is one of the core use cases for reverse ETL. Here, speed really matters. Why? The faster you get data into your marketing tool, the faster you can reach your customer and delight them with personalized messages. For example, with reverse ETL, marketing teams can build hyper-personalized marketing campaigns in HubSpot by merging product, customer support, and sales data to power customer segmentation. Now, successful business teams can self-serve deep, useful data so there are no more missed opportunities.

What types of companies use reverse ETL?

Tons of leading companies use reverse ETL.

Why? For business users, reverse ETL puts fresh, actionable product usage data in all their frontline business tools whenever they need it. For data teams, reverse ETL allows their data to reach its full potential to fuel operational excellence for every business team that consumes it.

So, whether you’re in sales, marketing, data, or RevOps, you can use reverse ETL to operationalize your data, make decisions based on that data, and drive more revenue.

It’s time for a new era for the data team

This scenario probably sounds familiar to everyone reading this: The business team gets a shiny new tool. They hack together as much functionality as they can in self-serve mode before pushing the remaining work over to the data team like a wonky game of telephone. The data team, overloaded with similar requests from every stakeholder in the business, churns out a “good enough” solution as quickly as possible. Three months later, the connection breaks. We’ve said it once and we’ll say it again: Building a data integration once is easy, but maintaining it is awful.

So, we’re not doing that era of data anymore.

See this? ☝ This is how we’re not going to operate anymore thanks to reverse ETL and operational analytics.

The flow of data from a cloud data warehouse (like Snowflake) into operational systems is no longer a nice-to-have stream of information, but an integral part of the modern company’s critical path, with data teams and tooling the shepherds that deliver it safely. After all, investment in quality reverse ETL tooling is like taking a well-engineered data cruise liner to your destination rather than an error-laden tugboat. Hint: The tugboat might sink your business.

Ready to learn what’s next? In part two of our series, we debate whether you should build vs buy a reverse ETL solution. Or, if you’re already ready to start shopping around for reverse ETL tools, grab a copy of The definitive guide to reverse ETL tool evaluation to make sure you find the best tool for your use case.

✨ Want to chat with a Census expert to see just what our reverse ETL tool can do? Book a demo with a product specialist.

Related articles

Product News
Sync data 100x faster on Snowflake with Census Live Syncs
Sync data 100x faster on Snowflake with Census Live Syncs

For years, working with high-quality data in real time was an elusive goal for data teams. Two hurdles blocked real-time data activation on Snowflake from becoming a reality: Lack of low-latency data flows and transformation pipelines The compute cost of running queries at high frequency in order to provide real-time insights Today, we’re solving both of those challenges by partnering with Snowflake to support our real-time Live Syncs, which can be 100 times faster and 100 times cheaper to operate than traditional Reverse ETL. You can create a Live Sync using any Snowflake table (including Dynamic Tables) as a source, and sync data to over 200 business tools within seconds. We’re proud to offer the fastest Reverse ETL platform on the planet, and the only one capable of real-time activation with Snowflake. 👉 Luke Ambrosetti discusses Live Sync architecture in-depth on Snowflake’s Medium blog here. Real-Time Composable CDP with Snowflake Developed alongside Snowflake’s product team, we’re excited to enable the fastest-ever data activation on Snowflake. Today marks a massive paradigm shift in how quickly companies can leverage their first-party data to stay ahead of their competition. In the past, businesses had to implement their real-time use cases outside their Data Cloud by building a separate fast path, through hosted custom infrastructure and event buses, or piles of if-this-then-that no-code hacks — all with painful limitations such as lack of scalability, data silos, and low adaptability. Census Live Syncs were born to tear down the latency barrier that previously prevented companies from centralizing these integrations with all of their others. Census Live Syncs and Snowflake now combine to offer real-time CDP capabilities without having to abandon the Data Cloud. This Composable CDP approach transforms the Data Cloud infrastructure that companies already have into an engine that drives business growth and revenue, delivering huge cost savings and data-driven decisions without complex engineering. Together we’re enabling marketing and business teams to interact with customers at the moment of intent, deliver the most personalized recommendations, and update AI models with the freshest insights. Doing the Math: 100x Faster and 100x Cheaper There are two primary ways to use Census Live Syncs — through Snowflake Dynamic Tables, or directly through Snowflake Streams. Near real time: Dynamic Tables have a target lag of minimum 1 minute (as of March 2024). Real time: Live Syncs can operate off a Snowflake Stream directly to achieve true real-time activation in single-digit seconds. Using a real-world example, one of our customers was looking for real-time activation to personalize in-app content immediately. They replaced their previous hourly process with Census Live Syncs, achieving an end-to-end latency of <1 minute. They observed that Live Syncs are 144 times cheaper and 150 times faster than their previous Reverse ETL process. It’s rare to offer customers multiple orders of magnitude of improvement as part of a product release, but we did the math. Continuous Syncs (traditional Reverse ETL) Census Live Syncs Improvement Cost 24 hours = 24 Snowflake credits. 24 * $2 * 30 = $1440/month ⅙ of a credit per day. ⅙ * $2 * 30 = $10/month 144x Speed Transformation hourly job + 15 minutes for ETL = 75 minutes on average 30 seconds on average 150x Cost The previous method of lowest latency Reverse ETL, called Continuous Syncs, required a Snowflake compute platform to be live 24/7 in order to continuously detect changes. This was expensive and also wasteful for datasets that don’t change often. Assuming that one Snowflake credit is on average $2, traditional Reverse ETL costs 24 credits * $2 * 30 days = $1440 per month. Using Snowflake’s Streams to detect changes offers a huge saving in credits to detect changes, just 1/6th of a single credit in equivalent cost, lowering the cost to $10 per month. Speed Real-time activation also requires ETL and transformation workflows to be low latency. In this example, our customer needed real-time activation of an event that occurs 10 times per day. First, we reduced their ETL processing time to 1 second with our HTTP Request source. On the activation side, Live Syncs activate data with subsecond latency. 1 second HTTP Live Sync + 1 minute Dynamic Table refresh + 1 second Census Snowflake Live Sync = 1 minute end-to-end latency. This process can be even faster when using Live Syncs with a Snowflake Stream. For this customer, using Census Live Syncs on Snowflake was 144x cheaper and 150x faster than their previous Reverse ETL process How Live Syncs work It’s easy to set up a real-time workflow with Snowflake as a source in three steps:

Best Practices
How Retail Brands Should Implement Real-Time Data Platforms To Drive Revenue
How Retail Brands Should Implement Real-Time Data Platforms To Drive Revenue

Remember when the days of "Dear [First Name]" emails felt like cutting-edge personalization?

Product News
Why Census Embedded?
Why Census Embedded?

Last November, we shipped a new product: Census Embedded. It's a massive expansion of our footprint in the world of data. As I'll lay out here, it's a natural evolution of our platform in service of our mission and it's poised to help a lot of people get access to more great quality data.