Your customer data is all over the place. But how do you bring it all together?
As you grow up as a company, each of your sales, marketing, and customer-facing teams buy tools to help solve the problems they have. In isolation, this makes total sense. But a growing set of tools within teams means your customer data ends up being siloed away.
… and so on. With data siloed away in different tools, your teams fall out of sync. Everyone is torturing data, trying to build a full picture of the customer journey with what data they have access to.
We call it the fRanKEnSTacK! And here's how you solve it.
Customer data integration is the process of bring all this siloed, separated data together again - breaking down your fRanKEnSTacK!
The goal is to get the data your sales, marketing, support, product, success - every customer facing team - needs in the tools they want to use, in the format that’s useful to them.
It’s helpful to think of a three step process for managing customer data - Extract-Transform-Load or “ETL”
Extract-Transform-Load is a common language for data integration. This is analogous to exporting a CSV file from one tool, sorting that data in a spreadsheet, then uploading that spreadsheet data to another tool. Extract-Transform-Load.
With all your customer data synced together, your tools, teams and data can act together as one system. There are huge advantages of data integration for your teams:
But data integration is hard - here’s why companies struggle to bring it all together, and end up with a fRanKEnSTacK instead.
There are many issues with customer data integration. At Hull, we call this the “6-6-6 of customer data integration” - the devil is in the data. These are the six common sources of customer data, six hard combinations of customer data, and six common customer data integration techniques.
Let’s start with the first…
At Hull, we say “hire tools to do a job”. There are six common types of tools which store customer data and need integrating with each other.
Analytics Tools track website sessions for analysis, and sometimes action too (like live chat triggers).
CRMs keep a record of each contact, their associated companies, deals and conversations. These form the basis of conversations that sales (and other teams like customer success) will have with leads, users, and customers. This also includes your sales tools which closely integrate with your CRM.
Marketing Automation & Email Tools keep a record of contacts, messages, reactions (email opened, email clicked etc.) so they can bucket those contacts into future workflows and campaigns.
Product Databases record each users actions and profile data. This tends to be bespoke for each business.
Data enrichment providers - The data about companies and individual people exists out there publicly. Data enrichment providers gather this and make it available to purchase and “enrich” your existing customer data.
Ad Audiences - Most advertising networks work with tracking pixels (and sometimes an email list) to build audiences to target ads. These audiences usually remain locked within the ad platform which makes it hard to integrate ad engagement with your other tools, data, and campaigns.
With your customer data scattered across this collection of different tools and data sources, it’s not simply a case of connecting them up. Not all data can be easily combined.
If data is hard to combine, then it is hard to use collectively as one. By understanding these differences, we can understand how best to integrate them together.
Client side and server side data - client-side data that is tracked from a users device (like a Google Analytics tracking script) is different to server-side data is tracked and recorded elsewhere like your CRM or product database.
Tools and databases - Databases tend to be the domain of technical teams only, and their setup tends to be unique to each business (like a customer’s account, billing, and usage history). These need someone technical (who knows and has access to the database) to stitch it to each tool you’re using in a way that makes sense.
Events (a users actions) and Properties (things about the user) are often different things too. Events like
Viewed pricing page then
Requested demo then
Opened email are recorded in a different way, like an analytics tool, whereas properties (like name, email, company) tend to be captured from form submissions, manual entry into a CRM, and from third party enrichment databases.
People and Company Data - For B2B marketing, it’s important to distinguish between individual people and company accounts. Companies buy things, but people take action. Often it’s hard to build a complete picture of the company account (so you can qualify them for sales) from a set of individuals actions (particularly from freemail addresses like gmail), without asking for a ton of data from forms which butcher your conversion rates.
Before (anonymous) and after signup (identified) users - Most tools which keep a record of a user or contact only do so once they’ve been “identified” with an email address, but they miss interesting actions and activity before hand (perhaps you viewed the pricing page thrice?).
Manual Data Entry and Automated Data Entry - Even if we can capture data, there’s often differences between data shared in one place vs another. Your CRM might have conflicting data to your new customer’s account signup. Once you’ve started automating data flows, how you resolve these differences in data is important.
Without carefully integrating your customer data, you will end up with a fRanKEnSTacK! There are some best practices to tie data together before it gets the chance to scatter and morph into a monster.
When users submit forms, you can pass additional data through with that form to associate them together. For instance, a contact might see a field
Email address but we’ll also pass through a
Session ID so we can associate their previous pageviews and actions with this email.
Email is only enough if all your contacts and actions have email. For everything else, you’re missing data. It’s the same story for accounts if you don’t have company domain name’s. You need other identifiers like website session IDs, custom dimensions in your analytics tool, IP addresses (for B2B accounts), and so on to piece together a full profile.
Don’t ask for the same thing twice. If you can source the answer already, it’s better to not ask it (or at least just have them re-confirm it).
Try not to have different naming conventions for ALL the things! Yes, different tools and systems have different naming conventions for different things, but where it's possible, try to contain these.
You don’t have to be technical to implement this best practice. You’re setting yourself up for a nightmare with customer data management by buying and using dozens of tools. Granted, this isn’t always possible when your different teams need to be able to do different things BUT the more system you have, the more ways your data is captured, stored and has to integrate together.
In reality, your customer data will gradually scatter unless you come up with a data integration strategy to reach a single version of truth across all your tools. Your customer data integration strategy will mostly depend on what tools and techniques can pull it together.
We see most teams using one or more of these six common data integration techniques to try to bring tools all in sync and manage their customer data.
The final part of the 6-6-6 of customer data integration is the types of tools you cna use. Most companies strategy depends on one or more of these six data integration methods.
Comparing the data integration architecture of these customer data integration tools and techniques in theory and in practice at scale can help make sense of how these different customer data integration solutions work. Understanding the architecture helps with assessing how it might impact your strategy.
Remember the ETL process - extract, transform, load.
Networks like the Salesforce AppExchange, Mailchimp Integrations Directory, and HubSpot Connect give you a range of integrations you can install in a few clicks.
These have the advantage of (generally) being built and maintained by the same people who built the product. This becomes quick to setup through logging into both tools. Someone else has already done the hard work of figuring out the wiring and logic. It’s almost fire and forget with a tried and test integration.
However, it’s not a dependable data integration strategy.
Your different teams may end up needing to buy and use tools which don’t integrate with others “out-of-the-box”. Even if the integrations are available, they might not do everything you need them to.
Whilst it helps to rely on product managers to build integrations, it’s up to them (not you) to decide what integrations get built and what their capability is. For product managers, their primary concern is investing in their core product. This takes the strategy out of your hands into a second priority for someone else.
One-click native integrations also aren’t setup to be manage complex data flows across more than one tool. They tend to only extract data from one tool and load to another, without processing the data enroute (eliminating the T in ETL). This becomes complex as the number of integrations increases and you need to be able to combine different data. For instance, your sales team may need their CRM to combine data from your analytics, marketing automation, and a data enrichment provider. These don’t always work together out-of-the-box without considerable setup.
Finally, integrations rarely come with transparent logs and insight into how the data flows. Often it feels like “wait and see” with no real idea whether data is flowing or not.
In summary: Where these integrations are available, it’s great to get started. But relying only one-click native integrations aren’t a dependable solution as you grow.
Best for: seed round, early teams, ongoing data integration.
This is the simplest example of ETL. Export a CSV from one tool, sort it in a spreadsheet, then re-upload it to another tool. Since most tools offer a way to export data in a CSV format, anyone who can work spreadsheet software can manage your customer data. With some more advanced spreadsheet skills, almost anything becomes possible.
Whilst the ease and flexibility is good, manual customer data management isn’t a realistic solution as you scale. It requires a human to act like a machine, where a machine can work better.
Although spreadsheets offer a lot of flexibility, it can take a advanced knowledge to correctly combine, manipulate and process data without causing problems. Often, the more technical members of your team are better used elsewhere than acting as a human API. Moreover, as you grow and your data needs become more complex, more spreadsheets (and sheets within spreadsheets) will be needed.
Customer data isn’t static. Since it frequently updates, the manual process of updating all this data, and re-processing it can become is necessarily repetitive. This is slow, boring, and prone to errors.
Finally, it’s difficult to update and spot errors across different versions of manually exported and imported data. Unless it’s available and up-to-date in shared team folder, it doesn’t give transparency over data flows.
In summary: Manual exports and spreadsheets make data accessible and flexible (as your spreadsheet skills), but it becomes slow, manual, and repetitive as you scale to manage complexity and keep it all up-to-date.
Best for: Seed round, early teams, one-off data integration.
Pros: - Data can be very accessible - Lots of flexibility with data (using spreadsheets) - Anyone can do it
Cons: - Slow data flows - Repetitive, manual, boring tasks - Difficult to combine and transform data - Advanced spreadsheet skills quickly needed - Data is not always shared across teams - Data is not always up to date
Where native “one click” integrations aren’t available, then “if this then that” zapier-style workflow tools can become attractive - particularly if you’re already automating other data flows like document sharing, invoice alerts, social media posting, and so on.
The simplicity of “if this then that” makes it accessible to a lot of your organization - at least at the early stage.
The trouble starts as soon as you add any complexity. Customer data management something like saving your Tweets to Evernote or forwarding your invoices to Gmail. In the same way you have a history and context with your friends and family, a lead or customer has a history with your company and team. As we’ve already discussed, this data gets scattered across many tools which need to work together.
Simple automations like
Send new lead to Salesforce lose this context. What if they’re already a lead? What if they’re not a good fit? Or if they’re a target named account? Have they viewed any pages of interest on the website? Have they responded to our emails?
As you scale and grow complexity, you end up having to build multiple workflows on top of each other. This becomes difficult and repetitive to build, maintain, and extend. Without one clear view of what’s going on, it can become a complex, tangled spaghetti-like mess with a lot of tribal knowledge. Only the person who set it up really knows how it all works (and how they might fix it).
Workflow tools were built for simple workflows with one-off actions (like sending an invoice paid alert), not as high-volume data integration software in system of complex customer data (like using thousands of website sessions per minute to power the Reveal Loop). This is because they aren’t designed to store data - only process it.
Workflow tools don’t have a “database” in the same way your CRM, email tools, and product database do - there is no customer profile. This limits their ability to transform data since they can only process what you immediately share with them. This is why they struggle at scale, and why they can’t quickly, reliably, and transparently deliver the customer data your teams need into the tools they use.
In summary: “If this then that” style workflow tools are great for simple, one-off automations. Customer data often has more context and more volume which makes these simple methods unsuitable as you scale.
Best for: seed round, early teams, simple one-off data integration flows where native integrations don’t already exist.
Pros: - Quick to setup - Accessible to many in the team - No code neccessary
Cons: - Not suitable for contextual customer data - Difficult to manage and maintain multiple workflows - Can’t handle large volumes of complex data - Not transparent in what data has flowed - Not flexible in processing and transforming data
As you grow, customer data management starts to become a bigger problem. You start to codify the complexity of your business, and move to more sophisticated tools and integrate your data together.
Moreover, as you grow, you hire sales, marketing, and customer success leaders who expect customer data to be managed correctly (how many new hires have you found spend a significant amount of time on setup before they can really get to work?).
Migrating to Salesforce or another “grown up” CRM is one of those defining moves. Similarly, moving to a marketing automation tool to equip your go-to-market team with a consolidated toolset and deep integrations with your growing sales team’s CRM setup.
There’s serious health and hygiene benefits in having less (remember, the best practices of customer data integration from earlier?). Less training. Less time on boarding new team members. Less management. Fewer schemas. More out-of-the-box functionality.
And importantly - all-in-one platforms drastically help with customer data integration.
But even then, it’s half a strategy. CRMs like Salesforce, marketing automation platforms like HubSpot, and all-in-one messaging platforms like Intercom are still limited. They’re not designed to be everything to everyone. They don’t soak up all the data and context over the entire customer lifecycle.
You only need to look at their integrations pages to see - “all-in-one” is a lie!
You still need to integrate customer data across other tools and with other teams. Your data may be consolidated into each of these tools - the “source of truth” for each team in your organization - but the data they need to do their job may still be elsewhere.
What do Salesforce, HubSpot, and Intercom all have in common? They all start as an empty box. They serve as a flagship tool for your teams, but you still need a strategy for integrating data from everywhere else.
In summary: “All-in-one” platforms help consolidate data and set teams up to scale, but they never do everything across the whole customer lifecycle. Customer data still needs to be integrated across your other tools.
Best for: Growing teams who can afford the tools, and are need more "grown up" complete and connected tools to scale and grow.
Pros: - Consolidates the number of tools - Less tools to learn - Creates a "source of truth" for individual teams - Enables you to hire talent easier
Cons: - Always require more than one tool (across many teams) - Lost capability (other tools may do individual roles better) - Still need integrations. - Expensive
Sooner or later, you’ll need to get a data integration engineer (often poached from engineering) to help you tie together tools how you need them to.
This gives the most flexibility and control over your data - you can do anything you can code together.
But internal integrations rarely become the top priority for the developers and engineers in your team. Unless you’ve engineering resources on your customer-facing teams already, it’s a slog to battle for the engineering hours needed to get something setup.
Since resources are usually tight, the scope of the integrations are often diminished. The API documentation (which engineers use to understand the integrations) are usually poor, or undocumented. This makes it difficult to ensure the integration is built to do everything the sales, marketing, and customer-facing teams need them to do.
Moreover, custom data integrations are rarely fire-and-forget. Just as with other engineering projects, they need to be tested, and maintained. APIs break and update. Problems inevitably happen. You need to have the resources available to fix these as they come up to avoid being left high and dry (back to manual customer data management, or whatever skeleton setup you can make do with).
In summary: If you’ve got the technical team available to build and maintain integrations between tools, this offers tremendous flexibility. But this often comes at the cost of product engineering effort, which is political and expensive.
Best for: Technical teams that can spare the resource, ongoing integrations.
Pros: - Complete flexibility and control - Scalable once implemented
Cons: - Need someone technical to build - The internal politics of poaching someone technical - Custom integrations need to be tested and maintained - API documentation can suck - API capability can suck too
Data warehouses are designed to be a dump of all data from one place. This solves the problem of data being scattered by creating a central repository of all your data - like a landfill.
This saves you having to build integrations between every single tool just so you can centralize your data. This can then be mashed up into a unified profile for every contact. Each team can then run the queries they want for the data they need.
But this only solves half of the problem. It misses the rest of the problem - transforming that customer data into a useful format to send back to the tools where the data is needed and used. Without this, that customer data becomes “dead data”.
Usually, this involves some combination of the previous techniques - often a manual SQL query to produce a CSV file that can be re-uploaded to other tools.
Customer data has a time value - a lead just looked at the pricing page - so the ability to take action quickly is important. Customer data warehouses sync with other tools hours or days apart. They’re not in the realm of minutes, seconds, and even near-real time synchronization of other data integration techniques.
If speed isn’t an issue, there’s still the problems from combining different data types. Though you’ve got all the customer data, they need to be associated with individual people, and then formatted in a way that makes sense.
Finally, customer data warehouses are expensive to setup. As with configuring a CRM or other traditional customer data system, you need to define all your schemas and mapping. You still need to invest in an ETL developer (Extract-Transform-Load) to build this all for you. If you thought a Salesforce admin was bad…
In summary: If you really struggle integrate your tools with each other, dumping them in a data warehouse can be a solution. But you need to invest heavily in building and maintaining your data warehouse, and still need to get the data back into your tools before the data is too old to use.
Best for: Large, funded teams (Series B or later) with extraordinarily large or complex datasets, multiple datasets which are otherwise incompatible, and the engineering resources to build and maintain a customer data warehouse.
Pros - Centralize all your customer data - Totally customizable setup
Cons - Requires an ETL developer - Expensive to setup and configure - Still have to combine data sources - Doesn’t pass data back to tools quickly #DeadData
Previously, teams have struggled to breakdown their fRanKEnSTacK setups because the 6-6-6 of customer data integration is such a hard set of problems to solve.
As more and more sales, marketing, and customer-facing tools come onto the market - over 5000 marketing tools alone in 2017 - your teams will continue to adopt more and more disparate tools.
The distance between your data is only likely to grow. And so will all the headaches that come with it:
The stakes are high. In cut-throat markets, the problems of customer data integration can drive a wedge between competitors.
Customer data affects all customer facing teams. The organizations that can figure out their customer data integration strategy out can scale, automate, and personalize their customer operation across the entire customer lifecycle.
What would it mean to have a decisive and absolute advantage across multiple teams in your organization versus your competitors?
But amongst the six common solutions, there’s no “good” solution to these problems. This is why a new customer data integration tool is emerging - the customer data platform.
Customer data management platforms are purpose-built to solve the problem of customer data integration. They combine the winning elements of each of the traditional six techniques.
With one data integration hub, you can see all your data in one place, in a way that makes sense. Simply put, it’s one customer profile to power them all.
Customer data platforms tend to be schemaless, so you don’t have to invest weeks in setting it up just right. You can simply sync that data you have in the formats they are, and the customer data platform will make sense of it for you. No need for the expensive Salesforce admins and ETL developers.
Whilst customer data platforms are designed to be flexible, they’re also easy to use. Non-technical sales, marketing, and operations teams can use them to setup advanced data flows without needing to write code. For even more advanced setups, the ability to write code unlocks even more flexibility.
Since it’s the platform’s job to maintain the integrations for many customers, they come tried and tested ready to use. If something breaks or doesn’t work, the platforms support and engineering team can help or take a look - no begging your own developers.
The downsides are small. A dedicated platform is of most use to companies which have outgrown their first setups with native one-click integrations, manually exporting and importing, and using simple if-this-then-that style workflows.
It’s also useful for those who have already invested in their customer data integration strategy with upgrading the capabilities of core “all-in-one” CRMs and platforms, replacing custom coded, and simplifying customer data warehouses. Customer data platforms compare favourably to the cost of buying and maintaining these these solutions too.
By comparing how different data integration techniques can handle different example scenarios.
Sending different onboarding emails by job title let’s you send far more relevant messages. To do this, you need to have combine a segment of new users with segments of job titles.
Let’s imagine we’re doing this in an email tool and a data enrichment provider.
✅ One-click native integrations If the integrations exist between your data enrichment provider with your email tool, this is perfectly valid. The new signup could be captured in the email tool (for instance with a form handler), and the segments then created locally.
🚫 Manual export and import Manual work struggles if onboarding emails need to be sent quickly - like shortly after signup. Though it’s possible to pull all the data into a spreadsheet, combine them, and then re-upload them, it’s not ideal.
✅ Workflow tools For one-off users, sending new signups to an enrichment tool, and then sending them to your email tool makes sense. This may be slow when processing a batch of users.
✅ All-in-one platform This would require another integration to bring the third party data in. Similar to the one-click native integration.
✅ Custom coded integration This may not be necessary, unless the integrations between your email tool and data provider simply don’t exist.
🚫 Customer data warehouse As with the manual export and import, this isn’t suitable if the emails need to be sent right away.
✅ Customer data platform Simply connect your email and enrichment tools, and create the segments to send new users to be enriched, and segments of new users by job title. These will be recreated inside your email tool, ready to trigger a drip email on boarding series.
With a free trial or low entry-level price, we might qualify leads for sales with product usage data from your analytics and SQL database, together with third party data enrichment. These qualified leads then need to be sent to your CRM. Learn more about product qualified leads in the Data-Driven Sales book.
🚫 One-click native integrations Since this combines multiple tools with a lot of custom data (events from product usage), this isn’t suitable.
✅ Manual export and import If there’s no hurry to take action (which may miss the opportunity), the data could be pulled from your SQL database, analytics tool and combined into a spreadsheet with data enrichment. Some matching by IDs and emails would be needed, but this could then result in a list of product qualified leads. This would be tedious to do repetitively, but good enough for a proof of concept.
🚫 Workflow tools The shear volume of product usage data (actions from any user) would overwhelm a workflow tool. It would also struggle with combining the data from multiple sources.
🚫 All-in-one platform The CRM is the key piece of the product qualified lead recipe, but the key data all sits in analytics and product databases. There’s no advantage to having an all-in-one tool over another setup.
✅ Custom coded integration This is an option given the complexity involved. It involves wiring together multiple tools and computing the data to produce a list of product qualified leads.
✅ Customer data warehouse Though the customer data warehouse might have all the data, it still has to be connected together, and pulled into a CRM. A complex SQL query would need to be run regularly - but it would miss the ability to take action in real time.
Like with a manual export and import, it’s good enough for a proof of concept. The added benefit of having a written a SQL query, it could be run again when the data is updated.
✅ Customer data platform With connections to the analytics tools, data enrichment providers, and CRM, with complete, up-to-date profile on each, this is as simple as creating a segment with your definition of a product qualified lead.
This is an advanced B2B account-based marketing technique to use Reverse IP lookup against a companies database to identify website visitors’ companies, and then engage them across multiple channels like email, ads, live chat, personalised websites and more.
You can read more about how ABM teams are landing over $10,000 a day with the Reveal Loop here.
🚫 One-click native integrations No integrations exist.
🚫 Manual export and import The Reveal Loop evaluates every website visitor for Reverse IP lookup. This becomes the key for the prospecting lookup, which becomes the key for enrichment. Even with an outsourced army of people pulling and crunching data, this wouldn’t be possible.
🚫 Workflow tools Without a database of each account and person’s profile, workflow tools can not manage all this complexity reliably when triggered after every website visitor.
🚫 All-in-one platform No all-in-one platform offers this across multiple channels. Integrations are still needed.
✅ Custom coded integration This is hard, but doable. 4-5 APIs to manage the accounts need to be tied together and processed along with every single marketing and messaging platform you need to engage people.
🚫 Customer data warehouse Since this involves website traffic, and the ability to respond fast (ideally, in real time), customer data warehouses are hours and days behind the time when action was needed to be taken.
✅ Customer data platform With all the tools connected and fast sync enabled, the Reveal Loop can be orchestrated with ease. The known data on the customer profile can trigger the next lookup, which is added to the customer profile, triggering the next action.
You need to clearly identify all the important data to each team, and which tools they sit in. Then you need to work out how best to join the dots. Start with the fRanKEnSTacK assessment.
It’s clear from the examples, that different data integration tools and techniques have different advantages and disadvantages. Let the horse fit the course!
Whatever method you choose, make sure it’s right for your customer data needs now and in the near future.
The objective is still the same - solve the 6-6-6 of customer data management and fear the fRanKEnSTacK!