Cloud Orchestration: Coordinate Your Tools, Teams & Data
14th Sep 2017This is the fourth chapter in our customer data academy - subscribe above to get the full series.
Previously, we’ve discussed the customer stack model for simply explaining how customer data should form a loop between messaging people, tracking reactions, recording to a profile, deciding actions, to sending more messages.
In the second chapter, we discussed how to “cleanse” your raw customer data, ready to be immediately useful in all of your tools with the combine-connect-compute framework. You need a “brain” for your customer data.
In the third chapter, we discussed how to get the data you need into the tools your teams are using including best practices and the seven common data integration methods.
In the fourth chapter, we’re going to explore how we can coordinate and use all this data and tool together in a system - “orchestration”.
What is cloud orchestration?
Orchestration is a metaphor for coordinating many individual things at once.
Like a composer and conductor would orchestrate music to create something bigger and better than the individual musicians, we want to orchestrate our tools, teams, and data to do something bigger and better than they would on their own.
Orchestration can add value, capability, and opportunity with your existing resources. As irritating as the “1 + 1 = 3” analogy can be, its 100% true.
You need to understand EVERY part of the system you’re trying to orchestrate.
No one team is responsible for your entire customer journey - its split up across marketing, sales, support, success and more. You need to understand each of these team’s role and responsibility, what their goals are, and how they work to achieve their goals.
Within each team, you need to understand every tool and their capabilities, which team(s) own them, and what they’re used for.
Third, you need to understand the customer data across the entire customer lifecycle. How does your collection of different tools and databases profile your customers?
Once you understand each of the individual parts, players, and “ingredients”, then you can think about how they can work together.
Mindset change: Think of the system, not individual parts
Your teams do NOT work alone.
Your tools do NOT work alone.
Your data does NOT work alone.
With orchestration, you need to think of the whole “system” instead of each individual part. But, this defies your current thinking and internal politics.
Just try making a change to your sales CRM setup - I dare ya!
Your teams are only setup to help themselves. Their tools. The job with the customer. Their view of the world.
In the second chapter of our customer data academy, we talked about how you need to combine your data first.
Customer data tends to fall into six types of tools:
- Web and product analytics to capture what people have done on your website and in your product.
- Backend database to capture all profile data, form fills, actions, and activity ever with each person and account.
- CRMs for sales, support, customer success, and others to have a record of each person, account, and job needed to be done. CRMs are usually the key data point for 1:1 communication.
- Messaging tools like email, live chat, and marketing automation. These send and track messages to larger audiences
- Enrichment providers which append data to your customer profiles in other tools
- Ad networks which allow you to display ads to targeted audiences.
Even with these six types of tools, it’s still difficult to understand how they’re supposed to fit together.
For instance, you may have multiple CRMs (sales may use one CRM tool, support may use another) which need data enrichment and analytics data, as well as syncing with several messaging tools.
This is why we started the customer data academy with the Customer Stack - a customer data model to show how data should flow between sending messages, tracking reactions, recording to a profile, deciding actions, to sending more messages. You need to understand your team's unique setup in this context.
Once you understand how each piece works, you can start to think about programming. Yes, programming!
Mindset change: Orchestration is programming!
At the highest level, programming is the process of creating sets of instructions to tell computers how to perform specific tasks. It is process design.
Software developers design processes with code.
But, as your team defines the logic that updates customer data, triggers workflows, syncing, and automations - you’re doing the same thing. If this, then that.
Instead of writing code, you’re designing process within the tools you’re using.
Marketing automation is process design…
Sales automation is process design…
Cloud orchestration is process design…
By thinking of your system of teams, tools, and data as elements of a program, you can start to tackle the problem of the whole system - not just individual parts.
It means our “program” is actually a complex set of rules, workflows, and actions scattered across several different tools.
It also means we can learn from the principles of programming.
Just because it is often point-and-click and doesn’t look like a command line does NOT mean you can’t learn from a developer.
Developers have had all these problems around orchestrating tools and data for forever. The other mindset changes come from programming principles.
Once you can think of data and process design like programming, you’ll have the mindset for customer data orchestration.
Let’s dive in…
Mindset change: Keep it simple, stupid
KISS came from the Lockheed Skunk Works in the 1960’s, but has been wholeheartedly embraced by the programming community (amongst many others).
Simplicity means less. Less to build. Less to test. Less can go wrong.
But simplicity isn’t always sexy. There’s a buzz from showing off clever, complex workflows with complex branching logic. Teams can get seduced by overly elaborate “orchestration” and workflow design.
Modern tools don’t make this easy. Visual marketing automation tools reward complexity - it looks impressive! ”Ah, what if I just add another clever little branch here? It'll be easy, right?”
WRONG.
Break it up. Break it down. Less is more.
Build advanced things from blocks of simple things.
Because the real world will happen. Stuff breaks!
Complexity is hard to build, hard to test, and hard to maintain. When your entire workflow falls over and you’re wading through 14 steps of 4 branches across 3 tools, you’ll wish you’d built something simpler.
For developers, the equivalent term is “spaghetti code” - complex, tangled mess of logic from early complexity, multiple contributors adding and fixing it, and time.
Our core values here at Hull include seeking elegant solutions. Complex is not clever! Keep it simple, stupid!
Mindset change: Shortcuts strangle scaling startups
Not everything looks like a shortcut…
… until a few months down the line.
This problem plagues developer teams. They call it “technical debt”.
The “debt” is the cost of having to rework something later so a quicker, easier solution can be implemented sooner.
But there’s also a “debt interest” element, in that it’s often harder to change later on.
The pressure for results (particularly with customer-facing teams who roll up to revenue!) drives shortcuts like:
- Dumping data in different tools to “sort it out later”
- Building and tweaking live data workflows without testing
- Not connecting each tool
- Not documenting how things work
- Cloning processes
- Pushing a huge set of data to Salesforce (or whatever your key tools are) without checking teams, ops, and documentation
But the combination of lack of testing, documentation, and knowledge how the whole system works (see the first mindset change) often leads to failure down the line. Your half-arsed analytics and botched CRM setup becomes more and more painful and hard to fix the longer you leave it.
Since it ties so closely to programming, Jeff Atwood’s essay on paying down your technical debt is just as relevant to customer operations teams as it is to developers.
Mindset change: Don’t repeat yourself (DRY)
In programming, it’s considered a very bad practice to write the same thing again. Repetitive code is a root cause of spaghetti code.
The Pragmatic Programmer states that:
Every piece of knowledge must have a single, unambiguous, authoritative representation within a system.
Instead of repeating the same thing, best practice says you abstract that particular function and write it once, then use the result of it elsewhere.
It’s easy to remember to be DRY - Don’t repeat yourself.
As is the opposite: WET.
“Write everything twice”.
“We enjoy typing”.
“Waste everyone’s time”.
🙃
Your customer operation is soaking WET.
It’s shocking to see how much duplication of functionality there is with tools used by growing startups. For instance:
- Multiple messaging tools that can all send email
- Duplication in analytics and product tracking
- Multiple different scores for leads, users, and accounts
- Duplicate contacts in different tools
- Email lists ≠ ad audiences ≠ live chat triggers ≠ website audiences
Chapter two of the Customer Data Academy discussed the importance of combining, connecting, and computing all your customer data in one place.
This is the DRY principle in action. Centralize your customer data and build one “master” profile for each person and account. Then you can abstract all your computation from all your different tools - cleansing, enriching, scoring, segmenting, and so on - and compute them in one place (where all your data is stored and most up-to-date anyway).
Mindset change: Structure your data so it’s easy to use
In chapter two, we also discussed the inflexibility of tools to digest unstructured and unformatted structured data.
Programmers can not tolerate a system which fails to digest the data they have - but this isn’t the case with most go-to-market teams.
Often, it’s sales who holds the biggest sway with customer data. Sales outnumber many other customer-facing teams (support, success, marketing), and live-and-die by the revenue they pull in.
So one does not simply screw with your sales team’s CRM.
But that doesn’t mean your sales CRM becomes the heart of your customer data management.
Take a CRM like Salesforce. It is unsuitable as a customer data platform because:
- Salesforce can only take highly structured data
- Salesforce is limited in its ability to cleanse data
- Salesforce can still have duplicate contacts, accounts, and more
- Salesforce isn’t designed to have multiple people configuring data flows (without total chaos)
- Salesforce charges significant fees per seat, so there’s a strong negative incentive to not include everyone
- Salesforce charges significant fees for API calls to send and receive data from other tools
- Salesforce charges significant fees for storing data
- Salesforce makes it slow and tedious to define logic, segments, workflows, and so on.
- Salesforce is NOT the key tool for everyone - you need to integrate other tools (and get charged).
None of this sets you up for orchestrating customer data with ease. You need a more flexible, open, cheaper, tolerant customer data platform.
In other words…
Do NOT “centralize” your data in your CRM or BI tools.
CRM tools like Salesforce are part of the customer data system - an important part of your customer data system. But it is NOT a whole system itself.
Data warehouses and “BI” tools are the opposite of a CRM. They can capture everything from everywhere (once you’ve set it all up). There’s a world of benefits to centralizing, copying, and analyzing data in one place.
But their data and insights are NOT immediately useful to all your teams in the tools they want to use. The data they capture is far from accessible and useful to all your teams in the tools they want to use. They’re not actionable of “reactive”. They cannot drive automation (and orchestration) across all your tools.
Remember, you’re building a system.
In the context of the whole system, data warehousing is simply taking an infrequent copy of your data every now and then. This doesn’t help you with automating rules to coordinate different tools, teams, and data in real-time.
Once you’ve realized you’re building a system of truth (not a source of truth - see the customer stack model), then you can move on to understanding orchestration.
Designing your orchestration layer
aka. How to whip up hyper-engaging, personalized communication at every stage of the customer journey
At the beginning of the Customer Data Academy, we discussed the four parts of the customer stack and how it forms a closed loop:
- Sending messages
- Tracking reactions
- Recording to profiles
- Deciding actions
… to send more messages.
In the second chapter, we looked at how we could cleanse customer data to make it immediately useful in all your tools with the combine-connect-cleanse framework. With this, we added a small extra “compute” loop below each profile.
If we map these out, we see two layers to customer data orchestration:
The engagement layer: This is where we send messages and meet with customers, automatically and in-person.
The intelligence layer: This is how we make sure we can react and prepare the most useful data to trigger personalized messaging and meetings.
With the principles we’ve learned, we want to keep our orchestration simple. Instead of your logic, data, tools, and teams scattered across your system, you have it centralized into one system. (Less spaghetti, more lasagna).
From our five-part personalization framework, we know our challenge is to send the right message to the right person.
It’s the job of the intelligence layer to inform WHO to target and WHAT to say. With one “master” profile built with data from the latest data in all our tools, we can cleanse and compute our segments, scores centrally and sync them out to the engagement layer.
Your messaging tools can then trigger personalized messages according to the logic defined by your intelligence layer.
To understand this in more detail, let’s look at a few examples.
Orchestration example #1: Product qualified leads
Product qualified leads are common in SaaS startups with a trial-based business model.
Instead of cherry picking leads, sales gets a feed of leads who are both a good fit and are clearly getting value from the product already.
This involves orchestrating data from product usage, data enrichment providers, and your CRM.
Here's how it works:
Orchestration example #2: The Reveal Loop
The Reveal Loop is a technique to retarget an entire company account based on a(n anonymous) website visit.
Using reverse IP lookup against a database, it identifies a company, then prospects for the best fit people from that company, and engages them all with your account-based marketing program.
This involves orchestrating data from your web analytics tools, data enrichment provider, backend database or CRM, and your engagement tools for your ABM program.
Here's how it works:
Orchestration example #3: Lifecycle marketing
Marketing never stops. Particularly after Series B, lifecycle marketing becomes a key driver of growth.
Orchestration is key to lifecycle marketing. Managing the customer journey when multiple teams and tools are engaged is difficult.
There’s a lot of sources of updates to a customer profile, and a lot of potential reasons for someone to message.
If you have an account manager on boarding them, support rep resolving an issue, or a sales person managing an upsell, they’ll more than likely want to be able to control (ergo: throttle) messaging from marketing and others during that too.
Your entire system of customer intelligence and engagement cannot work together without a central hub - a “brain”.
Customer data orchestration tools
In chapter three of the customer data guide, we explored the seven different options for integrating your customer data. That way, you can get the data your teams need into the tools they need to use.
Customer data orchestration = computation + integration.
Since we need to include the integration, to exchange data between our intelligence and engagement layers, we need to revisit the seven options for customer data integration.
“One click” native integrations
Integrations are designed to sync data, but unless there’s a central computation engine that can cleanse data, build and update segments, then push all that to other tools in real-time, it cannot be used for orchestration.
Principles violated:
- Think of the system, not individual parts
- Orchestration is programming!
Manual export and import via a spreadsheet
The first problem with manually working with data is it has to work in batches. Orchestration is about being able to react in near-realtime across all your tools, teams, and channels. Then, the challenge of mapping, cleansing, and formatting data on an ongoing basis makes this unsuitable for orchestration.
Manual data management doesn’t give you the chance to “program” and leverage the ability of tools
Principles violated:
- Orchestration is programming!
- Shortcuts strangle scaling startups!
Workflow tools and Zapier alternatives
At first glance, workflow tools like Zapier seem like the perfect answer. Send data from one place to the next.
But each operation and automation is separate and siloed. You have to repeat yourself and your logic. You have to depend on many other automations and tools to work. As you scale, all your logic gets built into a spaghetti of “if this then that” operations. It’s difficult to add, test, and maintain.
Principles violated:
- Don’t repeat yourself (DRY)
- Keep it simple, stupid!
All-in-one CRMs and marketing automation platforms
As discussed in the data integration guide, no single tool covers the entire customer journey. You have to integrate with other tools and teams. Orchestration is limited to the channels that the tool can reach itself.
Principles violated:
- Think of the system, not individual parts
- Structure your data so it’s easy to use!
Data integration engineers and custom code
Wiring up different APIs yourself to integrate data is one thing. But writing from scratch means every new integration and orchestration idea needs to built (pulling developer resources from other priorities in the company). Another database, another set of integrations, another workflow to build, test, and maintain.
Starting out, this can make sense. “Can you help me wire X and Y together?”. But just as with workflow tools, this quickly becomes a mess. Once you’ve built out more than a few different tools, customer data orchestration becomes a big gnarly software project in and of itself.
Principles violated:
- Keep it simple, stupid!
- Shortcuts strangle scaling startups!
Customer data warehouse
We’ve discussed previously how dumping data in a warehouse doesn’t aide orchestration. Centralizing all your data in one place within a system does NOT help the whole system coordinate and “react” to messages. Another integration method is needed.
Principles violated:
- Think of the system, not individual parts
- Structure your data so it’s easy to use!
Customer data platforms
Customer data platforms can manage both computation of customer profile data, and integration between all your tools in a network. This enables you to orchestrate your entire system of team, tools, and data across the entire customer journey.
By housing your computation and logic centrally, you avoid the FUBAR “spaghetti code” problem. One place to cleanse, enrich, score, segment, and transform data - then sync out to your other tools in real-time. The data your teams need in the tools they want to use.
Instead of exponential complexity, where each additional thing repeats itself and might break something else, there’s one central “function” for each thing which works predictably.
Principles met:
- Think of the system, not individual parts
- Orchestration is programming!
- Keep it simple, stupid!
- Shortcuts strangle scaling startups
- Don’t repeat yourself (DRY)
- Structure your data so it’s easy to use!
You can learn more how a customer data platform like Hull works here.
Next up: the biggest customer data challenge of them all
So far, we’ve talked about a model for our customer data, how to cleanse it into an immediately useful form, how to sync data to all your tools, then orchestrate your customer journey.
But there’s one glaring omission from our customer data academy.
Easily, the biggest challenge of them all.
Even if you nail everything else, this will be your downfall.
Why?

Prev 'Ed of Growth at Hull, working on all things content, acquisition & conversion. Conference speaker, flight hacker, prev. employee #1 at inbound.org (acq. HubSpot). Now at Behind The Growth
If you've questions or ideas, I'd love to geek out together on Twitter or LinkedIn. 👇