Compute


Visible, Hidden, Tracked traits (Attribute Changes)

Traits visibility

Traits in the Schema can be defined as invisible, which means that they will be actively ignored in the ingestion and indexing steps.

Attributes changes events

Changes on Traits and attributes will by default generate Attributes changed events, with :

  • app_id capturing the Connectors that introduced that change
  • props listing the attributed that actually changed, their previous and new values

example:

{
  "indexed_at": "2017-08-30T18:26:06+00:00",
  "created_at": "2017-08-30T18:26:06Z",
  "event": "Attributes changed",
  "app_id": "58c78a0dba9b600df6001acd",
  "app_name": "Datanyze Technologies Processor",
  "source": "Processor",
  "session_id": null,
  "type": "attributes_changed",
  "props": [
    {
      "field_name": "to/matching_technologies",
      "num_value": 2,
      "text_value": "2"
    },
    {
      "field_name": "from/matching_technologies",
      "num_value": 1,
      "text_value": "1"
    }
  ],
  "context": {
    "days_since_signup": 10
  }
}

Attributes changed tracking can be disabled on Traits in the Traits Schema configuration. Those events are indexed, searchable and usable for Segmentation, but they are not propagated in outgoing notifications.

Segmentation

Users and Accounts Segments are a way to group users who share a common set of properties and/or behaviours.

Segmentation is used for multiple reasons on the platform, the principal use cases being :

  • Filtering or whitelisting Users and Accounts to be processed by connectors
  • Building custom audiences to be replicated in 3rd party services (email marketing or advertising tools for example)
  • Saving often used queries used in the dashboard

Defining segments - Querying Users and Accounts traits

Segments are built via a dedicated query builder UI on our Dashboard. The query builder generates a high level representation of the query which is then translated into an Elasticsearch query. Both representations of the Segment query are persisted together in the UserSegment document.

Anatomy of a Predicate

  • predicates array of predicates that are combined to build the query
  • type defines how the predicates listed in the predicates array must be combined. all corresponds to a AND and any corresponds to a OR

We support 3 types of Predicates : properties, events and nested groups

Predicate on properties

{ “type”: “all”, “predicates”: [{ “property”: { “key”: “first_name” }, “operator”: “equals”, “options”: { “value”: [ “Romain” ] } }] }

supported operators by attribute type

string:

  • is_set, is_not_set : Tests the presence of a value for a given trait
  • equals, does_not_equal : Strict equality of the whole string, case sensitive
  • contains, does_not_contain : Partial match within the string
  • starts_with, does_not_start_with : Similar to contains but only tests the start of the whole value, without tokenizing it

The contains operator is the richest and potentially the most tricky to use because it’s not a full substring matching. A few limitations to keep in mind :

  • search is case and diacritics insensitive
  • the search string is matched against parts of the original string (tokens)

example searches :

value: "Hello Wørld"

=> contains "Hello" : true
=> contains "hello" : true
=> contains "world" : true
=> contains "ello" : false
=> contains "ello wo" : false

value: "hello@example.com"

=> contains "hello" : true
=> contains "hello@" : true
=> contains "hello@example" : true
=> contains "example" : true
=> contains "example.com" : true
=> contains "lo@ex" : false

value: https://example.com/hello

=> contains "https://example.com/hello" true
=> contains "example.com" true
=> contains "hell" true
=> contains "/hello" false


numeric:

  • is_set, is_not_set, equals, does_not_equal : Similar to string operators
  • is_greater_than, is_less_than : Strict inequality comparison with a given number

boolean:

  • is_true, is_false
  • is_not_set

geo_point:

  • is_within, is_not_within : Tests the inclusion of a geo-point within given geo boundaries

date:

  • is_set, is_not_set
  • is_on, is_not_on : Matches on a specific given date
  • is_before, is_after : Strict absolute inequality comparison with a given date
  • is_exactly, is_less_than, is_more_than, is_between : Relative date comparison

Defining segments - Querying Users Events

{ “type”: “all”, “predicates”: [{ “property”: { “key”: “event” }, “operator”: “at_least_once”, “options”: { “eventName”: “Completed Order”, “propsFilters”: [{ “section”: “property”, “key”: “total”, “value”: 100, “operator”: “gt” }], “timeframe”: “less_than”, “days”: 30 } } } }

supported operators count the number of occurrences of the Event within a given timeframe

on events occurences : zero_times, at_least_once, is_at_least, is_at_most, equals, does_not_equal on timeframe : on, exactly, not_on, less_than, more_than, before, after, between_days, between on event properties: : contains, equals, gt, lt, after, before, is_true, is_false, is_not_set, is_set

Defining segments - Combining predicates

Nested Predicates group

{ “type”: “all”, “predicates”: [{ “type”: “any”, “predicates”: [{ “property”: { “key”: “email” }, “operator”: “is_set” }, { “property”: { “key”: “external_id” }, “operator”: “starts_with”, “options”: { “value”: [“123”] } }] }] }

Predicate groups allow Predicates to be combined freely using the types any or all which correspond to logical or and and.

Lifecycle of a Segment

When a Segment is created a users_segment:update or accounts_segment:update notification is sent to all listening connectors via the commands channel and the platform launches a background job that re-evaluates the Segment on all the Entities of the organization.

When a Segment is destroyed, a users_segment:delete / accounts_segment:delete notification is sent to listening connectors.

Those lifecycle notifications sent to connectors are primarily used to maintain segment in sync with external services like MailChimp lists or Facebook ad audiences.

Re-evaluating segments when an Entity is touched

Each time an Entity is marked as dirty in the ingestion step, the list of segments it belongs to is re-evaluated.

Re-evaluating segments as time passes

Segments are only evaluated on Entities when a change is detected on that Entity ; a change being either an Attribute or Trait change or a new Event tracked.

Some Segments can contain relative time based predicates like “Users who signed up less that 7 days ago”. That means that after 7 days, those users need “leave” that segment even if no change is captured for them.

Hull detects those Segment that have relative time predicates and treats them automatically as live segments that need to be re-evaluated on a regular basis. (Currently live segments are refreshed on a rolling basis every 6 hours.)

Entering and Leaving segments

When an Entity enters or leaves a Segment, an Event is captured in his timeline, recording the actual changes.

Example “Segments changed” event:

{
  "indexed_at": "2018-03-08T14:47:59+00:00",
  "created_at": "2018-03-08T14:47:56Z",
  "event": "Segments changed",
  "source": "hull",
  "type": "segment",
  "props": [{
    "field_name": "entered",
    "text_value": [ "Never viewed dashboard or had demo" ]
  }, {
    "field_name": "left",
    "text_value": [ "panda test", "Health Test" ]
  }, {
    "field_name": "entered_ids",
    "text_value": [ "5aa13e004a6ef52f6000007d" ]
  }, {
    "field_name": "left_ids",
    "text_value": [ "5a302c8fa79337b537002a17", "5a720d1d5cb250f6d5000070" ]
  }],
  "context": {
    "days_since_signup": 457
  }
}

Those internal Events are then queryable as any other Event and can be used in the definition of new Segments. They are very useful for capturing information that would otherwise be lost.

Building UserReport and AccountReport Notifications

UsersReport and AccountReport Notifications are used to propagate changes to the outside world. Connectors subscribe to those notifications and use this information to synchronise, notify, transform or to apply whatever side effect they are meant to apply.

Format of a User update Notification

User update Notifications are built with the following sections :

  • user contains the latest known version of the user’s UserReport (without embedded the AccountReport)
  • account contains, if the User is linked to an Account, the latest known version of that AccountReport
  • segments contains the list of UsersSegments the User belongs to
  • account_segments contains the list of AccountsSegments the Account belongs to
  • changes captures a diff of the information that changed between the current and previous notification built for that User (description of the changes below)
  • events contains the list of Events associated to the user since the last Notification

Example payload

{
  "user": {
    "id": "5a7b32a5d57fdbde8f000007",    
    "accepts_marketing": false,
    "created_at": "2018-02-07T17:08:53Z",
    "indexed_at": "2018-02-12T10:17:24+01:00",    
    "domain": "hull.io",
    "email": "stephane@hull.io",
    "external_id": "1",
    "has_password": false,
    "is_approved": false,
    "segment_ids": ["5a815a3fd57fdbb78a000004"],
    "traits_hello": "world"
  },
  "segments": [{
    "id": "5a815a3fd57fdbb78a000004",
    "name": "Users with Email",
    "type": "users_segment",
    "created_at": "2018-02-12T09:11:42Z",
    "updated_at": "2018-02-12T09:16:33Z"
  }],
  "account": {
    "id": "5a7b32a5d57fdbde8f000005",    
    "created_at": "2018-02-07T17:08:53Z",
    "updated_at": "2018-02-07T17:08:53Z",
    "external_id": "account-1",
    "domain" : "hull.io",
    "name" : "Hull inc"
  },
  "account_segments": [{
    "id": "5a815a3fd56fceb79a000001",
    "name": "Accounts with Domain",
    "type": "accounts_segment",
    "created_at": "2018-02-12T09:11:42Z",
    "updated_at": "2018-02-12T09:16:33Z"
  }],
  "events": [
    {
      "properties": { "foo" : "bar" },
      "event_id": "5ad078db8463ba4a550002dd",
      "user_id": "561fba41450f34efa5000019",
      "event_source": "track",
      "app_name": "Shopify",
      "event": "page",
      "event_type": "track",
      "context": {
        "browser": {
          "name": "Safari",
          "version": "11.0.3",
          "major": 11
        },
        "campaign": {
          "name": null,
          "source": null,
          "medium": null,
          "term": null,
          "content": null
        },
        "device": {
          "name": "Other"
        },
        "ip": "10.52.8.14",
        "os": {
          "name": "Mac OS X",
          "version": "10.13.3"
        },
        "page": {
          "url": "https://example.com/account",
          "host": "example.com",
          "path": "/account"
        },
        "referrer": {
          "url": "https://example.com/",
          "host": "example.com",
          "path": "/",
          "campaign": {
            "name": null,
            "source": null,
            "medium": null,
            "term": null,
            "content": null
          }
        },
        "useragent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/604.5.6 (KHTML, like Gecko) Version/11.0.3 Safari/604.5.6"
      },
      "anonymous_id": "1519364950-b671ca2b-e54a-4700-a39f-e60f280a89a8",
      "created_at": "2018-04-13 09:30:52 UTC",
      "session_id": "1523611741-5f7a0160-fe87-4577-956a-ae3834ccc18e",
      "app_id": "561fb665450f34b1cf00000f"
    }
  ],
  "changes": {
    "user": {
      "accepts_marketing": [true, false],
      "email": [null, "stephane@hull.io"],
      "is_approved": [null, false],
      "traits_hello": [null, "world"]
    },
    "segments": {
      "left" : [{
        "id": "5a815a3fd57fdbb78a000003",
        "name": "Anonymous Users",
        "type": "users_segment",
        "created_at": "2018-02-12T09:11:42Z",
        "updated_at": "2018-02-12T09:16:33Z"
      }],
      "entered": [{
        "id": "5a815a3fd57fdbb78a000004",
        "name": "Users with Email",
        "type": "users_segment",
        "created_at": "2018-02-12T09:11:42Z",
        "updated_at": "2018-02-12T09:16:33Z"
      }]
    },
    "account": {
      "name": ["Hull", "Hull inc"],
      "updated_at": ["2018-01-10T09:00:09Z", "2018-02-12T09:22:37Z"]
    },
    "account_segments": {
      "left": [{
        "id": "5a815a3fd56fceb78a000009",
        "name": "Recently updated accounts",
        "type": "accounts_segment",
        "created_at": "2018-02-12T09:11:42Z",
        "updated_at": "2018-02-12T09:16:33Z"      
      }],
      "entered": [{
        "id": "5a815a3fd56fceb79a000001",
        "name": "Accounts with Domain",
        "type": "accounts_segment",
        "created_at": "2018-02-12T09:11:42Z",
        "updated_at": "2018-02-12T09:16:33Z"      
      }]
    },
    "is_new": false
  }
}

Diffing and propagating changes

The changes section of the notification contains the following sections :

  • user which contains the attributes that changed on the UserReport since the last Notification, capturing the key, previous and current values
  • account is similar to user but capturing changes on the AccountReport since last notification
  • segments contains the list of segments the User entered and left
  • account_segments contains the list of segments the Account entered and left
  • is_new is a boolean which is set to true if the notification is the first one that we receive for the user

The events section contains the list of Events captured for the User since the last Notification was sent.

Internal events

Attributes changed and Segments changed are not included in the outgoing notification payloads. They are considered as “Internal” events. The changes object included on the notifications is the prefered way to keep track and propagate changes to the outside world.

User, Account and Event data lifecycle

UserReport Notifications are triggered and sent when :

  • A User attributes changes in the Ingestion phase
  • A new User Event is Ingested
  • A user enters or leaves a segment that has a relative time predicate (see dedicated section on Segments).