🚀 We're hiring, check out our careers page to join our team!

Display data ingestion errors and input record in logs in Dashboard

Hey there,

Today we're announcing a major improvement: We've leveled up our logging and the result is that we've considerably improved the clarity in how we represent the traffic flows into your Hull account.

Previously, each connector was responsible for logging the incoming data individually. This worked, but it lacked reliability and clarity in some specific but critical situations:

  • It was not apparent if the platform ended up rejecting the payload due to invalid data
  • It didn't show you which user the log was actually tied to
  • It didn't show logs for data ingested through our Bulk import feature (which some connectors such as the SQL connector are using)
  • It sometimes lacked consistency because it was manually implemented.
  • Custom and forked connectors, which don't have access to the logging infrastructure couldn't expose those logs.
  • Inconsistent formats and listing of touched attributes

Today we're happy to announce that we're now generating reliable ingestion logging data from the platform's ingestion steps for all connectors and all data sources.

Here are some of the benefits of this new architecture:

Transparent support for logs even from forked connectors

For those of you who are using custom forked connectors, this is big! You'll now start seeing ingestion logs for your connectors and you will be able to see what was sent and how it was ingested.

Logging of import job ingestion logs (this one is big for SQL connector users)

The SQL connector and others who are using our bulk import API weren't generating per-user and per-account logs, which made that data invisible in the logs. Now, they're generating the same data as the others.

SQL_incoming_records.png

sql_importer_Table_details.png

More details about the ingestion decisions made by the platform.

The Hull platform has validation rules to ensure proper data is ingested. For instance, we filter domains at the account level so that we don't create accounts based off of "freemail" domains (think gmail.com). Sometimes, data is simply invalid and we're forced to drop the payload entirely. If such an error occurs now, you'll be able to see it and understand why.

error_message.png

Uniform support for captured attributes

We're now showing all the attributes that were received. This lets you search for them and see exactly what data was received and which user it was resolved to, making the process of analyzing data flow a much simpler process.

captured_attributes_incomingrecords.png

Uniform support for attribute changes

Within the logs, we're now showing the full list of attribute change made to a user. Not only do you see which data we captured, but you'll also see which attributes were modified, and the changes it made to the resolved profile. This makes it easy to identify loops and other data flow issues.

Changes_incoming_records (1).png

← Back to Changes