May 2018 Release Notes

During the past few months we have been busy adding and improving features you requested in our product, as well as many features that are designed to help our customers meet the requirements of the GDPR.

In this product update email you will find updates on:

New features:

  • Models running custom scripts
  • GDPR compliance support features
  • User Profile

Feature Improvements:

  • Query API limitation increase
  • User create date corrections
  • Facebook integration account selection

SDK Update

  • iOS 2.3

Models

In addition to the R and Python Reports we added in our latest update, Cooladata now supports running data manipulation scripts in R or Python, and saving the data frames  to tables in your project. The new Models feature opens up a new dimension of exploration and analysis of your data, encompassing all capabilities of these languages: you can run statistical models, data processing procedures, or even write your own custom integration fetching data from APIs and storing it in Cooladata.

Our customers are already using this feature to cluster users using R K-Means, schedule custom integrations to fetch data from affiliates and service providers such as Brandwatch or Rakuten as well as predict player LTV or likelihood to convert.

Models are similar to Aggregation Tables in the way they are built. They can run on any data source in you project, including events, tables, and linked data sources and can be scheduled to run automatically on a set frequency or as part of a Job.
The model generates results that are stored in a table, which can then be queried from anywhere, including reports, publications and Query API.

GDPR Compliance Support

A few weeks ago, we have shared our ongoing efforts to help our customers meet the requirements of the GDPR. We are happy to announce that we have finished the development of the major key features of the GDPR and they are now available as BETA features:

Opting out users

Cooladata enables you to opt-out a user in order to stop tracking its future activities. Opting out a user will be done either by the SDK or a rest API request to Cooladata, by sending a “cooladata_opt_out” event with the opted out user ID. Once received, our system will flag that user and will block all future events or sessions of that user. The “opt out” event technical documentation can be found here.

Delete API

Since the GDPR also entitles the data subject the “right of erasure”, Cooladata introduced a new Delete API that allows you to delete any user properties or historical events and sessions of this user. Keep in mind that if you do not opt-out the user and send any events for that user in the future, new data sent after the delete request will be stored. In addition, since Cooladata does not control your Aggregation Tables, external data sources or data uploaded through our integrations, it is your responsibility to delete that user from these external tables.

The Delete status API will let you know the status of your request for deletion (whether it’s in progress or done) as well as inform you how many events and sessions were deleted for this user once the erasure is completed. The “Delete API” technical documentation can be found here.

Depersonalization of property values

Cooladata also enables you to anonymize personal data in your project.  Since you control what data you send to Cooladata and what is stored, you are responsible for defining what is personal information and what is not.
If some properties in your project contain personal user information, you might want to hash that information based on a certain condition (for instance if a user asked to be anonymized or if that user is based in the EU). Cooladata now enables you to anonymize data based on project wide conditions – talk to your CSM to set this up.
Notice that we automatically collect IPs so if you need the IPs we have collected to be hashed before they are stored you are responsible to use the above mentioned functionality to set this up.  Hashing IP’s will not affect the geolocation enrichment we provide out-of-the box.

Update users API

To exercise the GDPR data subject “right of rectification”, Cooladata enables updating user scope properties with a new User Update API. This API can be used  to update user information without having to send events especially for this purpose. This can be used for a bulk update of user information, or for ongoing updates.

Since Cooladata allows you to match multiple identities to the same user, we’ve also introduced a new User Identities Matching API for adding more identities to the same user.

For more information on the GDPR and Cooladata please see The Cooladata Guidelines for the GDPR Preparations 

User Profile

The new User profile feature allows you to keep a list of all your project users and their attributes, which include behavioral stats such as the user’s frequency or number of visits to your app, as well as customizable user attributes such as total deposit amount or last viewed item. In addition, all user scope properties last values are added to the user profile automatically. This will allow you to easily create user segments as well as deepen your understanding of your user behavior. Questions like “what is my average user LTV?” or “are users who signed up in May 2017 stickier than users who signed up on 2018?” are now so much easier to answer.

Notice that User Profile is an add-on feature and once configured, is only open for project ADMINs. Please contact your CSM or support@cooladata.com to set it up.

Feature Improvements

Query API Limitation Increase

Our query API now allows you to query your raw-data and extract even more data with one request! Our new limit is 5M rows!

User create date corrections

As you know, Cooladata supports sending historical events in bulk uploads. The automatically added property sys_user_create_date keeps the first timestamp of each one of your users. Up until now, it kept the server time of the first event received for each of your users, which caused issues in historical event uploads.

As of the beginning of June sys_user_create_date shows the event_time_ts (original timestamp) of the first event received for each user. If our ETL ever detects an older time, we will update this field to contain the earlier time.

Facebook Ads Integration account selection

Cooladata provides a predefined connection that enables you to upload Facebook Ads data into your project in order to integrate it with your behavioral tracking data. This enables you to measure your social media return on investment and get the most out of your social media campaigns.

 Up until now, we’ve automatically connected all your Facebook Ads accounts to the Facebook user you have signed in with. From now on, you can select the accounts you want to integrate into Cooladata in the “connected accounts section” here:

iOS Version 2.3

The SDK version is now compiled in Apple BITCODE. See our Github for more details.

Step 1 – Plan which events to track

Planning

Try to answer the following questions when planning which events to track:

  1. What are your objectives?
    Think about your business goals and strategies in order to determine the purpose of the data that you want to extract from the events sent to CoolaData. For example, optimizing customer retention, increasing conversions, expanding business leads, growing revenue or enhancing usability.
  2. Which data enables you to achieve these objectives?
    Think about the kind of information that could be sent from your website that would enable you to achieve these business objectives. You can track anything that anyone does in your website and anything that happens at any specific point in time.
  3. Write It Down:
    Each business objective is typically comprised of a series of events that lead up to a target event, such as a purchase. We recommend sending an event for each stage of a user’s journey (funnel) through your website. List the events and the data properties that could be sent to CoolaData at various stages of your users’ experience.

Events

Events represent the path which your users follow through your site.

Examples of typical paths could be:

  • eCommerce: search → view_product_details → add_to_cart → click_checkout → purchase.
  • Gaming: login → start_game → level_up → purchase_bonus_item.
  • Media&Publishing: view_homepage → view_article → click_video → video_completed.

 

Properties

Cooladata supports events with any number of properties, and automatically enriches them with additional properties that are often used.

 

Events by industry

We’ve created a list of recommended events and properties to implement for each industry. See the following links for your industry:

 

 

 

Live Events

The Live Events page displays the full details of the last 1,001 events received in your project. It is specifically useful for monitoring incoming data and identifying invalid events or properties, conflicts in data schema, and other unexpected data issues.

Each row describes a single event.
Columns include all Common Properties collected by Cooladata, and special columns:

  • validation: valid/invalid/pending (a new event that will be added to the schema if valid)
  • invalidComments: will specify the reason if the event is invalid.
  • extraComments: will specify any changes made to the event data (such as invalid properties removed).
    For more information see Handling Invalid Events.
  • raw_data: shows the full events JSON string.

 

 

The page provides the following functions:

  • Search: click Search to filter the list to only show events which contain specific values. The search applies to all columns by default, but can be narrowed down to only apply to a specific column.
  • Refresh: click Refresh at the top right to update the list with the latest 101 events.
  • Select columns: click Select columns at the top right to choose which columns to display in the list.
  • See raw event data: click any row to see the original event JSON as received by Cooladata.

 

APIs Overview

This section describes the application programming interfaces (APIs) that are available for the CoolaData platform. The APIs enable customers to directly access the CoolaData platform (for example, using CoolaSQL (CQL) querying) directly or via 3rd party tools.
The REST-like API, which allows you to use and extend CoolaData’s Platform, provides access to CoolaData Discovery and Query capabilities. The CoolaData REST APIs are a set of URI resources, which grant access to data views.
CoolaData’s URI resources provide access for discovering metadata and executing queries.
The URI resources are now presented in the order of a typical API workflow:
  • Discovery API: This refers to the API that exposes information about the metadata.
  • Query API: The Query API enables you to query a data source using the CoolaData platform.
  • Delete API: Allows you to request to delete a specific user’s historical events and sessions from Cooladata.
  • User Update API: Enables updating user scope property values for specific users without having to send events.
  • User Identitities Matching API: Enables coupling between a registered user ID and an anonymous user ID without having to send events.

 Authorization

All of Cooladata’s API’s require Authorization Tokens.
Each Cooladata user has each own authorization token for carrying out API requests. Your Authorization Token can be found in the Cooladata App by clicking on the user icon in the top right hand side of the application:

REST API

Using CoolaData’s REST API enables customers to send data to CoolaData. All API calls are HTTPS based, and require an app-key, provided by CoolaData. REST API data is sent in POST method.
Each call should be sent as an array of JSONs – one JSON per event. We only support flat JSON files – nested JSON is not supported. A single call may include as many as 300 events in a batch. To achieve better throughput, and if you are having many events per second, you should send ONLY in batch (and not one after the other).

For each call CoolaData responds with a “200 ok” status.
Note: As we accept events in JSON format, avoid using escape characters in the data sent to CoolaData. See a list of all escape chars here: http://json.org/. As a best practice we recommend sending events in the event payload url encoded.

Method:
POST
Endpoint:

https://api.cooladata.com/v3/[app_key]/track

Headers:

Content-Type: application/x-www-form-urlencoded

Payload:

The payload must be in JSON format, and URL encoded (to avoid escape characters).
The JSON structure is:

{events:[{"parameter_name":"parameter_value",...},...]}

The following mandatory parameters must be sent with all events (events without these mandatory fields will be invalid and will not be saved in CoolaData):

NameTypeMandatoryDescription
event_nameStringYesThe name of the event to report.
user_idStringYesThe id (possibly anonymous) of the user who generated the event
event_timestamp_epochNumberYes**Must be sent if the project is NOT set to override event time-stamp with server time.
Format is milliseconds (13 digits) only. If the project is defined to override event time-stamp with server time, any time-stamp sent with the event will be ignored.
Supported time stamp formats:
yyyy-mm-ddThh:mm:ss
yyyy-mm-dd
yyyy/mm/dd
dd/mm/yyyy
dd-mm-yyyy
Epoch in milliseconds
Days and months can also be sent in 3 letter format, such as: Mon, Tue, etc. or Jan, Feb, etc.
custom propertiesString/NumberNoSee more information here

Example:

{
"events":[
{"event_name":"signup","user_id":"1001","event_timestamp_epoch":1436798955476,"first_name":"john","last_name":"doe"},
{"event_name":"login","user_id":"1005","event_timestamp_epoch":1436798956879,"first_name":"johanna","last_name":"jones"}
]
}

Response:

{"results":{},"status":true,"events_count":n,"version":"5"}

 

Auto Resolution

By default, CoolaData server resolves several properties such as IP and DUA to enrich the event with additional data

DUA Resolution:

User information such as browser, os other technology information is resolved based on the  session_dua property.

When one users a REST API, the users session_dua is best be send with every event to enable CoolaData to enrich the event with additional user agent info.

IP Resolution:

Geographic data such as city, country and state according is resolved based on the session_ip property.
When one uses a Rest API, the IP is determined according to your server, hence in such case, the geographical data won’t represent correctly your users’ geographical data. In case you would like to resolve this, send the actual session_ip as part of the event JSON, and if it’s not available set it to be 0 so we won’t override it with the server ip. For example:

curl -X POST https://api.cooladata.com/v2/aknek0zuo8i4ztcmgopn5wotsceg80m9/track -d
'{"events":[{"session_ip":"0","event_name":"Add Item","user_id":"uid_123456"}]}'