Take control of your Data – How Cooladata lets you Manipulate data in Multiple ways

As you know, Cooladata lets you control which data to collect from numerous sources. You decide what you want track and Cooladata warehouses it for all your analytical purposes.

Once you decided which sources you want to collect, cooladata normalizes the data and makes it very easy to execute queries no matter how complex they are, to summarize or aggregate data for reporting and serving to other people or other platforms. Since accuracy is essential, Cooladata manages the data at the raw event granularity.

Cooladata ensures that your data is validated against our self learning schema to ensure it is correctly formatted and processed successfully. On top of this validation, you can control how your data is processed, transformed and enriched before it is stored, or manipulate it after the storage is done for easier querying and reporting.

Recently, we’ve added two very useful features to the data manipulation “tool set” Cooladata offers their customers as part of every project:

  1. Custom Manipulations as part of the ETL
    This new feature allows you to customize manipulations on the data before it is stored and requires custom set-up with your Cooladata Customer Success Manager.

    – Blocking or invalidating events: Every data scientist knows: “garbage in – garbage out”. To avoid entering messy data to your project – you could block or invalidate events that match a certain condition as part of your custom project set-up. A common use case is blocking out bots or internal QA users.

    – Changing events before they are stored: manipulating events as part of the ETL can have several use case, such as: hashing personal information, changing data types, extracting specific parameters from long strings or arrays into designated columns, etc. The biggest advantage of this type of manipulation is that it’s done before the data is even stored so it doesn’t require any further manipulation and helps a lot with achieving consistency of data.

  2. Sending retroactive events back to the ETL
    Cooladata allows sending events to the ETL to be stored in the “cooladata” events table with the rest of your events. This is a very common case for events that can be uploaded from an external DB, historical data storage, the invalids table or even a service API. This task can be scheduled to run automatically like any other task in Cooladata and can even be set up as a step in a Job. Notice these events are out of context so the automatic sessionization done by our ETL might be affected. To avoid that you can turn off sessionization for these events.

These above mentioned  features are already deployed and are ready to be exploited. Just to serve as a reminder, here is the full list of data manipulation tools already available:

1. ETL – out of the box data manipulations:

  • Server – side Sessionization: unifies sessions from all platforms into one single session, based on the user ID and the custom session definitions defined per project
  • Enrichment : automatic breakdown of user agent and IP into device details and geo location.
  • User identities matching: unifying between several identities of the same user. The most common use case are anonymous and registered users that login in the middle of the session. Cooladata unifies the 2 users into one identity and stores the data accordingly. You can then query the entire journey of the user, from before he registered to after, understanding the complete picture.
  • Auto-population based on scope: each column is auto-populated based on the scope of the property – for instance: A FIRST USER scope property stores the first values received for this user in all of that user’s events, whereas LAST SESSION scope property stores the last non-null value in all the session’s events received after this value. This saves the analyst the effort of joining multiple events and tables. Each user also receives automatic property of create date, to easily extract the first timestamp the user was seen in the app and it’s first session is always marked as is_new=1.

2. Virtual Properties – Some properties require dynamic manipulations that are applied ad hoc while querying the data. This feature allows you to store SQL expressions as virtual properties, and select these properties in your reports or even filter by them in dashboards. The expression can contain a formula that hides the complex processing of a property. A common use case is total order amount calculations based on several columns or unifications of two columns using the IFNULL function.

3. Aggregation Tables – Aggregation Tables automatically run scheduled calculations and data aggregations and save them in a separate, permanent table. This can be used to compute and store aggregated data in order to enhance the performance of queries running on large scales of data. Most customer store aggregated tables based on several data sources and multiple joined tables. Querying these tables are easier then writing the complex joins and SQLs in each query.

4. Models- Models are designed to add R and Python capabilities to your Cooladata’s workflow. Models enable writing tasks based on R or Python scripts. This allows you to encompass the capabilities of these languages to manipulate your data and save the data frames created by those models into tables in your project.

Both the Aggregation Tables and Models, as well as the new retroactive event sender task, can be scheduled as steps in our Jobs to make up a repeating data manipulation workflow.

Print Friendly, PDF & Email

May 2018 Release Notes

During the past few months we have been busy adding and improving features you requested in our product, as well as many features that are designed to help our customers meet the requirements of the GDPR.

In this product update email you will find updates on:

New features:

  • Models running custom scripts
  • GDPR compliance support features
  • User Profile

Feature Improvements:

  • Query API limitation increase
  • User create date corrections
  • Facebook integration account selection

SDK Update

  • iOS 2.3

Models

In addition to the R and Python Reports we added in our latest update, Cooladata now supports running data manipulation scripts in R or Python, and saving the data frames  to tables in your project. The new Models feature opens up a new dimension of exploration and analysis of your data, encompassing all capabilities of these languages: you can run statistical models, data processing procedures, or even write your own custom integration fetching data from APIs and storing it in Cooladata.

Our customers are already using this feature to cluster users using R K-Means, schedule custom integrations to fetch data from affiliates and service providers such as Brandwatch or Rakuten as well as predict player LTV or likelihood to convert.

Models are similar to Aggregation Tables in the way they are built. They can run on any data source in you project, including events, tables, and linked data sources and can be scheduled to run automatically on a set frequency or as part of a Job.
The model generates results that are stored in a table, which can then be queried from anywhere, including reports, publications and Query API.

GDPR Compliance Support

A few weeks ago, we have shared our ongoing efforts to help our customers meet the requirements of the GDPR. We are happy to announce that we have finished the development of the major key features of the GDPR and they are now available as BETA features:

Opting out users

Cooladata enables you to opt-out a user in order to stop tracking its future activities. Opting out a user will be done either by the SDK or a rest API request to Cooladata, by sending a “cooladata_opt_out” event with the opted out user ID. Once received, our system will flag that user and will block all future events or sessions of that user. The “opt out” event technical documentation can be found here.

Delete API

Since the GDPR also entitles the data subject the “right of erasure”, Cooladata introduced a new Delete API that allows you to delete any user properties or historical events and sessions of this user. Keep in mind that if you do not opt-out the user and send any events for that user in the future, new data sent after the delete request will be stored. In addition, since Cooladata does not control your Aggregation Tables, external data sources or data uploaded through our integrations, it is your responsibility to delete that user from these external tables.

The Delete status API will let you know the status of your request for deletion (whether it’s in progress or done) as well as inform you how many events and sessions were deleted for this user once the erasure is completed. The “Delete API” technical documentation can be found here.

Depersonalization of property values

Cooladata also enables you to anonymize personal data in your project.  Since you control what data you send to Cooladata and what is stored, you are responsible for defining what is personal information and what is not.
If some properties in your project contain personal user information, you might want to hash that information based on a certain condition (for instance if a user asked to be anonymized or if that user is based in the EU). Cooladata now enables you to anonymize data based on project wide conditions – talk to your CSM to set this up.
Notice that we automatically collect IPs so if you need the IPs we have collected to be hashed before they are stored you are responsible to use the above mentioned functionality to set this up.  Hashing IP’s will not affect the geolocation enrichment we provide out-of-the box.

Update users API

To exercise the GDPR data subject “right of rectification”, Cooladata enables updating user scope properties with a new User Update API. This API can be used  to update user information without having to send events especially for this purpose. This can be used for a bulk update of user information, or for ongoing updates.

Since Cooladata allows you to match multiple identities to the same user, we’ve also introduced a new User Identities Matching API for adding more identities to the same user.

For more information on the GDPR and Cooladata please see The Cooladata Guidelines for the GDPR Preparations 

User Profile

The new User profile feature allows you to keep a list of all your project users and their attributes, which include behavioral stats such as the user’s frequency or number of visits to your app, as well as customizable user attributes such as total deposit amount or last viewed item. In addition, all user scope properties last values are added to the user profile automatically. This will allow you to easily create user segments as well as deepen your understanding of your user behavior. Questions like “what is my average user LTV?” or “are users who signed up in May 2017 stickier than users who signed up on 2018?” are now so much easier to answer.

Notice that User Profile is an add-on feature and once configured, is only open for project ADMINs. Please contact your CSM or support@cooladata.com to set it up.

Feature Improvements

Query API Limitation Increase

Our query API now allows you to query your raw-data and extract even more data with one request! Our new limit is 5M rows!

User create date corrections

As you know, Cooladata supports sending historical events in bulk uploads. The automatically added property sys_user_create_date keeps the first timestamp of each one of your users. Up until now, it kept the server time of the first event received for each of your users, which caused issues in historical event uploads.

As of the beginning of June sys_user_create_date shows the event_time_ts (original timestamp) of the first event received for each user. If our ETL ever detects an older time, we will update this field to contain the earlier time.

Facebook Ads Integration account selection

Cooladata provides a predefined connection that enables you to upload Facebook Ads data into your project in order to integrate it with your behavioral tracking data. This enables you to measure your social media return on investment and get the most out of your social media campaigns.

 Up until now, we’ve automatically connected all your Facebook Ads accounts to the Facebook user you have signed in with. From now on, you can select the accounts you want to integrate into Cooladata in the “connected accounts section” here:

iOS Version 2.3

The SDK version is now compiled in Apple BITCODE. See our Github for more details.

Print Friendly, PDF & Email

February 2018 Release Notes

Connect

We are constantly adding more ways to connect your data with Cooladata, to provide you with a single point of management for your data. We aim to make new connections available to any user, with just a few clicks, so you can configure them yourself and not need R&D to get insights quickly.

Data Sources Page (Admins only)

The new Data Sources Page provides you with an overview of the multitude of ways you can connect with Cooladata, both for sending data and querying it. To access the page, from the project menu, select Data Sources. Here, browse through the different categories: Event sources, Integrations, Linked data sources and BI Tools, and click an item to learn more about it.

​​

​Integrations (Admins only)

Salesforce

Salesforce develops CRM solutions that help customers increase sales, improve customer service, collaborate with their team, and develop customized applications on the Salesforce platform.

Integrating Salesforce into your Cooladata account will and allow you to enrich the behavioral data tracked by Cooladata with your CRM data.

Setting up the Salesforce integration will fetch the following Salesforce objects: Leads, Opportunities, Accounts, Contacts, Price books, Products, Campaigns, Events.

All the above objects’ fields, standard and custom, are fetched. See each object’s standard fields list here.
Data is updated daily. To configure a specific update hour please contact us at support@cooladata.com.

Intercom

Intercom is a powerful set of tools for better managing your company’s customer support system. It includes a help center with a feedback system, which you can use to focus future articles on the growing needs of your customers, and it also provides a robust conversation system that allows you to assign support teams to customers based on specific criteria (about the customer or discussion topic), rather than just based on availability. Intercom is designed to create a more effective customer support network by specifically tracking and targeting your customers’ needs.

Cooladata’s integration with Intercom empowers marketers and customer management teams to understand the entire user journey, and the impact of Intercom on their users’ lifetime and value.

The integration runs daily, updating the previous day’s end of day data in each table: intercom_conversations, intercom_users.

To integrate Intercom to your Cooladata project please contact your customer success manager or email us at support@cooladata.com.

Google Search Console

Google Search console (GSC) is a collection of Google SEO tools and reporting. It offers data and configuration control for your website as well as a variety of visitor metrics.

As an online marketer, the GSC “search analytics” section is probably the most useful report for you. GSC “search analytics” delivers insights into how your website is performing in regards to the overall organic search traffic from Google. You can see the most popular queries, pages, countries and devices. Additionally, for better understanding of your website’s performance in Google’s search results, you can use different filters to segment traffic even further. See what keyword searches on Google led to the site being listed in the SERPs, and the click through rates of such listings.

Google Sheets

The Google Sheets integration enables you to use your Google Spreadsheets as tables in your Cooladata project, and query them directly from your Cooladata project.

Connected spreadsheets are always updated and reflect the exact data your sheet contains.

The new table will be added to your project Schema, under Integration Tables, in the Table Name you selected. Query it by using the following syntax:

FROM < table name >

Firebase

Firebase is Google’s mobile platform that helps you quickly develop high-quality apps and grow your business.

You can now integrate Firebase with Cooladata so that all the events generated by Analytics for Firebase will be sent as any other event to your Cooladata project.

To do so, you will need to link your Firebase app to BigQuery. Linking your Firebase app to BigQuery allows you to access your raw, unsampled event data along with all of your parameters and user properties.

Once you’ve linked your Firebase app to a BigQuery project, we will connect it to your Cooladata project. This will stream all your event data directly into your Cooladata project, where you can query, export or join your data with data from other platforms and data sources in your project, to perform custom analysis.

To set up a new Firebase integration, contact your Customer Success Manager or email us at support@cooladata.com.

Process

Aggregation Tables – Run Multiple Tasks Per Table (Admins only)

You can now create multiple tasks per table. Each item in the Aggregation Tables page is now a task, which runs on a table. This enabled you to run several ETL processes for the same table, for example: extracting data from you Linked Database, adding column to include specific data, and running aggregations on that data. When creating a task you can select to run on an existing tables, or create a new table. Task name can be anything you want.

 

​URL Parameter Virtual Properties (Admins only)

This new virtual property lets you create a separate property from any parameter sent in your webpage URL. Use this to extract any lead information passed from marketing campaigns, forms, etc. To create a new virtual property, from the project menu select Properties, click the + icon, and choose URL Parameter. All virtual properties are calculated on the fly, and can be modified at any time.


Expressions in Virtual Properties (Admins only)

Previously you could use two types of calculations in virtual properties: Virtual Dimension and Virtual Measure. To simplify the process, we have combined both capabilities to one, easy to use type. In this new virtual property type, you can enter any CQL expression you want to calculate, be it a function, CASE, or calculation based on other properties. Existing Virtual Dimensions have been automatically converted to the equivalent CASE expression. Unlike Virtual Dimension, where you could only use strings as the results of the CASE expression, with Expression you can select any value, includes calculations on the data.


Insights

R and Python Reports

In the modern world of analytics, SQL queries can only go so far. Powerful insights rely on advanced machine learning and AI algorithms. Now, you can incorporate any such scripts in Cooladata.

As a first step, we enable you to run R and Python scripts in your reports. Run any script and plot any visualization to be shown in a report, embedded reports, dashboards or publications.

To create a new R or Python Report, from the New Report gallery go to Scripts and choose the report type. Enter the CQL query in the top section to draw the data, and run the script below. To use a library, see list of pre-installed libraries in the script, or add new ones in “Additional Libraries”.

Note that this feature is still in beta – your feedback will be much appreciated!


Alerts Builder (Admins only)

Getting notified on business events is critical to managing your business. To help you with this, Cooladata enables email alerts to be sent on a scheduled basis, whenever a condition applies in your data. Now you can set advanced behavioral triggers yourself, without writing the CQL query. Get alerted on changes in users activity, events and property values, compared to preceding period or parallel historical periods. To create a new alert, from the project menu select Alerts, click the + icon and select Builder. Set your trigger condition and schedule, and add any pertinent info and reports to the email body.


PIVOT CQL Function

Data is constantly changing, and requires us to change with it. When dissecting data across various dimensions, we often need to update our analysis as the dimensions changes. To help you automate this process, we have added support to the useful PIVOT function in our CQL (Cooladata SQL) language. Like in the pivot visualization, the PIVOT function lets use convert row values of a specific dimension to multiple columns, allowing you to treat each as a measure and compare them side by side in a single chart. See the PIVOT documentation for the exact syntax and options in the function.

In the following example we converted country values to columns, showing them together in a bar chart.


Share

Mobile Support

You can now view reports and dashboards on your mobile device. Specifically designed to let you access reports sent to your email, the mobile support also fits complete dashboards in the palm of your hand, to let you access them anywhere, anytime. Open Cooladata on a mobile Chrome browser to access it.


White Label Publications

Share reports with your clients via email directly from Cooladata. You can now request to set all your publications to be sent with your branding. If you use Cooladata as a whitelabel site, all links can be updated to direct to that site. If not, they can be removed. Contact your CSM or email us at support@cooladata.com to set up.

Print Friendly, PDF & Email

October 2017 Release Notes

Help and support

New help menu

The new menu includes a variety of support channels, to help you make the most of your experience on Cooladata. Click the  button at the top right corner from anywhere in the app to open it.
The new help menu includes:

  • Live chat: contact our live support via on-screen online chat. Live chat is available Sunday-Thursday on Israel working hours, or leave a message at any other time and we will get back to you as soon as we resume operations!
  • Cooladata Docs: link to the documentation – docs.cooladata.com
  • FAQ: link to the FAQ page – docs.cooladata.com/faq/
  • Query Bank: link to the Behavioral Analytics Query Bank – cooladata.com/querybank/
  • What’s new: link to the release notes page – docs.cooladata.com/releasenotes/
  • Getting Started: show the “new user tutorials”: “Explore the Dashboard”, “Create and add reports”, and “Create your first project” – you can restart them, or try them out if you haven’t had a chance to yet. Note that the first tutorial must be started when in a dashboard.

 

 

Data Management

Events and properties relation in Schema

In response to multiple requests, we’ve added a new feature to the project Schema panel – properties to events relation. Now, to see all the properties sent with a specific event, simply expand that event in the schema, and to see all the events a certain property is sent with, expand that event in the schema. This should make it easier to query all the data relevant to your events, without missing any important properties.

Properties sent with each event Events each property is sent with

 

Linked Data Sources in Schema

The project Schema now also provides you direct access to your Linked Data Sources metadata. If you have any Linked Data Sources set up, open the Linked Data Sources section to show the connections you’ve set up. Expand a connection to query all the tables it contains (this can take a moment). Expand a table to query all the data columns in contains (this can take a moment).

 

To query a Linked Data Sources, from CQL reports, use:

For example:

 

Full property list discovery

We’ve been working hard to give you more transparency on the data schema in your projects. As an additional feature to the Schema panel, CQL reports now support the use of

This will display:

  1. All the project’s active properties, not including virtual properties.
  2. Limited to top 1,000 rows (query must include “LIMIT” of up to 1,000).

 

“First value” properties

Properties (non-virtual) no support saving the first received value. When on, saves the first value received for each user/session (according to the property scope) after the property was created.
Note, does not apply to event scope, as will always save the data sent. “First value” cannot be modified after the property was created (on edit), only on new properties. This also allows you to create multiple properties from the same data source – up to 5 different scopes and first/last configurations (see table below).

For example, these are the result you will get when querying first value properties*, as opposed to their default, last value, counterparts:

Session Event (default) User first User last (default) Session first Session last (default)
1st a a f a c
1st b a f a c
1st c a f a c
2nd d a f d f
2nd e a f d f
2nd f a f d f

* The last session and user data values above are only applicable to queries on session/user level properties. Event level queries will display the last value that was sent up to that specific event. For more information see the FAQ, or contact us at support@cooladata.com.

In addition, all the following properties have been added as default common properties to all projects:

Name Scope Source
user_first_utm_source User utm_source
user_first_utm_campaign User utm_campaign
user_first_utm_content User utm_content
user_first_utm_medium User utm_medium
user_first_utm_term User utm_term
user_first_referring_domain User referring_domain
user_first_referring_url User referring_url
session_first_referring_domain Session referring_domain
session_first_referring_url Session referring_url

 

Segment and Amazon Kinesis support

Cooladata now supports two new event sources – Segment and Amazon Kinesis. This expands the variety of channels for you to send data to Cooladata, thus aggregating even more data to a single platform. See the individual documentation pages, or the Data Sources page in the app for more information

 

 

Analysis and Data Visualization

Report “Save as”

Quickly create copies or variations of an existing report. From the save button, hit the arrow next to it a select “Save as”, to save your latest changes to a separate report. You can name the new report anyway you’d like. Note that the app will not navigate to the newly created report – to quickly go to it click the link in the confirmation message.

 

Configure decimal points in table view

The table view settings now allow you to configure the decimals points for number. The default is 3, you can select up to 9.

Print Friendly, PDF & Email

July 2017 Release Notes

Jobs

Jobs allow you to schedule a sequence of tasks that will run consecutively, one after the other, once the previous task has been completed. This enables you to create more advanced processes on your data, without manual intervention, such as running aggregations over other aggregations, and conditioning alerts to only run once an integration has been updated.

 

Segments are open to non-admin users

Non-admin users can now create new and update their own segments. Segments are accessible from the Project menu, where they can be created or edited, or can be created from Funnel and Cohort reports – Explore users.

 

Updated integrations page

The integrations page has been revised to be easier to use.
The list shows the integration type, name, table names for querying, last update and current status.
Each integration shows run history (from the row options).

Supported integration types:

  • Google Cloud Storage: fully customizable
  • Gmail: fully customizable
  • Google AdWords: default integration, updated daily
  • Facebook Ads: default integration, updated daily
  • MailChimp: custom integration – contact your CSM to implement

 

Query integration tables using KPI report

The KPI report now supports querying integration tables. To do so:

  1. Open a new KPI report
  2. From the top right of the form area, click the data source button (“Cooldata” by default).
  3. From the tables list, select the integration table you would like to query.
  4. If the table contains a timestamp column, select it. This will allow you to use the date picker and time breakdowns.

 

Mobile friendly publications email

The publications email has been updated to fit any device, including mobile devices. Attached images can be expanded to full size (method depends on the device OS).

 

Default Real Time dashboard

Projects enabled for Real Time reports will include a preset dashboard with common reports. This allows you to monitor your data in real time, as you would in other tools. Contact your CSM to enable Real Time reports in your project.

Print Friendly, PDF & Email