Cooladata Winter 2019 Release Notes

In this product update you will find updates on:

  • Path analysis visualization (Sankey)
  • New enhanced date picker
  • Additional customized chart color palettes
  • Improved KPI report functionality: added Having and Sort by capabilities
  • Firebase support
  • Google Standard SQL support (We continue to support Legacy SQL)
  • Date condition for merging Aggregation and Models tables
  • UI / UX improvements
  • Shipment and cache notifications
  • Ability to download models logs

 

New / Improved Features:

  • Path analysis visualization (Sankey)
    We are aware that the best way to get insights is by visualizing the data, so we’ve looked for a better way to present the users journey: Meet our new Sankey visualization!
    It allows you to read the users actions in a simple and easy way and can help you maximize your conclusions about your users’ flow. It available in Path builder and CQL editor and the colors of the graph can be controlled through the visualization setting
  • New enhanced date picker
    We designed a new date picker that expands the querying abilities and improves the user experience.
    now you can query your data with more time functions including:
    – “Last N Days“: counting days back, not including the current day. In order to include the current day just set the “Include today” toggle ON.
    – “Previous N Weeks / Months / Quarters / Years“: Calculates calendar time period, that do not include the current ones.
    – “Current Week / Month / Quarter / Year“: calculates the current calendar time period.

    Custom date range by hours: adds the option to define start hour at the start date and end hour at the end date.
  • Additional customized chart color palettes
    We expanded our charts color palette and as from today, each report could be set with a different color palette to allow the best fit for your site’s design.
    In order to choose a new palette, open the report and change the color palette on the visualization setting.

  • Improved KPI report functionality: added Having and Sort by capabilities
    Cooladata added an advanced menu to the KPI report which enables you to add “HAVING” and “SORT BY” SQL function to the KPI query.
    The advanced menu will be shown at the bottom of the builder once the KPI report has at least one custom step and a breakdown.
    This ability allows you to filter aggregated reports  easily.

    For more information see our KPI documentation.
  • Firebase support
    Based on customer feedback and spotting an underserving in firebase analytics we released special features to help companies that rely on firebase as a development platform.

    1. JSON flattening at the ETL level for deserializing nested JSON and matching it to the known schema of firebase.  The result is the ability to query and not having to worry about unesting.
    2. Multiple project consolidation – companies running two projects or more enjoy consolidation for enhanced querying and comparison. This is also a significant cost reduction.
    3. Unlimited properties per event – 25 unique parameters with each event type. With Cooladata, there is no limit to the number of custom properties and dimensions per event.
    4. Enriching from Firebase real-time database – we managed to combine and manage update all properties and dimensions from Firebase into the fact table. We make it more efficient, producing insights by merging both raw and real-time data.
      For more information see our documentation.
  • Google Standard SQL support
    Up until now, Cooladata enabled running freehand queries over Cooladata using Standard SQL with CQL editors only. Cooladata now supports create a fully standard project, which automatically will support external tables with Standard dialect, support standard expressions and will allow querying the data with KPI report as part of a strategic plan to migrate Cooladata to operate in Standard SQL, encompassing all the advantages of the language, including performance enhancements, query optimizations and superior functionality.
    For more information about the new Standard SQL functions (like “Unnest”, “With”, “Array” etc.) see our Standard SQL documentation.
  • Date condition for merging Aggregation and Models tables
    Up until now, using “Append and Update strategy” allowed you to update existing rows (according to a unique key) and append new rows to the destination table. Starting from toady, Cooladata allows adding to partitions tables (Aggregation Tables or Models) with “Append and Update strategy” days filter (based on the partitions column) which enables to replace only the latest tables’ rows. This feature improves executions’ performance and helps to manage huge tables.
  • UI / UX improvements: Saving lists filters during the session and open in a new tab
    We’ve listened to our customers’ requests and continue to improve our user interface. As parts of our efforts, we enabled to keep the list’s filters until the end of the session. This includes the search, column filter, sorting, the active tab and the number of the presented results in the page (page size). In addition, we allow opening reports, dashboards and tasks in a new tab using CTRL+Click/middle mouse click.
  • Shipment and cache notifications
    In order to improve query performance, Cooladata uses extensively caching and shipment.
    Whenever a report is executed, Cooladata notify the user whether the data returned from the cache or used shipment 
    Controlling whether a report will use cache, as well as shipment, could have never been easier. Just click on the “Report Options” and set the toggle ON or OFF

    For more information see our report options documentation.
  • Ability to download models logs
    We added all the attached logs files to each run, available for download at the “View logs” window

 

Print Friendly, PDF & Email

Cooladata Fall 2018 Product Update 

In this product update you will find updates on:

New Features:

  • Standard SQL
  • Partitioning in External Tables
  • Writing Models to your external destination.
  • Merge data using Integrations

Feature Improvements:

  • Simplifying working with several projects in Cooladata
  • Cohort Auto Captions
  • Column filters for Table Visualization
  • ETL auto-population by scope ignoring NULL values
  • Lookup values for virtual properties and expressions

New features:

  • Standard SQL
    Up until now, Cooladata’s SQL was compatible with Google BigQuery SQL Legacy dialect. Cooladata now enables running freehand queries over Cooladata using Standard SQL, as part of a strategic plan to migrate Cooladata to operate solely in Standard SQL, encompassing all the advantages of the language, including performance enhancements, query optimizations and superior functionality. Writing Standard SQL is now available when creating a CQL report, aggregation table, model, CQL segment or alert. For more information see our Standard SQL documentation.
  • Partitioning in External Tables
     Cooladata’s events and sessions tables are automatically partitioned by time. From now on, Cooladata also enables the partitioning of other external tables in your project by a timestamp or date column. Partitioning makes sure you run only on the selected time range in the table instead of the entire table, improving performance and minimizing query run time. Partitioning is available only when creating aggregation tables or models using Standard SQL dialect, and are only available for querying in Standard SQL dialect.
  • Merge data using Integrations
    Integrations fetching external data can now write to a table in “Append and Update strategy”, which will update existing rows (according to a unique key) and append new rows to the destination table.  This can reduce the number of rows fetched and integrated in each run to only the updated and new rows.
  • Writing Models to your external destination
    Models can now save their results to external destinations as available in aggregation tables: your Google Bucket or BigQuery dataset.

Feature Improvements:

  • Simplifying working with several projects in Cooladata
    We’ve listened to our customers’ requests and enabled filtering all the dashboards and reports shown on the lists filtered by the active project. Since you can add reports from several projects to the same dashboard,  some dashboards might be shown in a few projects. When creating filters, make sure you choose the right project and table to filter by in the filter widget.
  • Chart Visualization – Choose whether or not to auto-fit your chart
    We are aware that sometimes, it’s the big picture showing big trends that is important in a chart, whereas in other times – it’s the little details we want our data consumers to focus on. That’s exactly why we have enabled auto-fitting a single chart in the report size – so you can choose how your report will be viewed in a dashboard.

  • Cohort Auto Captions
    The cohort widget now auto-populates the cohort captions that explain to your report consumers what it is they are seeing. You can override the captions manually as you wish.

  • ETL auto-population by scope ignoring NULL values
    Sending an explicit Null for a session or user scope property will be ignored and the last non-null value sent will be auto-populated (“smeared”) going forward.
  • Column filters for Table Visualization
    Filtering a single value has never been easier. Just click on a single column header and choose the value you want to see.

  • Lookup values for virtual properties and expressions
    Lookup values are the list of values in the filter drop down. These are populated as part of the ETL, so they weren’t available for virtual properties and expressions until now. From now on, all properties can have a drop down list of values in filters.
Print Friendly, PDF & Email

Take control of your Data – How Cooladata lets you Manipulate data in Multiple ways

As you know, Cooladata lets you control which data to collect from numerous sources. You decide what you want track and Cooladata warehouses it for all your analytical purposes.

Once you decided which sources you want to collect, cooladata normalizes the data and makes it very easy to execute queries no matter how complex they are, to summarize or aggregate data for reporting and serving to other people or other platforms. Since accuracy is essential, Cooladata manages the data at the raw event granularity.

Cooladata ensures that your data is validated against our self learning schema to ensure it is correctly formatted and processed successfully. On top of this validation, you can control how your data is processed, transformed and enriched before it is stored, or manipulate it after the storage is done for easier querying and reporting.

Recently, we’ve added two very useful features to the data manipulation “tool set” Cooladata offers their customers as part of every project:

  1. Custom Manipulations as part of the ETL
    This new feature allows you to customize manipulations on the data before it is stored and requires custom set-up with your Cooladata Customer Success Manager.

    – Blocking or invalidating events: Every data scientist knows: “garbage in – garbage out”. To avoid entering messy data to your project – you could block or invalidate events that match a certain condition as part of your custom project set-up. A common use case is blocking out bots or internal QA users.

    – Changing events before they are stored: manipulating events as part of the ETL can have several use case, such as: hashing personal information, changing data types, extracting specific parameters from long strings or arrays into designated columns, etc. The biggest advantage of this type of manipulation is that it’s done before the data is even stored so it doesn’t require any further manipulation and helps a lot with achieving consistency of data.

  2. Sending retroactive events back to the ETL
    Cooladata allows sending events to the ETL to be stored in the “cooladata” events table with the rest of your events. This is a very common case for events that can be uploaded from an external DB, historical data storage, the invalids table or even a service API. This task can be scheduled to run automatically like any other task in Cooladata and can even be set up as a step in a Job. Notice these events are out of context so the automatic sessionization done by our ETL might be affected. To avoid that you can turn off sessionization for these events.

These above mentioned  features are already deployed and are ready to be exploited. Just to serve as a reminder, here is the full list of data manipulation tools already available:

1. ETL – out of the box data manipulations:

  • Server – side Sessionization: unifies sessions from all platforms into one single session, based on the user ID and the custom session definitions defined per project
  • Enrichment : automatic breakdown of user agent and IP into device details and geo location.
  • User identities matching: unifying between several identities of the same user. The most common use case are anonymous and registered users that login in the middle of the session. Cooladata unifies the 2 users into one identity and stores the data accordingly. You can then query the entire journey of the user, from before he registered to after, understanding the complete picture.
  • Auto-population based on scope: each column is auto-populated based on the scope of the property – for instance: A FIRST USER scope property stores the first values received for this user in all of that user’s events, whereas LAST SESSION scope property stores the last non-null value in all the session’s events received after this value. This saves the analyst the effort of joining multiple events and tables. Each user also receives automatic property of create date, to easily extract the first timestamp the user was seen in the app and it’s first session is always marked as is_new=1.

2. Virtual Properties – Some properties require dynamic manipulations that are applied ad hoc while querying the data. This feature allows you to store SQL expressions as virtual properties, and select these properties in your reports or even filter by them in dashboards. The expression can contain a formula that hides the complex processing of a property. A common use case is total order amount calculations based on several columns or unifications of two columns using the IFNULL function.

3. Aggregation Tables – Aggregation Tables automatically run scheduled calculations and data aggregations and save them in a separate, permanent table. This can be used to compute and store aggregated data in order to enhance the performance of queries running on large scales of data. Most customer store aggregated tables based on several data sources and multiple joined tables. Querying these tables are easier then writing the complex joins and SQLs in each query.

4. Models- Models are designed to add R and Python capabilities to your Cooladata’s workflow. Models enable writing tasks based on R or Python scripts. This allows you to encompass the capabilities of these languages to manipulate your data and save the data frames created by those models into tables in your project.

Both the Aggregation Tables and Models, as well as the new retroactive event sender task, can be scheduled as steps in our Jobs to make up a repeating data manipulation workflow.

Print Friendly, PDF & Email

May 2018 Release Notes

During the past few months we have been busy adding and improving features you requested in our product, as well as many features that are designed to help our customers meet the requirements of the GDPR.

In this product update email you will find updates on:

New features:

  • Models running custom scripts
  • GDPR compliance support features
  • User Profile

Feature Improvements:

  • Query API limitation increase
  • User create date corrections
  • Facebook integration account selection

SDK Update

  • iOS 2.3

Models

In addition to the R and Python Reports we added in our latest update, Cooladata now supports running data manipulation scripts in R or Python, and saving the data frames  to tables in your project. The new Models feature opens up a new dimension of exploration and analysis of your data, encompassing all capabilities of these languages: you can run statistical models, data processing procedures, or even write your own custom integration fetching data from APIs and storing it in Cooladata.

Our customers are already using this feature to cluster users using R K-Means, schedule custom integrations to fetch data from affiliates and service providers such as Brandwatch or Rakuten as well as predict player LTV or likelihood to convert.

Models are similar to Aggregation Tables in the way they are built. They can run on any data source in you project, including events, tables, and linked data sources and can be scheduled to run automatically on a set frequency or as part of a Job.
The model generates results that are stored in a table, which can then be queried from anywhere, including reports, publications and Query API.

GDPR Compliance Support

A few weeks ago, we have shared our ongoing efforts to help our customers meet the requirements of the GDPR. We are happy to announce that we have finished the development of the major key features of the GDPR and they are now available as BETA features:

Opting out users

Cooladata enables you to opt-out a user in order to stop tracking its future activities. Opting out a user will be done either by the SDK or a rest API request to Cooladata, by sending a “cooladata_opt_out” event with the opted out user ID. Once received, our system will flag that user and will block all future events or sessions of that user. The “opt out” event technical documentation can be found here.

Delete API

Since the GDPR also entitles the data subject the “right of erasure”, Cooladata introduced a new Delete API that allows you to delete any user properties or historical events and sessions of this user. Keep in mind that if you do not opt-out the user and send any events for that user in the future, new data sent after the delete request will be stored. In addition, since Cooladata does not control your Aggregation Tables, external data sources or data uploaded through our integrations, it is your responsibility to delete that user from these external tables.

The Delete status API will let you know the status of your request for deletion (whether it’s in progress or done) as well as inform you how many events and sessions were deleted for this user once the erasure is completed. The “Delete API” technical documentation can be found here.

Depersonalization of property values

Cooladata also enables you to anonymize personal data in your project.  Since you control what data you send to Cooladata and what is stored, you are responsible for defining what is personal information and what is not.
If some properties in your project contain personal user information, you might want to hash that information based on a certain condition (for instance if a user asked to be anonymized or if that user is based in the EU). Cooladata now enables you to anonymize data based on project wide conditions – talk to your CSM to set this up.
Notice that we automatically collect IPs so if you need the IPs we have collected to be hashed before they are stored you are responsible to use the above mentioned functionality to set this up.  Hashing IP’s will not affect the geolocation enrichment we provide out-of-the box.

Update users API

To exercise the GDPR data subject “right of rectification”, Cooladata enables updating user scope properties with a new User Update API. This API can be used  to update user information without having to send events especially for this purpose. This can be used for a bulk update of user information, or for ongoing updates.

Since Cooladata allows you to match multiple identities to the same user, we’ve also introduced a new User Identities Matching API for adding more identities to the same user.

For more information on the GDPR and Cooladata please see The Cooladata Guidelines for the GDPR Preparations 

User Profile

The new User profile feature allows you to keep a list of all your project users and their attributes, which include behavioral stats such as the user’s frequency or number of visits to your app, as well as customizable user attributes such as total deposit amount or last viewed item. In addition, all user scope properties last values are added to the user profile automatically. This will allow you to easily create user segments as well as deepen your understanding of your user behavior. Questions like “what is my average user LTV?” or “are users who signed up in May 2017 stickier than users who signed up on 2018?” are now so much easier to answer.

Notice that User Profile is an add-on feature and once configured, is only open for project ADMINs. Please contact your CSM or support@cooladata.com to set it up.

Feature Improvements

Query API Limitation Increase

Our query API now allows you to query your raw-data and extract even more data with one request! Our new limit is 5M rows!

User create date corrections

As you know, Cooladata supports sending historical events in bulk uploads. The automatically added property sys_user_create_date keeps the first timestamp of each one of your users. Up until now, it kept the server time of the first event received for each of your users, which caused issues in historical event uploads.

As of the beginning of June sys_user_create_date shows the event_time_ts (original timestamp) of the first event received for each user. If our ETL ever detects an older time, we will update this field to contain the earlier time.

Facebook Ads Integration account selection

Cooladata provides a predefined connection that enables you to upload Facebook Ads data into your project in order to integrate it with your behavioral tracking data. This enables you to measure your social media return on investment and get the most out of your social media campaigns.

 Up until now, we’ve automatically connected all your Facebook Ads accounts to the Facebook user you have signed in with. From now on, you can select the accounts you want to integrate into Cooladata in the “connected accounts section” here:

iOS Version 2.3

The SDK version is now compiled in Apple BITCODE. See our Github for more details.

Print Friendly, PDF & Email