Setup

CoolaData provides an integration option that enables you to upload the content of any CSV or JSON file from a Google Cloud Storage bucket into your CoolaData project. This type of integration is typically used to enrich your CoolaData events with your own data.

CoolaData creates a dedicated bucket for you in Google Cloud Storage and then automatically uploads the file name and file type that you specify every hour.

To upload content into CoolaData from a Google Cloud Storage Bucket:

  1. Contact your CoolaData customer success representative or write to support@CoolaData.com. Ask them to create a Google Cloud Storage bucket for you into which you can dump your CSV and JSON files to be integrated into CoolaData.
  2. Upload your files into this Google Cloud Storage bucket. For example, by using the UPLOAD FILES button in the Google Cloud Platform user interface. You could also use the gsutil tool in order to automate this process. See the following link for some useful commands: https://cloud.google.com/storage/docs/gsutil
    7-14
  3. Follow the instructions in Integrating with Predefined Data Sources and then select Google Cloud Storage:
    7-16
  4. Fill in the following:
    • Integration Name – The name of this integration.
    • Load data only once – Check this option to specify that CoolaData only loads the data from your CoolaData bucket once. Otherwise, data is loaded every hour, if available.
    • Upload Data Format – Select CSV or JSON to specify the format of the files to be uploaded from your CoolaData bucket. JSON files must be flat (not nested) newline – delimited.
    • File Name – The file name to be uploaded. Make sure to add the file type ending, such as csv/json.
    • Use File Name as Table Name – Check this box to specify that the name of the table that is created in CoolaData is the same as the File Name (described above). This is the table name to be used in the queries that you will perform on the uploaded data. If you choose this option, then make sure that the file name is a valid table name.
    • Table Name – If you did not select the option above, then enter the name of the table to be created in CoolaData to contain the data that is uploaded. Like all table names, it is case sensitive and cannot include spaces or special characters.
    • Append Date to the Table Name – Appends the date when the table is created to the table name. A new table partition is created for each date. Its format is TableName_YYYYMMDD. Selecting this feature enables you to use Google’s BigQuery Data Partitioning feature. You may contact your CoolaData’s customer success representative to hear more about this feature.
    • File Scheme – Define the scheme of the columns of the table to be uploaded by defining the name and data type of each column. Click the Add + button to add each new column. The following data types are supported – string, integer, float, Boolean and timestamp. For example, name:STRING, id:INTEGER, birthdate:TIMESTAMP.
    • In the Insert Strategy field, select either –
      • Append New data is added to the table each time data is uploaded.
        – OR –
      • Replace The table is overwritten each time the data is uploaded.
    • In the Emails to Notify If the Failure field, type in the emails to whom to send integration upload and failure notifications.
    • [Optional] You can use the Google Project ID and Google Dataset fields to define that CoolaData uploads data from your CoolaData bucket into your own Google project instead of into your CoolaData project. Contact your CoolaData customer success representative for more information.
  5. Click the Save button.

Each hour CoolaData will then integrate the files that are dropped into this bucket into your project. The first integration process should take place within a few minutes.
After CoolaData has integrated the files into your CoolaData project, the CoolaData Google bucket is emptied. The files are moved to a subfolder named Uploaded in the bucket and a timestamp is appended to the beginning of the file name. The syntax is – YYYYMMDD_filname.filetype. For example, 20160823_stats.csv.

The Status column of the Integrations list changes to show Data Received.
If the integration process fails for any reason, then the file is moved to a subfolder in the bucket named Failed. An email is sent to the specified recipient(s) (described above) alerting them regarding the integration failure. The same filename convention would be applied in the Failed folder as well.

Data can now be queried using the following syntax:

 

Date Format

Define the Date Format to recognize and convert timestamps columns so they can be used as such in Cooladata. All JAVA standard time formats are supported. The following characters can be used – 1-3 chars represents the abbreviated form, if one exists; 4+ chars represents the full form. The date syntax is case sensitive, so (for example) MM represents month and mm represents seconds.

Letter Date or Time Component Presentation Examples
G Era designator Text AD
y Year Year 1996; 96
M Month in year Month July; Jul; 07
w Week in year Number 27
W Week in month Number 2
D Day in year Number 189
d Day in month Number 10
F Day of week in month Number 2
E Day in week Text Tuesday; Tue
a Am/pm marker Text PM
H Hour in day (0-23) Number 0
k Hour in day (1-24) Number 24
K Hour in am/pm (0-11) Number 0
h Hour in am/pm (1-12) Number 12
m Minute in hour Number 30
s Second in minute Number 55
S Millisecond Number 978
z Time zone General time zone Pacific Standard Time; PST; GMT-08:00
Z Time zone RFC 822 time zone -800

Examples:

Date Format
04-18-17 MM-dd-yy
18-04-2017 dd-MM-yyyy
18/04/2017 19:36:55 dd/MM/yyyy HH:mm:ss
18/04/2017 07:36:55 dd/MM/yyyy hh:mm aa
Tue 04/18/2017 EEE MM/dd/yyyy
Print Friendly, PDF & Email