Webhooks setup guide

Ingesting webhook events into Propel

This guide will show you how to send JSON events to Propel over HTTPS from any application or SaaS service.

You will learn how to:

1. Create a Webhook Data Pool
2. Send events to the Webhook Data Pool
3. View your events in the Data Pool

---

Requirements

• You have a Propel account.

---

Step 1: Create a Webhook Data Pool

Console

1. Navigate to Data Pools

   In the Console, click on “Data Pools” in the left-hand menu. Click on “Create Data Pool” and select “Webhook”.

   
2. Define the schema

   The default schema contains two columns:

   Column	Type	Description
   _propel_received_at	TIMESTAMP	The timestamp when the event was collected in UTC.
   _propel_payload	JSON	The JSON Payload of the event.

   In the Payload section on the right-hand side, you can enter or paste a sample JSON event. This feature allows you to:

   1. Visualize the structure of your incoming data
   2. Automatically extract top-level and nested JSON keys
   3. Create specific columns for these extracted keys

   

   By providing a sample event, you can easily customize your Data Pool’s schema to match your data structure.

   For this guide, we’ll use the TacoSoft sample data:

   {
     "customer_id": 5,
     "order_id": 34,
     "store_id": 4445,
     "order_details": {
       "taco_count": 5,
       "total_price": 30.4,
       "checkout_time": "2022-08-01T09:03:32Z"
     },
     "created_at": "2022-08-01T09:02:15Z"
   }
   

   After adding the sample JSON, click on “Extract nested properties” to create columns representing the nested JSON keys.

   Set created_at as your default timestamp. Click “Next”.

   If a required field is missing from the sample event, Propel will reject the event with an HTTP 400 Bad Request error.

3. Configure Authentication

   Configure authentication for your webhook URL:

   • To enable HTTP basic authentication, specify a username and password.
   • For initial testing, you can leave these fields blank.
   • You can always edit these settings later.

   After configuring (or skipping) authentication, click “Next” to proceed.

   
4. Configure data type and settings

   Select whether your data is “Append-only” or “Mutable data”.

   To learn more, read out guide on Selecting table engine and sorting key.

   

   Answer the questions in the wizard to complete the setup.

   

   Confirm your table settings and click “Continue”.

   
5. Set a name and description

   Enter a name and description for your new Data Pool and click “Next”.

   

   After creating the Data Pool, you’ll be provided with a unique HTTP URL. This URL is where you’ll send your JSON data for ingestion.

   

API

For the Webhook Data Pool, creating the Data Source below automatically creates the Data Pool.

mutation {
  createWebhookDataSource(input: {
    uniqueName: "WebhookCredentials"
    description:"My Webhook Credentials"
    connectionSettings: {
      basicAuth: {
        username: "propel-user"
        password: "<SUPER_SECURE_PASSWORD>"
      }
      columns: [
        { name: "created_at", jsonProperty: "created_at", type: TIMESTAMP, nullable: false },
        { name: "customer_id", jsonProperty: "customer_id", type: INT32, nullable: false },
        { name: "order_id", jsonProperty: "order_id", type: INT32, nullable: false },
        { name: "store_id", jsonProperty: "store_id", type: INT32, nullable: false },
        { name: "order_details", jsonProperty: "order_details", type: JSON, nullable: false }
      ]
      accessControlEnabled: true,
      tableSettings: {
        engine: {
          mergeTree: {
            type: MERGE_TREE
          }
        }
        orderBy: ["created_at"],
        partitionBy: ["toYYYYMM(created_at, 'UTC')"]
      }
    }

  }){
    dataSource {
      id
      uniqueName
      status
    }
  }
}

Terraform

For the Webhook Data Pool, creating the Data Source below automatically creates the Data Pool.

resource "propel_data_source" "my_webhook {
  unique_name = "WebhookCredentials"
  description = "My Webhook Credentials"
  type        = "WEBHOOK"
  webhook_connection_settings {
    timestamp = "created_at"
    column {
      name = "created_at"
      type = "TIMESTAMP"
      nullable = false
      json_property = "created_at"
    }
    column {
      name = "customer_id"
      type = "INT32"
      nullable = false
      json_property = "customer_id"
    }
    column {
      name = "order_id"
      type = "INT32"
      nullable = false
      json_property = "order_id"
    }
    column {
      name = "store_id"
      type = "INT32"
      nullable = false
      json_property = "store_id"
    }
    column {
      name = "order_details"
      type = "JSON"
      nullable = false
      json_property = "order_details"
    }

    basic_auth {
      username = "propel-user"
      password = var.http_basic_auth_password
    }
    access_control_enabled = true
    table_settings {
      engine {
        merge_tree {
          type = "MERGE_TREE"
        }
      }
      order_by = ["created_at"]
      partition_by = ["toYYYYMM(created_at, 'UTC')"]
    }
  }
}

variable "http_basic_auth_password" {
  type = string
  sensitive = true
}

---

Step 2: Send events to the Webhook Data Pool

1. Prepare your data

   Create a JSON array of events you want to send to the Webhook Data Pool.

   Here’s an example:

   [
     {
       "customer_id": 5,
       "order_id": 34,
       "store_id": 4445,
       "order_details": {
         "taco_count": 7,
         "total_price": 25.90,
         "checkout_time": "2023-07-31T15:20:10Z"
       },
       "created_at": "2023-07-31T14:50:35Z"
     },
     {
       "customer_id": 8,
       "order_id": 22,
       "store_id": 1199,
       "order_details": {
         "taco_count": 3,
         "total_price": 15.75,
         "checkout_time": "2023-07-31T12:40:21Z"
       },
       "created_at": "2023-07-31T12:30:55Z"
     }
   ]
   
2. Send the data

   Send a POST request to the Webhook Data Pool’s URL using curl or any HTTP client. Here’s an example using curl:

   curl https://webhooks.us-east-2.propeldata.com/v1/WHK... \
   -X POST \
   -H "Content-Type: application/json" \
   -d '[
     {
       "customer_id": 5,
       "order_id": 34,
       "store_id": 4445,
       "order_details": {
         "taco_count": 7,
         "total_price": 25.90,
         "checkout_time": "2023-07-31T15:20:10Z"
       },
       "created_at": "2023-07-31T14:50:35Z"
     },
     {
       "customer_id": 8,
       "order_id": 22,
       "store_id": 1199,
       "order_details": {
         "taco_count": 3,
         "total_price": 15.75,
         "checkout_time": "2023-07-31T12:40:21Z"
       },
       "created_at": "2023-07-31T12:30:55Z"
     }
   ]'
   
3. Check the response

   You should expect a 200 OK with multiple “Event processed successfully” messages in the body of the response, one for each event.

Step 3: View your events in the Data Pool

1. Navigate to your Data Pool

   Click on “Data Pools” in the Console, and then select your Data Pool.

2. View the data

   Head over to the “Preview Data” tab, and you should see the events data as POSTed, reflected in the schema we defined.