> ## Documentation Index
> Fetch the complete documentation index at: https://www.propeldata.com/llms.txt
> Use this file to discover all available pages before exploring further.

# 2024 Changelog

> New features, fixes, and improvements.

<div />

<Update label="December">
  ## 1️⃣ Amazon Data Firehose ingestion

  Ingest data from Amazon Firehose streams into Propel's Serverless ClickHouse with zero configuration.

  * Zero-code setup with just a few clicks
  * Real-time ingestion without batch jobs
  * Automatic JSON flattening and column mapping
  * Built-in authentication for Firehose integration

  📢 [Announcement & demo](https://x.com/PropelDataCloud/status/1863609202213781623)

  📚 [Documentation](/docs/ingestion/amazon-data-firehose/overview)

  ## 2️⃣ Time Range Picker UI component

  A customizable React component for selecting dashboard time ranges, with seamless Propel Query API integration.

  * Relative and absolute time ranges
  * Date-to-now selections
  * Custom time period options
  * Themeable UI

  📢 [Announcement & demo](https://x.com/PropelDataCloud/status/1863968975467741427)

  📚 [Documentation](/docs/embeddable-ui/components/time-range-picker)

  ## 3️⃣ Segment event ingestion

  Stream Segment events directly to Propel for real-time analytics.

  * Real-time event processing
  * Automatic JSON property flattening
  * Batch and individual event support
  * Real-time transformations

  📢 [Announcement & demo](https://x.com/PropelDataCloud/status/1864340450917929396)

  📚 [Documentation](/docs/ingestion/twilio-segment)

  ## 4️⃣ Data Grid UI component

  The Data Grid is a React component for easily visualizing data in a table format.

  Features you no longer need to build:

  * Built-in pagination
  * Ordering and filtering
  * Automatic column name formatting
  * Built-in drawer for displaying row or JSON values
  * Fully themable and customizable

  📢 [Announcement & demo](https://x.com/PropelDataCloud/status/1864693769716814226)

  📚 [Documentation](/docs/embeddable-ui/components/data-grid)

  ## 5️⃣ DynamoDB → Propel's ClickHouse

  DynamoDB excels as a transactional database, but it's not ideal for analytics involving aggregations and GROUP BY queries.

  That's where Propel's DynamoDB connector shines. It allows you to replicate DynamoDB data in real time to Propel's Serverless ClickHouse, enabling advanced analytics that are both fast and cost-effective.

  Key features:

  * Seamless integration with DynamoDB streams
  * Real-time replication of inserts, updates, and deletes
  * No infrastructure management needed
  * No need to pay for expensive ETL tools ❤️

  📢 [Announcement & demo](https://x.com/PropelDataCloud/status/1865159001882923445)

  📚 [Documentation](/docs/ingestion/amazon-dynamodb/overview)
</Update>

<Update label="November">
  ## Improvements and fixes

  ### Management API

  * Added CREATE, MODIFY and GET Environment operations to the Management API with new ENVIRONMENT\_ADMIN scope
  * Added Delete Environment mutation
  * Improved hardening of the Copy Job functionality

  ### Serverless ClickHouse

  * Added support for SHOW commands including:
    * `SHOW DATABASES`
    * `SHOW TABLES`
    * `SHOW COLUMNS`
    * `SHOW INDEXES`
    * `SHOW CREATE DATABASE`
    * `SHOW CREATE TABLE`
    * `SHOW FUNCTIONS`
    * `SHOW ENGINES`

  * Added support for system tables including:
    * `system.databases`
    * `system.tables`
    * `system.columns`
    * `system.aggregate_function_combinators`
    * `system.formats`
    * `system.functions`
    * `system.licenses`
    * `system.settings`
    * `system.settings_changed`
    * `system.table_engines`
    * `system.table_functions`
    * `system.time_zones`

  * Improved PopSQL compatibility:
    * Switch to `date_time_output_format = 'simple'` when Faraday user agent is detected
    * Added support for HTTP GET requests with queries in body

  * Enhanced timezone handling:
    * Added timezone parameters to Propel functions
    * Improved timezone management for Time Series API date/time boundaries

  * Fixed bug where tables were added without "propel" database qualifier

  * Changed `currentDatabase()` to return "propel"

  * Added support for database query parameter and x-clickhouse-database header

  * Added support for `wait_end_of_query` parameter used by Python clients

  * Improved authentication handling for missing X-ClickHouse-Key header

  * Added support for client\_protocol\_version in ClickHouse HTTP endpoint

  ### Console

  * Introduced new DataGrid component for Preview Data and SQL console
  * Moved ClickHouse Data Pools to Federated tables section
</Update>

<Update label="October">
  ## 1️⃣ ClickHouse SQL interface

  We've launched a native ClickHouse SQL HTTP interface that lets you query Propel's Serverless ClickHouse directly. You can use any standard [ClickHouse library or SDK](/docs/connect#clickhouse-clients-and-sdks) to run [`SELECT`](/docs/sql-reference/statements/select) and [`SHOW`](/docs/sql-reference/statements/show) queries against your data.

  To get started, use the following connection details:

  | Parameter | Value                                         |
  | --------- | --------------------------------------------- |
  | Host      | `https://clickhouse.us-east-2.propeldata.com` |
  | Port      | 8443                                          |
  | Database  | propel                                        |
  | user      | Your Propel Application ID                    |
  | password  | Your Propel Application secret                |

  Here's an example of querying data using the ClickHouse HTTP interface with curl:

  ```bash theme={"system"}
  echo 'SELECT 1' | \
  curl https://clickhouse.us-east-2.propeldata.com:8443 \
    --user "$APPLICATION_ID:$APPLICATION_SECRET" \
    --data-binary @-
  ```

  Learn more about the [ClickHouse SQL interface](/docs/connect#connection-details).

  ## 2️⃣ New docs

  We've completely revamped our documentation to make it easier for developers to get started and be successful with Propel.

  <Frame>
    <img src="https://mintcdn.com/propeldocs/hcO21NZvCQh4ySdA/images/changelog/new_docs.jpeg?fit=max&auto=format&n=hcO21NZvCQh4ySdA&q=85&s=239a085731fa2b5fe4d7758f04ddba9b" alt="A screenshot of Propel's new docs" width="1920" height="975" data-path="images/changelog/new_docs.jpeg" />
  </Frame>

  The new docs include the following sections:

  * [**Serverless ClickHouse**](/docs/): [Connection details](/docs/connect), [supported SQL syntax](/docs/sql-reference/syntax), [SQL reference](/docs/sql-reference/), [functions](/docs/sql-reference/overview#functions), and everything you need to know to manage, transform and query your data.

  * [**Ingestion**](/docs/ingestion): Guides on ingesting data from a variety of sources to Propel, including Webhooks, events streams, and data warehouses.

  * [**Query APIs**](/docs/query-apis): Guides and examples for using the SQL and Query APIs for common analytics use cases and data visualizations.

  * [**Embeddable UIs**](/docs/embeddable-ui): Guides and documentation for embedding Propel's visualization components into your applications.

  Explore the [new docs](/docs/).

  ## 3️⃣ Server-side authentication

  You can now authenticate server-side applications using HTTP Basic Authentication with your Application ID and secret.

  ```bash {2} theme={"system"}
  curl -X POST https://api.us-east-2.propeldata.com/graphql \
  -u $APPLICATION_ID:$APPLICATION_SECRET \
  -H "Content-Type: application/json" \
  -d '{"query": ...'
  ```

  Learn more about [server-side authentication](/docs/query-apis/authentication#server-side-authentication).

  ## 4️⃣ New GroupBy UI component

  The [`<GroupBy>`](/docs/embeddable-ui/components/group-by) component is a dropdown menu that allows users to dynamically change the grouping of data for multidimensional analysis.

  <Frame>
    <img src="https://mintcdn.com/propeldocs/hcO21NZvCQh4ySdA/images/changelog/2024-11-12-groupby.png?fit=max&auto=format&n=hcO21NZvCQh4ySdA&q=85&s=e1bd51c0921e2ea8634ac49e81085429" alt="A screenshot of the GroupBy component" width="330" height="338" data-path="images/changelog/2024-11-12-groupby.png" />
  </Frame>

  Learn more about the [`<GroupBy>` component](/docs/embeddable-ui/components/group-by).

  ## Improvements and fixes

  ### Serverless ClickHouse

  * Added support for authentication using `X-ClickHouse-User` and `X-ClickHouse-Key` headers.
  * Fixed issues with [Materialized Views](/docs/materialized-views) and Copy Jobs that contain common table expressions (CTEs).

  ### Ingestion

  * Fixed an issue that ocassionally prevented [Webhook syncs](/docs/ingestion/webhooks) from progressing beyond the `PENDING` state.

  ### Console

  * Fixed issue with GitHub login authentication.
  * Added new [Console](https://console.propeldata.com) start page and insights section.
  * For SSO-enabled accounts, the "Invite Users" button is now disabled and displays a message explaining that user management must be done through the SSO provider.
  * Added ability to modify the default timestamp column in Data Pools.
  * Added SQL dialect selector to the SQL Console, allowing users to switch between ClickHouse and PostgreSQL syntax

  ### API

  * Exposed the SQL dialect in the [SQL API](/docs/query-apis/sql-api).
  * Fixed a case where the [Metric Report API](/docs/query-apis/metric-report) was not returning correct results.

  ### Embeddable UI

  * Fixed type definitions for `chartConfigProps` in the [`<TimeSeries>`](/docs/embeddable-ui/components/time-series) component.
  * Removed unintended Storybook environment variable exports.
  * Added `groupBy` functionality to [`<TimeSeries>`](/docs/embeddable-ui/components/time-series) and [`<FilterProvider>`](/docs/embeddable-ui/components/filter-provider) components for dynamic data grouping.
  * Standardized border radius styling across all components.
  * Updated [`<Leaderboard>`](/docs/embeddable-ui/components/leaderboard) component to properly sync with time range from [`<FilterProvider>`](/docs/embeddable-ui/components/filter-provider).
  * Corrected type definition for `legendPosition` prop in [`<PieChart>`](/docs/embeddable-ui/components/pie-chart) component.
  * Added customizable `accentColors` prop to [`<Leaderboard>`](/docs/embeddable-ui/components/leaderboard)  component for visual styling.
</Update>

<Update label="September">
  ## Improvements

  ### Ingestion

  * Improved Webhook Data Pool performance: Increased request per second (RPS) and concurrent request capacity 4x, significantly reducing 429 responses.
  * The Webhook Data Pool now accepts payloads up to 1 MiB.

  ### Console

  * Enhanced account security: Users can now set up Multi-Factor Authentication (MFA) for their accounts.
  * Improved user experience: The last selected Propeller for the Data Pool’s Preview Data will be remembered and displayed for future sessions.
  * Updated console for deprecated API Scope: Applications without prior access to `DATA_POOL_READ`, `DATA_POOL_STATS`, `METRIC_READ`, and `METRIC_STATS` scopes will no longer be able to set these scopes as they are now deprecated

  ### API

  * Deprecated `DATA_POOL_READ`, `DATA_POOL_STATS`, `METRIC_READ`, and `METRIC_STATS` API scopes.
  * Expanded `DATA_POOL_QUERY`, `METRIC_QUERY` API scopes to allow seeing the schema for the Data Pools and columns they have access to.
  * Improved the resiliency of copy jobs for large tables, reducing errors and preventing jobs from getting stuck. Enhanced error messages to provide more informative feedback when jobs fail.
  * Propellers no longer have a limit on maximum bytes to read making them more powerful.
  * Materialize View creation now tests the SQL query prior to creating the Materialized View. This protects developers from creating Materialized Views with incorrect SQL. I handles the case when queries take a long time for large datasets gracefully.

  ### SQL

  * Enhanced JOIN clause functionality:
    * Added support for multiple column expressions using `AND` and `OR` operators
    * Expanded operator support beyond `=` to include `>`, `<`, `LIKE`, and `IN` among others.
  * Increased query result size limit from 500KB to 10MB, allowing for larger data retrieval
  * New ClickHouse functions for enhanced string manipulation and searching:
    * Position and Search functions like:- `position`, `locate`, `positionCaseInsensitive`, `positionUTF8`, `positionCaseInsensitiveUTF8`- multiSearch functions for various use cases (e.g., `AllPositions`, `FirstPosition`, `FirstIndex`)
    * Pattern Matching:- `match`, `REGEXP`, `multiMatchAny`, `multiMatchAnyIndex`, `multiMatchAllIndices`- Fuzzy matching: `multiFuzzyMatchAny`, `multiFuzzyMatchAnyIndex`, `multiFuzzyMatchAllIndices`
    * Extraction and Comparison:- `extract`, `extractAll`, `extractAllGroupsHorizontal`, `extractAllGroupsVertical`- `like`, `notLike`, `ilike`, `notILike` functions
    * N-gram and Substring Operations:- `ngramDistance`, `ngramSearch` (with case-sensitive and UTF8 variants)- `countSubstrings`, `countMatches` (with case-insensitive options)
    * Specialized String Functions:- `regexpExtract`, `hasSubsequence`, `hasToken` (with various options for case sensitivity and UTF8)

  ### Embeddable UI (0.11.1)

  * New layout components: `<Container>`, `<Flex>`, `<Grid>` , and `<Card>`.
  * New typography components: `<Text>` and `<Heading>` .
  * New `<Tabs>` component for tab-based layouts with support for Card components as tabs.
  * Added importable theme colors: Users can now import specific colors (e.g., `gray`) from `"@propeldata/ui-kit/colors"` for consistent styling across the application.
  * `<TimeRangePicker>` now integrates seamlessly with `<FilterProvider>` for improved data filtering across components.
  * Added new `<TimeGrainPicker>` component that also integrated seamlessly with the `<FilterProvider>` .
  * Improved React compatibility: All components are now compatible with React Server Components (RSC) and exported as client-side components, enhancing performance and flexibility.
  * Enhanced debugging capabilities: Components now log prop mismatch errors, facilitating easier troubleshooting and development.
  * Added `groupBy` functionality to the `<TimeSeries>` component, enabling data grouping and more flexible visualizations.
  * Enhanced the `<SimpleFilter>` component with clearable functionality. Developers can use the `disableClearable` prop to turn off this feature if needed.
  * Improved color customization: Replaced `accentColor` with `accentColors` to provide more versatile theming options.

  ### Terraform (v1.3.4)

  * Enhanced flexibility in Data Pool configuration: Added support for explicitly setting empty values for `partition_by`, `order_by`, and `primary_key` fields. Users can now use the syntax `[""]` to define these fields as empty when needed. This improvement allows for more precise control over Data Pool settings.

  ## Fixes

  ### Console

  * Resolved issues with Google sign-in for accepting invitations.
  * Fixed a bug that prevented sign-ups for users who had previously registered but not verified their email.
  * Fixed a bug that caused the SQL Console to attempt to send a Propeller for Applications resulting in an error.

  ### API

  * The API now returns a `NOT_FOUND` error when the requested resource doesn't exist. This applies to Materialized Views, Copy Jobs, and Data Pool access policies, improving error handling and user experience.

  ### Embeddable UI

  * Fixed inconsistent border radius across components for improved visual coherence.
  * Aligned `<SimpleFilter>` component styles with select-based components like the `<TimeRangePicker>` for a more uniform user interface.

  ### Terraform (v1.3.4)

  * Fixed an issue where `order_by`, `partition_by`, and `primary_key` columns were being applied in an incorrect order when defining Table Settings during Data Pool creation. We replaced `Set` with `List` to ensure order is preserved in fields where it's critical.
</Update>

<Update label="August">
  ## Improvements

  ### Console

  * Developers can now sign up and sign in using their Google account.
  * Developers can now sign up and sign in using their GitHub account.

  ### SQL

  * Enhanced support for ClickHouse and PostgreSQL array functions.
  * Introduced new ClickHouse SQL functions:
    * `arrayJoin`: Allows for the expansion of arrays into separate rows.
    * `JSONExtractArrayRaw`: Extracts an array from a JSON string.
    * `JSONExtractKeys`: Retrieves keys from a JSON object.
    * `JSONArrayLength`: Determines the length of a JSON array.
  * Added ClickHouse tuple manipulation functions:
    * `tupleNames`: Returns the names of tuple elements.
    * `tupleElement`: Extracts a specific element from a tuple.
  * Added ClickHouse geospatial functions for advanced geographical data processing.
</Update>

<Update label="July">
  ## New Features

  ### Up to 47% price reduction for your queries

  Our most powerful Propellers, the P1\_LARGE and P1\_X\_LARGE, which can read 250 and 500 million rows per second respectively, now have a significantly lower price.

  | Propeller    | Old price          | New price          | % Price drop |
  | ------------ | ------------------ | ------------------ | ------------ |
  | P1\_MEDIUM   | \$0.10 per GB read | \$0.06 per GB read | 40%          |
  | P1\_LARGE    | \$0.12 per GB read | \$0.07 per GB read | 42%          |
  | P1\_X\_LARGE | \$0.15 per GB read | \$0.08 per GB read | 47%          |

  As we gain scale, we are committed to passing those savings to our customers.

  [See new pricing](https://www.propeldata.com/pricing).

  ### User management in Console

  Customers can now invite their team members to their Propel account. This feature enhances collaboration by allowing multiple users to access and work on the same account. Team members can share resources, manage Data Pools, and streamline their workflows within a single, unified account.

  [Log in and invite your teammates.](https://console.propeldata.com/users/)

  ## Fixes

  ### Console

  * SQL Playground will not show an error when the query is missing a `FROM` statement.
  * We fixed an issue where the Snowflake Data Pool creation flow was incorrectly sending `ver` param for MergeTree tables, causing creation to fail.
  * Customers will not see cached tables after running an introspection on any data source.

  ### Ingestion

  * We fixed an issue with new Snowflake Data Pools on ReplacingMergeTree that were not supporting Re-sync functionality, the `FINAL` clause was not being correctly added to the underlying ClickHouse queries, and `_propel_is_deleted` filtering wasn’t working in some cases.

  ### API

  * We fixed an issue when creating and modifying Applications that was not allowing nullable values for the unique name and description fields. Now unique name and description fields are optional as defined in the GraphQL schema.

  ## Improvements

  ### Console

  * The SQL Console and Materialized View creation flow now supports formatting the current query.
  * When creating a Materialized View with an existing Data Pool, developers will only see Data Pools with a compatible schema.
  * Customers will be able to enable/disable access control on Snowflake Data Pool creation.
  * You can now copy Metric name and ID from the Metric list page.

  ### Terraform

  * Developers can now manage (create, delete, update) Applications via Terraform.
</Update>

<Update label="June">
  ## Materialized Views

  We're introducing Materialized Views in Propel’s Serverless ClickHouse as a powerful tool for data transformation. Developers can leverage Materialized Views to reshape, filter, or enrich data with SQL. Materialized Views are persistent query results that update dynamically as the original data changes.

  The key benefit? Data is transformed in real time. No scheduling. No full-refreshes.

  <img src="https://mintcdn.com/propeldocs/G8bGSg66eBpnYaHB/images/docs/2024-06-14-materialized-views-header.svg?fit=max&auto=format&n=G8bGSg66eBpnYaHB&q=85&s=558fa64ad1a0e7fd62e9b07793d2e566" alt="Materialized Views in ClickHouse" width="1186" height="200" data-path="images/docs/2024-06-14-materialized-views-header.svg" />

  [Learn more about Materialized Views](/docs/materialized-views)

  ## Rockset Migration Service

  OpenAI has announced the acquisition of [Rockset](https://rockset.com/blog/openai-acquires-rockset/), and as a result, the Rockset service will cease to operate. For those unfamiliar with Rockset, it was a cloud-hosted real-time analytics database that enabled millisecond-latency queries for aggregations and joins, similar to Propel.

  We are pleased to announce the immediate availability of the [Rockset Migration Service](https://www.propeldata.com/blog/rockset-migration-service). This service is designed to offer a seamless transition for companies from Rockset.

  To get started with the migration process, please schedule a kick-off call with our team [here](https://www.propeldata.com/blog/rockset-migration-service).

  <img src="https://mintcdn.com/propeldocs/7MySejCVmSGgOSJb/images/changelog/rockset-to-propel_thumb.png?fit=max&auto=format&n=7MySejCVmSGgOSJb&q=85&s=72f3fa56d9f0b571614039f4cd9f073f" alt="Propel's Rockset Migration Service" width="593" height="255" data-path="images/changelog/rockset-to-propel_thumb.png" />

  ## Customizable Table Engine and Sorting Key for all Data Pools

  We are thrilled to announce that Propel now supports customizable table engines and sorting keys for all Data Pools. What does this mean? Better query performance, more cost-efficient reads, and support for real-time updates and delete on any Data Pool type.

  Table engines in Propel's Serverless ClickHouse determine how tables store, process, read, and update their data.

  The sorting key is a set of one or more columns that Propel uses to organize the rows within a table. It determines the order of the rows in the table and significantly impacts the query performance. If the rows are sorted well, Propel can efficiently skip over unneeded rows and thus optimize query performance.

  This enhancement provides users with more flexibility and control over their data, allowing them to optimize their data pools for their specific use cases.

  <img src="https://mintcdn.com/propeldocs/YxRDddM1lFWvUiPZ/images/docs/2024-06-06-table_engine_sorting_key-01.png?fit=max&auto=format&n=YxRDddM1lFWvUiPZ&q=85&s=2717d68b24dd6bec4d82ad44d058ea12" alt="Customizable Table Engine and Sorting Key for all Data Pools" width="1184" height="192" data-path="images/docs/2024-06-06-table_engine_sorting_key-01.png" />

  [Learn more about the table engine and sorting key](/docs/guides/table-engine-and-sorting-key)

  ## Fixes

  ### Console

  * The “**Operations**” tab for all Data Pool types.
  * All Data Pool types can now have a customizable Propeller in the “Preview Data” section.

  ## Improvements

  ### Terraform

  * The resource for Kafka Data Pools is now available.
  * The resource for ClickHouse Data Pools is now available.
  * The resource for Materialized View is now available.
  * The Data Pool resource now supports table settings.
  * The Unique ID field is now deprecated in the Data Pool resource.
  * The Tenant ID field is now deprecated in the Data Pool resource.
  * The timestamp field is now Optional in the Data Pool terraform.

  ### API

  * The API ADMIN scope is sufficient for Applications to get and list other Applications, but it does not allow them to fetch other Application secrets. This simplifies Terraforming of Data Pool Access Policies, which previously failed when the Terraform Application had only ADMIN scope.
  * The ClickHouse Data Source API is now public.
  * You can now modify the `timestamp` field in the Data Pool API.
  * The `uniqueId` and `tenantId` fields are now deprecated in the Webhook Connection Settings object
</Update>

<Update label="May">
  ## New Features

  ### Expanded SQL function support

  We have significantly expanded our SQL function support, extending it to a broad range of functions for PostgreSQL and ClickHouse SQL dialects, as well as unique Propel functions. This improvement offers developers greater flexibility and control when querying, transforming, and managing data.

  [Learn more about Propel SQL function support](/docs/sql-reference/overview#functions).

  <img src="https://mintcdn.com/propeldocs/hcO21NZvCQh4ySdA/images/changelog/2024-05-31-sql-functions.png?fit=max&auto=format&n=hcO21NZvCQh4ySdA&q=85&s=eb03a947ea32233cba9a2c57cb6e3746" alt="A screenshot of Propel's SQL function documentation" width="1200" height="929" data-path="images/changelog/2024-05-31-sql-functions.png" />

  ### New Console Navigation

  We’ve rolled out an updated Console navigation. The new menu structure and design organizes the Console into two primary sections: “Data” and “API”. The “Data” section houses all Serverless ClickHouse-related functionalities, and “API” contains all API-related functionalities.

  [Log in to the Console to see the new navigation](https://console.propeldata.com/login).

  <img src="https://mintcdn.com/propeldocs/hcO21NZvCQh4ySdA/images/changelog/2024-05-31-new-navigation.png?fit=max&auto=format&n=hcO21NZvCQh4ySdA&q=85&s=0b4f4db876c9b1fcf7beba5c59c0bb2e" alt="A screenshot of Propel's new Console navigation" width="1200" height="885" data-path="images/changelog/2024-05-31-new-navigation.png" />

  ### GraphQL Schema Explorer

  We introduced a GraphQL Schema Explorer in the Console. Developers can now actively search through the Propel API GraphQL schema, access API endpoints with ease, and directly download the schema from a provided URL.

  [Check out the new GraphQL Schema Explorer in the Console](https://console.propeldata.com/graphql-schema/).

  <img src="https://mintcdn.com/propeldocs/hcO21NZvCQh4ySdA/images/changelog/2024-05-31-graphql-schema.png?fit=max&auto=format&n=hcO21NZvCQh4ySdA&q=85&s=da268ed98000c03cd8e28cd3318dfaa9" alt="A screenshot of Propel's GraphQL Schema Explorer" width="1200" height="905" data-path="images/changelog/2024-05-31-graphql-schema.png" />

  ## Fixes

  ### Ingestion

  * We resolved an issue with the S3 Data Pool that caused a persistent “CONNECTING” state when customers used an “s3://” or “https\://” prefix in the URL for their table path, rather than a relative path within the bucket. Now, if the provided “s3://” or “https\://” URL points to the named bucket, the URL’s path is used as the relative path into the bucket. Invalid table paths provided by the customer will now be synchronously rejected.

  ### Console

  * We fixed the SQL Console to respect the Propeller selection.
  * We fixed the GraphQL Playground to respect the Propeller selection.
  * When creating a Data Pool, we fixed the case when a timestamp is not selected.
  * In the API Playground, we fixed leaderboard dimensions clearing for ClickHouse Data Pools.
  * In the API Playground, we fixed the leaderboard table view width and height.
  * In the API Playground, we fixed the time dimension not pre-populating.
  * In the SQL Console, we fixed result cell wrapping.
  * Data Pools created by Materialized Views now see the "Operations" tab.

  ## Improvements

  ### SQL

  * The SQL interface now properly handles identifier quoting.
  * We now support CASE statements in SQL.

  ### API

  * The query APIs now support SQL-style filters. Everywhere you could provide `filters`, you can now provide `filterSql`. [For example, see the `filterSql` parameter in the counter API](https://www.propeldata.com/docs/api/query/inputs#CounterInput).
  * Data Pool Access Policies can now be defined with the new `filterSql` parameter that supports SQL-style filters.

  ### Console

  * The API Playground will now populate filter values using the Top Values API.
  * The API Playground now includes an “All time” option for the relative time range.
  * The API Playground has a new layout for TimeSeries, Counter, and Leaderboard APIs.
  * The API Playground now supports setting a “Time Dimension” that is different from the Data Pool’s default timestamp.
  * The API Playground will now show the Data Pool selection first.
  * The API Playground will now only show the “Existing Metrics” for the selected Data Pool.
  * The SQL Console has a new improved layout.
  * The SQL Console has new improved SQL syntax validation.
  * The SQL Console will now show long result values in a tooltip when they are trimmed.
  * The Data Pool list page will not break if the sync activity query fails.
  * When creating and updating Amazon S3 credentials, the path and bucket will be trimmed to prevent white spaces from being accidentally entered.
</Update>

<Update label="April">
  ## New features

  ### ClickHouse Data Pool beta

  The ClickHouse Data Pool enables you to read through to your self-hosted ClickHouse or ClickHouse Cloud rather than syncing data to Propel. This allows you to utilize the data in your analytic dashboards, reports, and workflows directly from your ClickHouse instance through the Propel APIs and UI components.

  [Learn more about the ClickHouse Data Pool](/docs/federation/clickhouse/overview).

  <img src="https://mintcdn.com/propeldocs/QOy5SmMMP2AzzqCU/images/docs/clickHouse-architecture-diagram-docs.svg?fit=max&auto=format&n=QOy5SmMMP2AzzqCU&q=85&s=38a148fabf4fefdc4d71c510c0a23b19" alt="Propel's Webhook latency demo." width="1100" height="301" data-path="images/docs/clickHouse-architecture-diagram-docs.svg" />

  ### GraphQL Playground

  The GraphQL Playground enables you to run GraphQL queries directly from the Console, offering a simple way to interact with your data when building applications.

  Key Features:

  * GraphQL schema autocompletion
  * Code examples
  * Access token generator

  [Log in to the Console and click **"Playground"**, then select **"API: GraphQL**".](https://console.propeldata.com/playground/graphql/)

  <img src="https://mintcdn.com/propeldocs/hcO21NZvCQh4ySdA/images/changelog/2024-04-graphql-playground.png?fit=max&auto=format&n=hcO21NZvCQh4ySdA&q=85&s=a05d8165c4ae275e865108c59e78ff0c" alt="Propel's GraphQL Playground" width="1677" height="926" data-path="images/changelog/2024-04-graphql-playground.png" />

  ## Fixes

  * Fixed AND/OR logic bug in SQL.
  * Fixed typo in default Webhook Data pool name.
  * Free trial plan no longer expires incorrectly.
  * Fixed case-insensitivity in Materialized Views.
  * Fix automatic syncing of HTTP Data Sources which contain multiple HTTP Data Pools.
  * Fixed the column `_propel_synced_at` that was incorrectly set for some Webhook Data Pools, resulting in out-of-range values.
  * Fixed columns length validation for Kafka on Data Pool creation.
  * Fixed a race condition while re-assigning Access Policies.

  ## Improvements

  * You can now create Data Pools with ClickHouse types via the API.
  * Implemented improvements to our access token API that reduced/eliminated the HTTP 500 errors that some customers experienced.
  * Added support for AggregatingMergeTree table engine for Data Pools via the API.
  * The Webhook Data Pool now returns HTTP 413 Content Too Large error if the payload exceeds 1,048,320 bytes or has more than 500 events.
  * Add support for read-only filterSql with a SQL representation of the filters.
  * Raised the GraphQL aliases limit to 250.
  * Improved the case insensitivity matching for identifiers in SQL.
  * Added support lists of strings and lists of numbers in Postgres SQL.
  * Added support NOT IN and AT TIME ZONE operators in SQL.
  * Added support column auto aliasing in SQL.
  * Added support for unary expressions in SQL.
  * Added support for CURRENT\_DATE in SQL.
</Update>

<Update label="March">
  ### ⚡️ 10x faster ingestion for the Webhook Data Pool

  The Webhook Data Pool now ingests events 10x faster. We have optimized ingestion so that data is available within single digit seconds.

  [Learn more about the Webhook Data Pool](/docs/ingestion/webhooks/overview).

  <img src="https://mintcdn.com/propeldocs/hcO21NZvCQh4ySdA/images/changelog/2024-03-webhook_latency.gif?s=9111a7ffb6db6854767f4ca101f24b0b" alt="Propel's Webhook latency demo." width="1660" height="900" data-path="images/changelog/2024-03-webhook_latency.gif" />

  ### 🛝 SQL Playground

  With Propel’s new SQL Playground, you can now execute SQL queries directly from the Console. It provides you with an easy way to explore your data when building applications.

  **Key Features:**

  * Code examples:  easily integrate queries into your code. Get code examples for querying data using SQL via the API, cURL, and JavaScript.
  * "Query as" selector: simulate querying as your app. Different apps have unique data query permissions set through Access Policies. The Playground allows you to test queries as a specific app.
  * "Propeller" selector: experiment with query speeds. Test different query speeds to optimize performance and cost for your data application.

  [Log in to the Console and click **"Playground"**, then select **"API: SQL**".](https://console.propeldata.com/playground/sql/)

  <img src="https://mintcdn.com/propeldocs/hcO21NZvCQh4ySdA/images/changelog/2024-03-changelog-sql-playground.png?fit=max&auto=format&n=hcO21NZvCQh4ySdA&q=85&s=5964fb491dc3ae35c8f9a53398d30707" alt="Propel's SQL Playground" width="1670" height="917" data-path="images/changelog/2024-03-changelog-sql-playground.png" />

  ### ✈️ Airbyte destination

  The Airbyte destination lets you synchronize data from over [350+ sources](https://airbyte.com/connectors) to Propel's Serverless ClickHouse infrastructure. It provides an easy way to power your customer-facing analytics and data applications with data from any SaaS application, database, or platform supported by Airbyte.

  [Learn more about the Airbyte destination.](/docs/ingestion/airbyte/overview)

  <img src="https://mintcdn.com/propeldocs/hcO21NZvCQh4ySdA/images/changelog/propel-airbyte-2.png?fit=max&auto=format&n=hcO21NZvCQh4ySdA&q=85&s=752951ba1f97f4bce39f6a6f1adde94e" alt="Propel's Airbyte destination" width="1920" height="760" data-path="images/changelog/propel-airbyte-2.png" />

  ## Fixes

  * Customers will be able to change the URL in their ClickHouse credentials on creation flow in the Console.
  * Customers will not see stale data when modifying credentials for ClickHouse and Kafka in the Console.
  * Signed in customers will be redirected from /login to dashboard.
  * In the GraphQL Playground, customers will be able to clear the variables input and run queries with no variables.
  * Fixed a bug when parsing table names that contain aliases in SQL.
  * Fixed a bug where table aliases where being lowercased in SQL.

  ## Improvements

  * By setting the `disable_partial_success=true` query parameter, you can ensure that, if any individual event in a batch of events fails validation, the entire request will fail. For example: `https://webhooks.us-east-2.propeldata.com/v1/WHK00000000000000000000000000?disable_partial_success=true`
  * Added SUBSTRING function to SQL.
  * Added support for extracting parts from a timestamp in SQL.
  * Raised SQL response size limit to 2 MB.
  * The PostgreSQL interface now supports extended queries.
</Update>

<Update label="February">
  ### Kafka Data Pool

  The new Kafka [Data Pool](/docs/ingestion/overview#understanding-data-pools) lets you ingest real-time streaming data into Propel. It provides an easy way to power real-time dashboards, streaming analytics, and workflows with a low-latency data API on top of your Kafka topics.

  <img src="https://mintcdn.com/propeldocs/9FdNDTkhOu7EgAUd/images/docs/kafka-architecture-diagram-docs.svg?fit=max&auto=format&n=9FdNDTkhOu7EgAUd&q=85&s=0e699706fd29725df1e6f61fa335af83" alt="The architectural overview when connecting Kafka to Propel." width="1100" height="301" data-path="images/docs/kafka-architecture-diagram-docs.svg" />

  [Learn more about the Kafka Data Pool](/docs/ingestion/kafka/overview).

  ### 🆓 New Generous Free Tier

  We are introducing a new, generous free tier! It includes up to \$15 of usage per month, and the best part is, it does not expire.

  <img src="https://mintcdn.com/propeldocs/hcO21NZvCQh4ySdA/images/changelog/2024-03-04-changelog-free-tier.png?fit=max&auto=format&n=hcO21NZvCQh4ySdA&q=85&s=1bb474198400d2de6bc0ed03115d4290" alt="Propel Free Tier" width="170" height="145" data-path="images/changelog/2024-03-04-changelog-free-tier.png" />

  [Sign up and get started today](https://console.propeldata.com/get-started).

  ### ䷰ Schema evolution: Add a column to Data Pool operation

  We are introducing Schema Evolution for Data Pools with the ability to add new columns to your Data Pools. Now, you can add new columns to your Data Pools, allowing you to evolve your data schema as your needs grow and change.

  <img src="https://mintcdn.com/propeldocs/hcO21NZvCQh4ySdA/images/changelog/2024-03-04-changelog-add-column.png?fit=max&auto=format&n=hcO21NZvCQh4ySdA&q=85&s=48fffa514422b9176736027d060d93a9" alt="Propel Schema Evolution" width="1402" height="351" data-path="images/changelog/2024-03-04-changelog-add-column.png" />

  [Learn more about the Add column to Data Pool operation](/docs/ingestion/snowflake/overview#schema-changes).

  ### 🚚 Batch delete and update operation

  The new batch delete operation helps you stay GDPR compliant by providing a straightforward way to permanently delete data from a Data Pool. Meanwhile, the batch update operation helps maintain data integrity and facilitates data backfilling in the event of schema changes. Both operations can be done via the Console or API.

  <img src="https://mintcdn.com/propeldocs/hcO21NZvCQh4ySdA/images/changelog/2024-03-04-changelog-batch-delete.png?fit=max&auto=format&n=hcO21NZvCQh4ySdA&q=85&s=44208b89715a6fa8f474ba42258922dc" alt="Propel batch delete" width="1399" height="500" data-path="images/changelog/2024-03-04-changelog-batch-delete.png" />

  Learn more about batch [updates](/docs/updating-data#batch-updates) and [deletes](/docs/deleting-data#batch-deletes).

  ### 🪵 React UI Kit logging controls

  The Propel UI Kit now features logging capabilities for faster development and clean logging in production. By default, all errors are logged to the browser's console. This behavior can be customized using the LogProvider component. The LogProvider uses React's context mechanism to propagate log settings to nested components, allowing for specific component logging. Available log levels include "error", "warn", "info", or "debug".

  <img src="https://mintcdn.com/propeldocs/hcO21NZvCQh4ySdA/images/changelog/2024-03-04-changelog-ui-kit-logging.png?fit=max&auto=format&n=hcO21NZvCQh4ySdA&q=85&s=4f6abca9bb6c1cf07441f303b73c09c1" alt="Propel UI Kit Logging" width="598" height="208" data-path="images/changelog/2024-03-04-changelog-ui-kit-logging.png" />

  [Learn more about the React UI Kit’s logging controls](https://storybook.propeldata.com/?path=/docs/customization-logging--overview).

  ### Fivetran preview

  The Fivetran destination lets you synchronize data from over [400 sources](https://www.fivetran.com/connectors) to Propel's Serverless ClickHouse infrastructure.

  <img src="https://mintcdn.com/propeldocs/hcO21NZvCQh4ySdA/images/changelog/2024-03-04-changelog-fivetran.png?fit=max&auto=format&n=hcO21NZvCQh4ySdA&q=85&s=d4fd09aff4cb43188e9edf79acc07dc4" alt="Propel Fivetran" width="512" height="256" data-path="images/changelog/2024-03-04-changelog-fivetran.png" />

  [Learn more about our Fivetran destination](/docs/ingestion/fivetran/overview).

  ### Bring your own ClickHouse preview

  The ClickHouse Data Pool reads through to your self-hosted ClickHouse or ClickHouse Cloud rather than syncing data to Propel.

  <img src="https://mintcdn.com/propeldocs/QOy5SmMMP2AzzqCU/images/docs/clickHouse-architecture-diagram-docs.svg?fit=max&auto=format&n=QOy5SmMMP2AzzqCU&q=85&s=38a148fabf4fefdc4d71c510c0a23b19" alt="ClickHouse with Propel Architecture" width="1100" height="301" data-path="images/docs/clickHouse-architecture-diagram-docs.svg" />

  [Learn more about the ClickHouse “read-through” Data Pool](/docs/federation/clickhouse/overview).

  ## Fixes

  * Fix timezone argument on `toStartOfWeek`, `toStartOfMonth`, and `toStartOfYear` SQL functions.
  * Fixed login loop for accounts with Okta integration.
  * Fixed environments dropdown in Console in the new Data Pool page.
  * Fixed the **Preview Data** section of the Console where customers can now change time range and page size when the query results in a time out or an error.

  ## Improvements

  * Add support for timestamps without timezones
  * Support `NOW()` and `CURRENT_DATE` functions in SQL.
  * Support `INTERVAL` in SQL.
  * An alternative `timestamp` can be supplied to `TimeRangeInput` when querying.
  * Customers can view TableSettings (`engine`, `partitionBy`, `primaryKey`, and `orderBy`) for their Data Pools via the API.
  * Allow creating Data Pools (including Webhook Data Pools) without `timestamp` via the API.
  * Allow setting TableSettings (`engine`, `partitionBy`, `primaryKey`, and `orderBy`) when creating a Data Pool via the API.
  * In the Console, customers can have a different environment in multiple tabs without losing the last selected state.
  * In the Console, Customers will see the `processedRecords` instead of `newRecords` in the Processed Records column for the Syncs table.
  * Customers can now change the sort and timestamp column in the Console in the **Preview Data** tab.
</Update>

<Update label="January">
  ### 🔎 SQL API

  You can now query any Data Pool using SQL over the GraphQL API. Need to join, group by, or perform complex queries? No problem. Propel's SQL supports PostgreSQL syntax, including joins, unions, and common table expressions for more complex queries. The SQL API allows you to query your data however you'd like, and Propel's multi-tenant access policies ensure that customers can only query their own data.

  <img src="https://mintcdn.com/propeldocs/hcO21NZvCQh4ySdA/images/changelog/2024-01-31-changelog-sql.png?fit=max&auto=format&n=hcO21NZvCQh4ySdA&q=85&s=d91b4cef4f1c45bd52bf3ccd9709595d" alt="Propel's SQL API" width="1592" height="496" data-path="images/changelog/2024-01-31-changelog-sql.png" />

  [Learn more about the SQL API](/docs/query-apis/sql-api).

  ### 🐘 PostgreSQL-compatible SQL interface

  You can now connect any BI tool or PostgreSQL client to Propel. Essentially, Propel mimics a PostgreSQL instance, providing a seamless connection to a variety of tools or client applications.

  For SaaS applications, this simplifies the process of providing a customer-facing SQL interface for custom reporting and data sharing.

  <img src="https://mintcdn.com/propeldocs/hcO21NZvCQh4ySdA/images/changelog/2024-01-31-changelog-sql-interface.png?fit=max&auto=format&n=hcO21NZvCQh4ySdA&q=85&s=9d6884109170b4ca335e1765ab12d69a" alt="Propel's SQL interface" width="678" height="196" data-path="images/changelog/2024-01-31-changelog-sql-interface.png" />

  [Learn more about the SQL interface](/docs/connect).

  ### 🗄️ Data Grid API

  The new Data Grid API efficiently retrieves individual records from a Data Pool, with the added convenience of built-in pagination, filtering, and sorting. It's perfect for displaying data in a table format, making it ideal for data tables with individual events, orders, requests, or log messages.

  <img src="https://mintcdn.com/propeldocs/hcO21NZvCQh4ySdA/images/changelog/2024-01-31-changelog-datagrid.png?fit=max&auto=format&n=hcO21NZvCQh4ySdA&q=85&s=5a2f4c4aca7afcb6b6dd979f69da7c73" alt="Propel's Data Grid API use cases" width="1446" height="528" data-path="images/changelog/2024-01-31-changelog-datagrid.png" />

  [Learn more about the Data Grid API](/docs/query-apis/data-grid)

  ### 📌 Records by ID API

  The new Records by ID API is optimized for quick, unique ID lookups. It returns the records corresponding to the given IDs. This API can present detailed record information in a data table or record detail page.

  [Learn more about the Records by ID API](/docs/query-apis/overview).

  ### 🧮 Top Values API

  The new Top Values API returns the most common values in a specified column ordered by frequency. The Top Values API can populate UI filters, prompt available values to AI agents, or showcase trending values within a column.

  <img src="https://mintcdn.com/propeldocs/hcO21NZvCQh4ySdA/images/changelog/2024-01-31-changelog-top-values.png?fit=max&auto=format&n=hcO21NZvCQh4ySdA&q=85&s=25d09dec0229f5ea83dffc031d72d0b1" alt="Propel's Top Values API use cases" width="1482" height="398" data-path="images/changelog/2024-01-31-changelog-top-values.png" />

  ### 💚💙💜 UI Kit themes

  You can now control the look and feel of all your UI components in one theme. The theme of the UI Kit determines all essential visual elements, including the colors of components, the depth of shadows, and the overall light or dark appearance of the interface. We provide light and dark themes out of the box and the ability to customize your own theme.

  <img src="https://mintcdn.com/propeldocs/hcO21NZvCQh4ySdA/images/changelog/2024-01-31-changelog-ui-kit-themes.png?fit=max&auto=format&n=hcO21NZvCQh4ySdA&q=85&s=bfaa6b1b90abdae16b27e30ce17166ce" alt="Propel's UI Kit Themes" width="1181" height="431" data-path="images/changelog/2024-01-31-changelog-ui-kit-themes.png" />

  [Learn more about themes in the UI Kit](https://storybook.propeldata.com/?path=/docs/customization-theming--overview).

  ### 🔓 UI Kit Access Token Provider

  You can now easily fetch and refresh API access tokens from the frontend. The new  `AccessTokenProvider` component allows you to provide a function that fetches an access token from your backend. Using this function, the provider will serve the fetched access token to all its child components and automatically refresh the token when it expires.

  <img src="https://mintcdn.com/propeldocs/hcO21NZvCQh4ySdA/images/changelog/2024-01-31-changelog-ui-kit-access-token.png?fit=max&auto=format&n=hcO21NZvCQh4ySdA&q=85&s=6fd069c00822d44bcbbc7458adcc3b29" alt="Code example of Propel's UI Kit Access Token provider" width="1440" height="518" data-path="images/changelog/2024-01-31-changelog-ui-kit-access-token.png" />

  [Learn more about the Access Token Provider](https://storybook.propeldata.com/?path=/docs/getting-started-authentication--overview).

  ### ⏳ UI Kit Filter component

  The new Filter component simplifies the process of adding filters to your dashboards. It uses [Propel's Top Values API](https://www.propeldata.com/docs/query-your-data/top-values) to fill the dropdown list with unique values from a specific column, arranged by their frequency.

  <img src="https://mintcdn.com/propeldocs/hcO21NZvCQh4ySdA/images/changelog/2024-01-31-changelog-ui-kit-filter.png?fit=max&auto=format&n=hcO21NZvCQh4ySdA&q=85&s=ddab6334ae5c506db4809f75eabd4725" alt="Example of Propel's UI Kit Filter component" width="958" height="465" data-path="images/changelog/2024-01-31-changelog-ui-kit-filter.png" />

  [Learn more about the filter component](https://storybook.propeldata.com/?path=/docs/components-simplefilter--overview).

  ### 🍰 UI Kit Pie Chart component

  The PieChart component is designed to create pie or doughnut charts using the [Leaderboard API](/docs/query-apis/leaderboard).

  <img src="https://mintcdn.com/propeldocs/hcO21NZvCQh4ySdA/images/changelog/2024-01-31-changelog-ui-kit-pie-chart.png?fit=max&auto=format&n=hcO21NZvCQh4ySdA&q=85&s=cafe8088eba4e77d88b063ec679e428c" alt="Example of Propel's UI Kit Pie Chart component" width="560" height="296" data-path="images/changelog/2024-01-31-changelog-ui-kit-pie-chart.png" />

  [Learn more about the Pie Chart component](https://storybook.propeldata.com/?path=/docs/components-piechart--overview).

  ### 🪝 UI Kit Query Hooks

  Propel's UI Kit provides prebuilt React components for querying data from Propel's GraphQL API. These components can be used to query data for custom visualizations or to build with third-party libraries such as [D3.js](https://d3js.org/), [Recharts](https://recharts.org/en-US/), [Nivo](https://nivo.rocks/), or [Chart.js](https://www.chartjs.org/).

  <img src="https://mintcdn.com/propeldocs/hcO21NZvCQh4ySdA/images/changelog/2024-01-31-changelog-query-hooks.png?fit=max&auto=format&n=hcO21NZvCQh4ySdA&q=85&s=692977bcc58c1a76cb4bf09dd8f6f60e" alt="Code example of Propel's UI Kit Query Hooks" width="699" height="470" data-path="images/changelog/2024-01-31-changelog-query-hooks.png" />

  [Learn more about the Query Hooks](https://storybook.propeldata.com/?path=/docs/hooks-overview--overview).

  ## Fixes

  * Fix timezone argument on `toStartOfWeek`, `toStartOfMonth`, and `toStartOfYear` SQL functions.
  * Fixed login loop for accounts with Okta integration.

  ## Improvements

  * Support LIKE and NOT LIKE filter operators in SQL and the GraphQL API.
  * Support TO\_TIMESTAMP function in SQL.
  * Support CAST function in SQL.
  * Mark the tenant ID field in the Data Pool as deprecated. No longer needed with the new Access Policies.
  * Support WITH statements in the SQL API.
  * Support UNION statements in the SQL API.
  * New `data_pool:read` scope to list Data Pools and their schemas.
  * We made `timeRange` optional in GraphQL API.
  * The dimensions stats API has now been deprecated and replaced with the Top Values API.
</Update>