shorturl-analytics/Date Format Handling for ClickHouse Events API.md

Date Format Handling for ClickHouse Events API

Problem Description

The event tracking API was experiencing issues with date format compatibility when inserting records into the ClickHouse database. ClickHouse has specific requirements for datetime formats, particularly for its DateTime64 type fields, which weren't being properly addressed in the original implementation.

Root Cause

• JavaScript's default date serialization (toISOString()) produces formats like 2023-08-24T12:34:56.789Z, which include T as a separator and Z as the UTC timezone indicator
• ClickHouse prefers datetime values in the format YYYY-MM-DD HH:MM:SS.SSS for seamless parsing
• The mismatch between these formats was causing insertion errors in the database

Solution Implemented

We created a formatDateTime utility function that properly formats JavaScript Date objects for ClickHouse compatibility:

const formatDateTime = (date: Date) => {
  return date.toISOString().replace('T', ' ').replace('Z', '');
};

This function:

1. Takes a JavaScript Date object as input
2. Converts it to ISO format string
3. Replaces the 'T' separator with a space
4. Removes the trailing 'Z' UTC indicator

The solution was applied to all date fields in the event payload:

• event_time
• link_created_at
• link_expires_at

Additional Improvements

• We standardized date handling by using a consistent currentTime variable
• Added type checking for JSON fields to ensure proper serialization
• Improved error handling for date parsing failures

Best Practices for ClickHouse Date Handling

1. Always format dates as YYYY-MM-DD HH:MM:SS.SSS when inserting into ClickHouse
2. Use consistent date handling utilities across your application
3. Consider timezone handling explicitly when needed
4. For query parameters, use ClickHouse's parseDateTimeBestEffort function when possible
5. Test with various date formats and edge cases to ensure robustness