Nanosecond precision

Release Overview

This release introduces nanosecond precision support for timestamps, aligning the client library with the recent QuestDB 9.1.0 server update.

Nanosecond Precision Support

Previously, while the API allowed timestamps to be specified in nanoseconds, they were stored on the server with only microsecond precision.
QuestDB 9.1.0 added nanosecond accuracy through the new TIMESTAMP_NS data type, and this library update brings full client-side support for it.

Starting with this release:

• When using protocol version v2 or higher, and timestamps are provided in nanoseconds, they are now recorded on the server with full nanosecond precision, provided the column type is TIMESTAMP_NS.
• If the column does not yet exist and the client sends nanosecond timestamps, the server will automatically create the column as TIMESTAMP_NS.
• This behavior also applies to the designated timestamp column.

For compatibility:

• Protocol version v1 remains unchanged and continues to store timestamps with microsecond precision, ensuring full backward compatibility.

Export SenderOptions class

A maintenance release to fix exports in the library.
SenderOptions was exported only as a type, its methods were not available for use.
Many thanks to @nicochatzi for providing the fix!

This release also exports the createBuffer() and createSender() factory functions, so now we can do this:

    const options: SenderOptions = await SenderOptions.fromConfig('http::addr=localhost:9000');
    const buffer: SenderBuffer = createBuffer(options);
    const transport: SenderTransport = createTransport(options);
    
    buffer.table("test").symbol("sym", "sym1").floatColumn("fp64", 1.123).atNow();
    await transport.send(buffer.toBufferNew());
    await transport.close();

By pooling a number of buffer and transport objects, and implementing our own flushing strategy, potentially can reach better performance than just simply using Sender instances.

Array support, modularized client, migration to typescript, Undici transport

Release overview

Major upgrade of the library with new features, such as ingesting arrays into the database, support for Undici, and bringing a new modularized codebase to enable users to build their own client.

Typescript migration and Undici

The entire codebase have been migrated to Typescript, and tools replaced/updated to pnpm, vitest, bunchee, eslint.
Starting with this release the client switches to Undici as its default HTTP transport.
The above changes were all implemented by @semoal, many thanks again for his huge contribution to the project!

Breaking change: The Undici dependency requires Node.js v20+. This is the minimum version required for this version of the client.

Arrays and protocol extension for binary encoding of 64-bit floating point values

This release also delivers support for the new QuestDB array type, and adds support for sending 64-bit floating point values in binary form. Arrays use the binary protocol extension too.

Example usage:

  const sender = await Sender.fromConfig('http::addr=localhost:9000');

  await sender
    .table('order_book_l2')
    .symbol('symbol', 'BTC-USD')
    .symbol('exchange', 'Coinbase')
    .arrayColumn('bid_prices', [50100.25, 50100.20, 50100.15, 50100.10, 50100.05])
    .arrayColumn('bid_sizes', [0.5, 1.2, 2.1, 0.8, 3.5])
    .arrayColumn('ask_prices', [50100.30, 50100.35, 50100.40, 50100.45, 50100.50])
    .arrayColumn('ask_sizes', [0.6, 1.5, 1.8, 2.2, 4.0])
    .atNow();
  
  await sender.flush();
  await sender.close();

Modularized client

We have also extracted the buffer and transport interfaces from the client, and introduced different implementations.
The different buffers and transports are selected based on the provided configuration string.
This allows us to use the above mentioned new features by default, but at the same time also provide the option for backwards compatibility.
This change also enables us to build a custom client by pooling a number of buffer and transport objects, and owning the flushing strategy, potentially reaching better performance than just using Sender instances.

Backwards compatibility options:

• protocol_version: set it to 1 to remove support for arrays and binary encoding of floating point values.

  const sender = await Sender.fromConfig('http::addr=localhost:9000;protocol_version=1');

• stdlib_http: set it to on to switch back to the standard http/https modules instead of using Undici.

  const sender = await Sender.fromConfig('http::addr=localhost:9000;stdlib_http=on');

HTTP transport and configuration string

Release overview

Major API improvements, including support for InfluxDB Line Protocol over HTTP.

Breaking change: With the introduction of the HTTP transport it is now mandatory to select the protocol to be used by the client.

What is new

Support for the ILP protocol over HTTP transport, as well as the new config string syntax

• The new HTTP sender has better error reporting. In particular, it returns an error if the QuestDB server failed to write ILP messages for any reason.
• HTTP sender handles network errors and retries writes automatically.
• HTTP sender reuses connections to avoid open connection overhead for each flush() call.
• The new fromConf() method provides a convenient way for constructing a Sender via a configuration string.

Migration

v2 code example:

const sender = new Sender();
await sender.connect({ port: 9009, host: 'localhost' });
sender.table('tablename').intColumn('id', 0).atNow();
await sender.flush();
await sender.close();

Migrated v3 code:

const sender = Sender.fromConfig('http::addr=localhost:9000');
await sender.table('tablename').intColumn('id', 0).atNow();
await sender.flush();
await sender.close();

Note, that the migrated code uses the HTTP sender instead of the TCP one.

Simpler authentication

Previous versions of the client required the entire JWK for authentication.

const sender = new Sender({
    jwk: {
        kid: '<username>',
        d: '<private key>'
        x: '<public key x>',
        y: '<public key y>',
        kty: 'EC',
        crv: 'P-256'
    }
});

Starting with v2.1.0 the client supports a simpler way of authentication, where it is enough to specify only the username and the user's private key.

const sender = new Sender({
    auth: {
        keyId: '<username>',
        token: '<private key>'
    }
});

Safer timestamp API

An optional time unit parameter can be passed to timestamp-accepting methods:

  const bday = Date.parse('1856-07-10');
  sender
    .table('inventors')
    .symbol('born', 'Austrian Empire')
    .timestampColumn('birthday', bday, 'ms') // notice 'ms' here
    .intColumn('id', 0)
    .stringColumn('name', 'Nicola Tesla')
    .at(Date.now(), 'ms'); // notice 'ms' here

The time unit defaults to 'us' (native QuestDB resolution) in both timestampColumn() and at().

Supported values are:

• 'ns' - nanoseconds
• 'us' - microseconds
• 'ms' - milliseconds

Note: This release contains a breaking change.
at() does not accept string type anymore, and number/BigInt values are interpreted as microseconds instead of nanoseconds.

BigInt support for timestamp values

The client's API will accept BigInt types wherever a timestamp needs to be passed.
The API change is backwards compatible.

Sender option 'copyBuffer' defaults to 'true'

• sender option 'copyBuffer' defaults to 'true'
• type checks for sender options
• worker threads example

TCP keepalive

TCP keepalive has been enabled for the QuestDB socket connection to avoid disconnects caused by client inactivity.
Should work in most cases; if the connection goes through a firewall this solution might not be good enough.
In that case users could add empty messages on application layer probably to keep the connection alive.
The long term solution is definitely to improve the client further and implement keepalive on ILP protocol level (or whatever will be replacing it in the future).

Custom logging

A new option to pass a custom logging function to the client.
If no logger specified the client will write log messages to console, default logging level is info.