Skip to content

Conversation

@fenos
Copy link
Contributor

@fenos fenos commented Oct 3, 2025

What kind of change does this PR introduce?

Feature

What is the new behaviour?

Implement Vector Bucket data source

Supported Operations:

  • CreateIndex
  • DeleteIndex
  • GetIndex
  • ListIndexes
  • PutVectors
  • ListVectors
  • ListVectorBuckets
  • QueryVectors
  • DeleteVectors
  • GetVectorBucket
  • GetVectors

Authentication mechanisms:

  • SignV4
  • JWT service_role

@snyk-io
Copy link

snyk-io bot commented Oct 3, 2025

Snyk checks have failed. 8 issues have been found so far.

Status Scanner Critical High Medium Low Total (8)
Code Security 0 8 0 0 8 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

@fenos fenos force-pushed the feat/vector-buckets branch 6 times, most recently from 01797cc to a2715e7 Compare October 10, 2025 11:12
@coveralls
Copy link

coveralls commented Oct 10, 2025

Pull Request Test Coverage Report for Build 18909481244

Details

  • 2819 of 3697 (76.25%) changed or added relevant lines in 58 files are covered.
  • 23 unchanged lines in 2 files lost coverage.
  • Overall coverage increased (+0.2%) to 76.514%

Changes Missing Coverage Covered Lines Changed/Added Lines %
src/http/plugins/jwt.ts 3 5 60.0%
src/http/routes/vector/create-bucket.ts 43 45 95.56%
src/http/routes/vector/create-index.ts 64 66 96.97%
src/http/routes/vector/delete-bucket.ts 43 45 95.56%
src/http/routes/vector/delete-index.ts 51 53 96.23%
src/http/routes/vector/delete-vectors.ts 48 50 96.0%
src/http/routes/vector/get-bucket.ts 43 45 95.56%
src/http/routes/vector/get-index.ts 62 64 96.88%
src/http/routes/vector/get-vectors.ts 47 49 95.92%
src/http/routes/vector/list-buckets.ts 44 46 95.65%
Files with Coverage Reduction New Missed Lines %
src/http/plugins/signature-v4.ts 1 46.09%
src/storage/protocols/s3/signature-v4.ts 22 73.11%
Totals Coverage Status
Change from base Build 18866876416: 0.2%
Covered Lines: 23983
Relevant Lines: 31057

💛 - Coveralls

src/app.ts Outdated
app.register(routes.cdn, { prefix: 'cdn' })
app.register(routes.healthcheck, { prefix: 'health' })
app.register(routes.iceberg, { prefix: 'iceberg/v1' })
app.register(routes.vectors, { prefix: 'vectors' })
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should this be singular ("vector")? to match: bucket, object, ...

: 400

if (statusCode === 500) {
console.log('error')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

debug log, remove

}
}

throw ERRORS.TransactionError('Transaction failed after maximum retries', lastError)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might be nice to make "after maximum retries" conditional on if there were retires, and/or maybe include the retry count in the error message to avoid confusion if we need to debug this in the future.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure i understood this, but in isolation mode: 'serialize' retrying the transaction is a common pattern
if the max retries are reached means that there are too many concurrent operations, which makes this transaction fail

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just meant adding the count to the error message or only saying "after maximum retries" if there were retries as currently it says that even for requests that had no retries.

@fenos fenos force-pushed the feat/vector-buckets branch from a2715e7 to 6bf5f59 Compare October 13, 2025 09:35
@fenos fenos force-pushed the feat/vector-buckets branch from 6bf5f59 to f2ae953 Compare October 17, 2025 09:33
search: query.search,
})

return response.status(200).send(bucket)
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  Cross-site Scripting (XSS)

Unsanitized input from an HTTP parameter flows into send, where it is used to render an HTML page returned to the user. This may result in a Cross-Site Scripting attack (XSS).

Line 119 | CWE-79 | Priority score 848 | Learn more about this vulnerability
Data flow: 14 steps

Step 1 - 3

const query = request.query

Step 4 - 7 src/http/routes/iceberg/bucket.ts#L112

Step 8 - 12 src/http/routes/iceberg/bucket.ts#L111

Step 13 - 14

return response.status(200).send(bucket)

@fenos fenos force-pushed the feat/vector-buckets branch from 02f1ec8 to ee74d2a Compare October 29, 2025 13:25
@fenos fenos force-pushed the feat/vector-buckets branch from ee74d2a to 111c2b2 Compare October 29, 2025 13:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants