set up suites which sync with upstream projects?

[He-Pin]

hepin@Mac test_suite % ls|wc
     336     336   11360
hepin@Mac test_suite % ls|wc -l
     336
hepin@Mac test_suite % cd /Users/hepin/GolandProjects/go-jsonnet/testdata
hepin@Mac testdata % ls | wc -l
    2100
hepin@Mac testdata % 

I just checked there are more tests upstream

[johnbartholomew]

The go-jsonnet tests.sh already runs the C++ jsonnet test suite, so there is some sharing going on already. But it probably does need some cleanup and improvement to avoid divergence and make sure both C++ jsonnet and go-jsonnet are running all the relevant tests.

I am also interested in finding out whether other jsonnet implementations reuse the test suites from C++ or Go jsonnet, and if there is any interest from their authors in doing so.

@CertainLach Does jrsonnet re-use any/all of the test suite from here or from go-jsonnet?

@eduardosm Same question for rsjsonnet?

@rudo-thomas Same question for databricks/sjsonnet?

[rudo-thomas]

Pinging @stephenamar-db for sjson et

[johnbartholomew]

Cross-ref, issue #141 is a much older suggestion to have a shared test suite that can be used by multiple implementations; there is some prior discussion there (mostly about whether it's ok/not ok to require Bazel to run the test suite)

[CertainLach]

@CertainLach Does jrsonnet re-use any/all of the test suite from here or from go-jsonnet?

I have tried to integrate jsonnet testsuite in jrsonnet, but ended up only running it manually and locally
I would like to automate this testing, but I need to create my own golden files for everything and patch some of the test files that include expected evaluation results, because

1. Stack-traces in jrsonnet are completely different
2. Number formatting doesn't match jsonnet at all
3. Couple of tests were failing just because I was in a disagreement with CppJsonnet implementation of some things. E.g in Jsonnet std.jsonnet, many of the functions have no argument type checking, and couple of the tests have been expecting dubious output because of dubious input... I can't however remember which one was it. It might have been removed after many functions were reimplemented as builtins in go-jsonnet

[eduardosm]

@eduardosm Same question for rsjsonnet?

I have a script to run in CI that downloads C++ jsonnet sources and runs its tests. For error tests, I just check that it fails without comparing the outputs, since rsjsonnet prints errors with a different format.

[He-Pin]

There is a testsuite in jsonpath, https://github.com/jsonpath-standard/jsonpath-compliance-test-suite