codeflare-operator-manager Keeps going into CrashLoopBackOff #698
Description
OpenShift v4.18
Installed via Operator: CodeFlare Operator V1.16.0
codeflare-operator-manager starts then crashes with CrashLoopBackOff - which then causes the Operator to re-install, which then re-starts the codeflare operator, and the loop starts again.
The logs from the Pod..
2025-09-02T11:09:54Z INFO setup Build info {"operatorVersion": "", "appwrapperVersion": "UNKNOWN", "date": "2025-06-16 10:15"}
2025-09-02T11:09:54Z INFO setup Successfully configured operator {"config": {"clientConnection":{"qps":50,"burst":100},"metrics":{"bindAddress":":8080"},"health":{"bindAddress":":8081","readinessEndpointName":"readyz","livenessEndpointName":"healthz"},"leaderElection":{"leaderElect":null,"leaseDuration":"0s","renewDeadline":"0s","retryPeriod":"0s","resourceLock":"","resourceName":"","resourceNamespace":""},"kuberay":{"rayDashboardOAuthEnabled":true,"ingressDomain":"","mTLSEnabled":true},"appwrapper":{"enabled":false,"Config":{"autopilot":{"injectAntiAffinities":true,"monitorNodes":true,"resourceTaints":{"nvidia.com/gpu":[{"key":"autopilot.ibm.com/gpuhealth","value":"WARN","effect":"PreferNoSchedule"},{"key":"autopilot.ibm.com/gpuhealth","value":"TESTING","effect":"NoSchedule"},{"key":"autopilot.ibm.com/gpuhealth","value":"EVICT","effect":"NoExecute"}]},"preferNoScheduleWeight":50},"userRBACAdmissionCheck":true,"faultTolerance":{"admissionGracePeriod":60000000000,"warmupGracePeriod":300000000000,"failureGracePeriod":60000000000,"resetPause":90000000000,"retryLimit":3,"deletionGracePeriod":600000000000,"gracePeriodCeiling":86400000000000,"successTTLCeiling":604800000000000}}}}}
2025-09-02T11:09:54Z INFO We detected being on OpenShift!
2025-09-02T11:09:54Z INFO setup setting up health endpoints
2025-09-02T11:09:54Z INFO setup setting up RayCluster controller
2025-09-02T11:09:54Z INFO We detected being on OpenShift!
2025-09-02T11:09:54Z INFO setup setting up AppWrapper components
2025-09-02T11:09:54Z INFO setup Embedded AppWrapper controller is disabled by config
2025-09-02T11:09:54Z INFO setup starting manager
2025-09-02T11:09:54Z INFO controller-runtime.metrics Starting metrics server
2025-09-02T11:09:54Z INFO controller-runtime.metrics Serving metrics server {"bindAddress": ":8080", "secure": false}
2025-09-02T11:09:54Z INFO starting server {"name": "health probe", "addr": "[::]:8081"}
2025-09-02T11:09:54Z INFO Starting EventSource {"controller": "cert-rotator", "source": "kind source: *v1.Secret"}
2025-09-02T11:09:54Z INFO Starting EventSource {"controller": "cert-rotator", "source": "kind source: *unstructured.Unstructured"}
2025-09-02T11:09:54Z INFO Starting EventSource {"controller": "cert-rotator", "source": "kind source: *unstructured.Unstructured"}
2025-09-02T11:09:54Z INFO Starting Controller {"controller": "cert-rotator"}
2025-09-02T11:09:54Z INFO cert-rotation starting cert rotator controller
2025-09-02T11:09:54Z INFO cert-rotation no cert refresh needed
2025-09-02T11:09:54Z INFO cert-rotation certs are ready in /tmp/k8s-webhook-server/serving-certs
2025-09-02T11:09:54Z INFO Starting workers {"controller": "cert-rotator", "worker count": 1}
2025-09-02T11:09:54Z INFO cert-rotation no cert refresh needed
2025-09-02T11:09:54Z ERROR cert-rotation Webhook not found. Unable to update certificate. {"name": "codeflare-operator-validating-webhook-configuration", "gvk": "admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration", "error": "ValidatingWebhookConfiguration.admissionregistration.k8s.io \"codeflare-operator-validating-webhook-configuration\" not found"}
github.com/go-logr/logr.Logger.Error
/root/go/pkg/mod/github.com/go-logr/logr@v1.4.2/logr.go:301
github.com/open-policy-agent/cert-controller/pkg/rotator.(*ReconcileWH).ensureCerts
/root/go/pkg/mod/github.com/open-policy-agent/cert-controller@v0.12.0/pkg/rotator/rotator.go:822
github.com/open-policy-agent/cert-controller/pkg/rotator.(*ReconcileWH).Reconcile
/root/go/pkg/mod/github.com/open-policy-agent/cert-controller@v0.12.0/pkg/rotator/rotator.go:791
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/controller/controller.go:116
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/controller/controller.go:303
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/controller/controller.go:263
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/controller/controller.go:224
2025-09-02T11:09:54Z ERROR cert-rotation Webhook not found. Unable to update certificate. {"name": "codeflare-operator-mutating-webhook-configuration", "gvk": "admissionregistration.k8s.io/v1, Kind=MutatingWebhookConfiguration", "error": "MutatingWebhookConfiguration.admissionregistration.k8s.io \"codeflare-operator-mutating-webhook-configuration\" not found"}
github.com/go-logr/logr.Logger.Error
/root/go/pkg/mod/github.com/go-logr/logr@v1.4.2/logr.go:301
github.com/open-policy-agent/cert-controller/pkg/rotator.(*ReconcileWH).ensureCerts
/root/go/pkg/mod/github.com/open-policy-agent/cert-controller@v0.12.0/pkg/rotator/rotator.go:822
github.com/open-policy-agent/cert-controller/pkg/rotator.(*ReconcileWH).Reconcile
/root/go/pkg/mod/github.com/open-policy-agent/cert-controller@v0.12.0/pkg/rotator/rotator.go:791
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/controller/controller.go:116
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/controller/controller.go:303
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/controller/controller.go:263
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/controller/controller.go:224
2025-09-02T11:09:55Z INFO setup Waiting for certificate generation to complete
2025-09-02T11:09:55Z INFO cert-rotation CA certs are injected to webhooks
2025-09-02T11:09:55Z INFO setup Certs ready
2025-09-02T11:09:55Z INFO controller-runtime.builder Registering a mutating webhook {"GVK": "ray.io/v1, Kind=RayCluster", "path": "/mutate-ray-io-v1-raycluster"}
2025-09-02T11:09:55Z INFO controller-runtime.webhook Registering webhook {"path": "/mutate-ray-io-v1-raycluster"}
2025-09-02T11:09:55Z INFO controller-runtime.builder Registering a validating webhook {"GVK": "ray.io/v1, Kind=RayCluster", "path": "/validate-ray-io-v1-raycluster"}
2025-09-02T11:09:55Z INFO controller-runtime.webhook Starting webhook server
2025-09-02T11:09:55Z INFO controller-runtime.webhook Registering webhook {"path": "/validate-ray-io-v1-raycluster"}
2025-09-02T11:09:55Z INFO controller-runtime.certwatcher Updated current TLS certificate
2025-09-02T11:09:55Z INFO controller-runtime.webhook Serving webhook server {"host": "", "port": 9443}
2025-09-02T11:09:55Z INFO Starting EventSource {"controller": "codeflare-raycluster-controller", "controllerGroup": "ray.io", "controllerKind": "RayCluster", "source": "kind source: *v1.RayCluster"}
2025-09-02T11:09:55Z INFO controller-runtime.certwatcher Starting certificate watcher
2025-09-02T11:09:55Z INFO Starting EventSource {"controller": "codeflare-raycluster-controller", "controllerGroup": "ray.io", "controllerKind": "RayCluster", "source": "kind source: *v1.ServiceAccount"}
2025-09-02T11:09:55Z INFO Starting EventSource {"controller": "codeflare-raycluster-controller", "controllerGroup": "ray.io", "controllerKind": "RayCluster", "source": "kind source: *v1.Service"}
2025-09-02T11:09:55Z INFO Starting EventSource {"controller": "codeflare-raycluster-controller", "controllerGroup": "ray.io", "controllerKind": "RayCluster", "source": "kind source: *v1.Secret"}
2025-09-02T11:09:55Z INFO Starting EventSource {"controller": "codeflare-raycluster-controller", "controllerGroup": "ray.io", "controllerKind": "RayCluster", "source": "kind source: *v1.Ingress"}
2025-09-02T11:09:55Z INFO Starting EventSource {"controller": "codeflare-raycluster-controller", "controllerGroup": "ray.io", "controllerKind": "RayCluster", "source": "kind source: *v1.NetworkPolicy"}
2025-09-02T11:09:55Z INFO Starting EventSource {"controller": "codeflare-raycluster-controller", "controllerGroup": "ray.io", "controllerKind": "RayCluster", "source": "kind source: *v1.Route"}
2025-09-02T11:09:55Z INFO Starting EventSource {"controller": "codeflare-raycluster-controller", "controllerGroup": "ray.io", "controllerKind": "RayCluster", "source": "kind source: *v1.ClusterRoleBinding"}
2025-09-02T11:09:55Z INFO Starting Controller {"controller": "codeflare-raycluster-controller", "controllerGroup": "ray.io", "controllerKind": "RayCluster"}
2025-09-02T11:09:55Z ERROR controller-runtime.source.EventHandler if kind is a CRD, it should be installed before calling Start {"kind": "RayCluster.ray.io", "error": "no matches for kind \"RayCluster\" in version \"ray.io/v1\""}
sigs.k8s.io/controller-runtime/pkg/log.(*delegatingLogSink).Error
/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/log/deleg.go:142
github.com/go-logr/logr.Logger.Error
/root/go/pkg/mod/github.com/go-logr/logr@v1.4.2/logr.go:301
sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind[...]).Start.func1.1
/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/source/kind.go:71
k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func1
/root/go/pkg/mod/k8s.io/apimachinery@v0.31.4/pkg/util/wait/loop.go:53
k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext
/root/go/pkg/mod/k8s.io/apimachinery@v0.31.4/pkg/util/wait/loop.go:54
k8s.io/apimachinery/pkg/util/wait.PollUntilContextCancel
/root/go/pkg/mod/k8s.io/apimachinery@v0.31.4/pkg/util/wait/poll.go:33
sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind[...]).Start.func1
/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/source/kind.go:64
2025-09-02T11:10:05Z ERROR controller-runtime.source.EventHandler if kind is a CRD, it should be installed before calling Start {"kind": "RayCluster.ray.io", "error": "no matches for kind \"RayCluster\" in version \"ray.io/v1\""}
sigs.k8s.io/controller-runtime/pkg/log.(*delegatingLogSink).Error
/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/log/deleg.go:142
github.com/go-logr/logr.Logger.Error
/root/go/pkg/mod/github.com/go-logr/logr@v1.4.2/logr.go:301
sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind[...]).Start.func1.1
/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/source/kind.go:71
k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func2
/root/go/pkg/mod/k8s.io/apimachinery@v0.31.4/pkg/util/wait/loop.go:87
k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext
/root/go/pkg/mod/k8s.io/apimachinery@v0.31.4/pkg/util/wait/loop.go:88
k8s.io/apimachinery/pkg/util/wait.PollUntilContextCancel
/root/go/pkg/mod/k8s.io/apimachinery@v0.31.4/pkg/util/wait/poll.go:33
sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind[...]).Start.func1
/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/source/kind.go:64
2025-09-02T11:10:15Z ERROR controller-runtime.source.EventHandler if kind is a CRD, it should be installed before calling Start {"kind": "RayCluster.ray.io", "error": "no matches for kind \"RayCluster\" in version \"ray.io/v1\""}
sigs.k8s.io/controller-runtime/pkg/log.(*delegatingLogSink).Error
/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/log/deleg.go:142
github.com/go-logr/logr.Logger.Error
/root/go/pkg/mod/github.com/go-logr/logr@v1.4.2/logr.go:301
sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind[...]).Start.func1.1
/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/source/kind.go:71
k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func2
/root/go/pkg/mod/k8s.io/apimachinery@v0.31.4/pkg/util/wait/loop.go:87
k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext
/root/go/pkg/mod/k8s.io/apimachinery@v0.31.4/pkg/util/wait/loop.go:88
k8s.io/apimachinery/pkg/util/wait.PollUntilContextCancel
/root/go/pkg/mod/k8s.io/apimachinery@v0.31.4/pkg/util/wait/poll.go:33
sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind[...]).Start.func1
/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/source/kind.go:64
2025-09-02T11:10:25Z ERROR controller-runtime.source.EventHandler if kind is a CRD, it should be installed before calling Start {"kind": "RayCluster.ray.io", "error": "no matches for kind \"RayCluster\" in version \"ray.io/v1\""}
sigs.k8s.io/controller-runtime/pkg/log.(*delegatingLogSink).Error
/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/log/deleg.go:142
github.com/go-logr/logr.Logger.Error
/root/go/pkg/mod/github.com/go-logr/logr@v1.4.2/logr.go:301
sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind[...]).Start.func1.1
/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/source/kind.go:71
k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func2
/root/go/pkg/mod/k8s.io/apimachinery@v0.31.4/pkg/util/wait/loop.go:87
k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext
/root/go/pkg/mod/k8s.io/apimachinery@v0.31.4/pkg/util/wait/loop.go:88
k8s.io/apimachinery/pkg/util/wait.PollUntilContextCancel
/root/go/pkg/mod/k8s.io/apimachinery@v0.31.4/pkg/util/wait/poll.go:33
sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind[...]).Start.func1
/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/source/kind.go:64
2025-09-02T11:10:35Z ERROR controller-runtime.source.EventHandler if kind is a CRD, it should be installed before calling Start {"kind": "RayCluster.ray.io", "error": "no matches for kind \"RayCluster\" in version \"ray.io/v1\""}
sigs.k8s.io/controller-runtime/pkg/log.(*delegatingLogSink).Error
/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/log/deleg.go:142
github.com/go-logr/logr.Logger.Error
/root/go/pkg/mod/github.com/go-logr/logr@v1.4.2/logr.go:301
sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind[...]).Start.func1.1
/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/source/kind.go:71
k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func2
/root/go/pkg/mod/k8s.io/apimachinery@v0.31.4/pkg/util/wait/loop.go:87
k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext
/root/go/pkg/mod/k8s.io/apimachinery@v0.31.4/pkg/util/wait/loop.go:88
k8s.io/apimachinery/pkg/util/wait.PollUntilContextCancel
/root/go/pkg/mod/k8s.io/apimachinery@v0.31.4/pkg/util/wait/poll.go:33
sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind[...]).Start.func1
/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/source/kind.go:64
2025-09-02T11:10:45Z ERROR controller-runtime.source.EventHandler if kind is a CRD, it should be installed before calling Start {"kind": "RayCluster.ray.io", "error": "no matches for kind \"RayCluster\" in version \"ray.io/v1\""}
sigs.k8s.io/controller-runtime/pkg/log.(*delegatingLogSink).Error
/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/log/deleg.go:142
github.com/go-logr/logr.Logger.Error
/root/go/pkg/mod/github.com/go-logr/logr@v1.4.2/logr.go:301
sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind[...]).Start.func1.1
/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/source/kind.go:71
k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func2
/root/go/pkg/mod/k8s.io/apimachinery@v0.31.4/pkg/util/wait/loop.go:87
k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext
/root/go/pkg/mod/k8s.io/apimachinery@v0.31.4/pkg/util/wait/loop.go:88
k8s.io/apimachinery/pkg/util/wait.PollUntilContextCancel
/root/go/pkg/mod/k8s.io/apimachinery@v0.31.4/pkg/util/wait/poll.go:33
sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind[...]).Start.func1
/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/source/kind.go:64
2025-09-02T11:10:55Z ERROR controller-runtime.source.EventHandler if kind is a CRD, it should be installed before calling Start {"kind": "RayCluster.ray.io", "error": "no matches for kind \"RayCluster\" in version \"ray.io/v1\""}
sigs.k8s.io/controller-runtime/pkg/log.(*delegatingLogSink).Error
/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/log/deleg.go:142
github.com/go-logr/logr.Logger.Error
/root/go/pkg/mod/github.com/go-logr/logr@v1.4.2/logr.go:301
sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind[...]).Start.func1.1
/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/source/kind.go:71
k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func2
/root/go/pkg/mod/k8s.io/apimachinery@v0.31.4/pkg/util/wait/loop.go:87
k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext
/root/go/pkg/mod/k8s.io/apimachinery@v0.31.4/pkg/util/wait/loop.go:88
k8s.io/apimachinery/pkg/util/wait.PollUntilContextCancel
/root/go/pkg/mod/k8s.io/apimachinery@v0.31.4/pkg/util/wait/poll.go:33
sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind[...]).Start.func1
/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/source/kind.go:64
2025-09-02T11:11:05Z ERROR controller-runtime.source.EventHandler if kind is a CRD, it should be installed before calling Start {"kind": "RayCluster.ray.io", "error": "no matches for kind \"RayCluster\" in version \"ray.io/v1\""}
sigs.k8s.io/controller-runtime/pkg/log.(*delegatingLogSink).Error
/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/log/deleg.go:142
github.com/go-logr/logr.Logger.Error
/root/go/pkg/mod/github.com/go-logr/logr@v1.4.2/logr.go:301
sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind[...]).Start.func1.1
/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/source/kind.go:71
k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func2
/root/go/pkg/mod/k8s.io/apimachinery@v0.31.4/pkg/util/wait/loop.go:87
k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext
/root/go/pkg/mod/k8s.io/apimachinery@v0.31.4/pkg/util/wait/loop.go:88
k8s.io/apimachinery/pkg/util/wait.PollUntilContextCancel
/root/go/pkg/mod/k8s.io/apimachinery@v0.31.4/pkg/util/wait/poll.go:33
sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind[...]).Start.func1
/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/source/kind.go:64
2025-09-02T11:11:15Z ERROR controller-runtime.source.EventHandler if kind is a CRD, it should be installed before calling Start {"kind": "RayCluster.ray.io", "error": "no matches for kind \"RayCluster\" in version \"ray.io/v1\""}
sigs.k8s.io/controller-runtime/pkg/log.(*delegatingLogSink).Error
/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/log/deleg.go:142
github.com/go-logr/logr.Logger.Error
/root/go/pkg/mod/github.com/go-logr/logr@v1.4.2/logr.go:301
sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind[...]).Start.func1.1
/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/source/kind.go:71
k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func2
/root/go/pkg/mod/k8s.io/apimachinery@v0.31.4/pkg/util/wait/loop.go:87
k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext
/root/go/pkg/mod/k8s.io/apimachinery@v0.31.4/pkg/util/wait/loop.go:88
k8s.io/apimachinery/pkg/util/wait.PollUntilContextCancel
/root/go/pkg/mod/k8s.io/apimachinery@v0.31.4/pkg/util/wait/poll.go:33
sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind[...]).Start.func1
/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/source/kind.go:64
2025-09-02T11:11:25Z ERROR controller-runtime.source.EventHandler if kind is a CRD, it should be installed before calling Start {"kind": "RayCluster.ray.io", "error": "no matches for kind \"RayCluster\" in version \"ray.io/v1\""}
sigs.k8s.io/controller-runtime/pkg/log.(*delegatingLogSink).Error
/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/log/deleg.go:142
github.com/go-logr/logr.Logger.Error
/root/go/pkg/mod/github.com/go-logr/logr@v1.4.2/logr.go:301
sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind[...]).Start.func1.1
/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/source/kind.go:71
k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func2
/root/go/pkg/mod/k8s.io/apimachinery@v0.31.4/pkg/util/wait/loop.go:87
k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext
/root/go/pkg/mod/k8s.io/apimachinery@v0.31.4/pkg/util/wait/loop.go:88
k8s.io/apimachinery/pkg/util/wait.PollUntilContextCancel
/root/go/pkg/mod/k8s.io/apimachinery@v0.31.4/pkg/util/wait/poll.go:33
sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind[...]).Start.func1
/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/source/kind.go:64
2025-09-02T11:11:35Z ERROR controller-runtime.source.EventHandler if kind is a CRD, it should be installed before calling Start {"kind": "RayCluster.ray.io", "error": "no matches for kind \"RayCluster\" in version \"ray.io/v1\""}
sigs.k8s.io/controller-runtime/pkg/log.(*delegatingLogSink).Error
/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/log/deleg.go:142
github.com/go-logr/logr.Logger.Error
/root/go/pkg/mod/github.com/go-logr/logr@v1.4.2/logr.go:301
sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind[...]).Start.func1.1
/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/source/kind.go:71
k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func2
/root/go/pkg/mod/k8s.io/apimachinery@v0.31.4/pkg/util/wait/loop.go:87
k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext
/root/go/pkg/mod/k8s.io/apimachinery@v0.31.4/pkg/util/wait/loop.go:88
k8s.io/apimachinery/pkg/util/wait.PollUntilContextCancel
/root/go/pkg/mod/k8s.io/apimachinery@v0.31.4/pkg/util/wait/poll.go:33
sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind[...]).Start.func1
/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/source/kind.go:64
2025-09-02T11:11:45Z ERROR controller-runtime.source.EventHandler if kind is a CRD, it should be installed before calling Start {"kind": "RayCluster.ray.io", "error": "no matches for kind \"RayCluster\" in version \"ray.io/v1\""}
sigs.k8s.io/controller-runtime/pkg/log.(*delegatingLogSink).Error
/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/log/deleg.go:142
github.com/go-logr/logr.Logger.Error
/root/go/pkg/mod/github.com/go-logr/logr@v1.4.2/logr.go:301
sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind[...]).Start.func1.1
/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/source/kind.go:71
k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func2
/root/go/pkg/mod/k8s.io/apimachinery@v0.31.4/pkg/util/wait/loop.go:87
k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext
/root/go/pkg/mod/k8s.io/apimachinery@v0.31.4/pkg/util/wait/loop.go:88
k8s.io/apimachinery/pkg/util/wait.PollUntilContextCancel
/root/go/pkg/mod/k8s.io/apimachinery@v0.31.4/pkg/util/wait/poll.go:33
sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind[...]).Start.func1
/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/source/kind.go:64
2025-09-02T11:11:55Z ERROR Could not wait for Cache to sync {"controller": "codeflare-raycluster-controller", "controllerGroup": "ray.io", "controllerKind": "RayCluster", "error": "failed to wait for codeflare-raycluster-controller caches to sync: timed out waiting for cache to be synced for Kind *v1.RayCluster"}
github.com/go-logr/logr.Logger.Error
/root/go/pkg/mod/github.com/go-logr/logr@v1.4.2/logr.go:301
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1
/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/controller/controller.go:200
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2
/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/controller/controller.go:205
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start
/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/controller/controller.go:231
sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile.func1
/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/manager/runnable_group.go:226
2025-09-02T11:11:55Z INFO Stopping and waiting for non leader election runnables
2025-09-02T11:11:55Z INFO Shutdown signal received, waiting for all workers to finish {"controller": "cert-rotator"}
2025-09-02T11:11:55Z INFO cert-rotation stopping cert rotator controller
2025-09-02T11:11:55Z INFO All workers finished {"controller": "cert-rotator"}
2025-09-02T11:11:55Z INFO Stopping and waiting for leader election runnables
2025-09-02T11:11:55Z INFO Stopping and waiting for caches
2025-09-02T11:11:55Z INFO pkg/mod/k8s.io/client-go@v0.31.4/tools/cache/reflector.go:243: watch of *v1.Secret ended with: an error on the server ("unable to decode an event from the watch stream: context canceled") has prevented the request from succeeding
2025-09-02T11:11:55Z INFO pkg/mod/k8s.io/client-go@v0.31.4/tools/cache/reflector.go:243: watch of *v1.Route ended with: an error on the server ("unable to decode an event from the watch stream: context canceled") has prevented the request from succeeding
2025-09-02T11:11:55Z INFO pkg/mod/k8s.io/client-go@v0.31.4/tools/cache/reflector.go:243: watch of *v1.Service ended with: an error on the server ("unable to decode an event from the watch stream: context canceled") has prevented the request from succeeding
2025-09-02T11:11:55Z ERROR controller-runtime.source.EventHandler if kind is a CRD, it should be installed before calling Start {"kind": "RayCluster.ray.io", "error": "no matches for kind \"RayCluster\" in version \"ray.io/v1\""}
sigs.k8s.io/controller-runtime/pkg/log.(*delegatingLogSink).Error
/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/log/deleg.go:142
github.com/go-logr/logr.Logger.Error
/root/go/pkg/mod/github.com/go-logr/logr@v1.4.2/logr.go:301
sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind[...]).Start.func1.1
/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/source/kind.go:71
k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func2
/root/go/pkg/mod/k8s.io/apimachinery@v0.31.4/pkg/util/wait/loop.go:87
k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext
/root/go/pkg/mod/k8s.io/apimachinery@v0.31.4/pkg/util/wait/loop.go:88
k8s.io/apimachinery/pkg/util/wait.PollUntilContextCancel
/root/go/pkg/mod/k8s.io/apimachinery@v0.31.4/pkg/util/wait/poll.go:33
sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind[...]).Start.func1
/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/source/kind.go:64
2025-09-02T11:11:55Z INFO Stopping and waiting for webhooks
2025-09-02T11:11:55Z INFO controller-runtime.webhook Shutting down webhook server with timeout of 1 minute
2025-09-02T11:11:55Z INFO Stopping and waiting for HTTP servers
2025-09-02T11:11:55Z INFO shutting down server {"name": "health probe", "addr": "[::]:8081"}
2025-09-02T11:11:55Z INFO controller-runtime.metrics Shutting down metrics server with timeout of 1 minute
2025-09-02T11:11:55Z INFO Wait completed, proceeding to shutdown the manager
2025-09-02T11:11:55Z ERROR setup error running manager {"error": "failed to wait for codeflare-raycluster-controller caches to sync: timed out waiting for cache to be synced for Kind *v1.RayCluster"}
sigs.k8s.io/controller-runtime/pkg/log.(*delegatingLogSink).Error
/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/log/deleg.go:142
github.com/go-logr/logr.Logger.Error
/root/go/pkg/mod/github.com/go-logr/logr@v1.4.2/logr.go:301
main.exitOnError
/workspace/main.go:398
main.main
/workspace/main.go:237
runtime.main
/usr/local/go/src/runtime/proc.go:272