FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

CKS: NPE when trying to remove a external node from a cks cluster · Issue #11581 · apache/cloudstack · GitHub

  1. apache
  2. cloudstack

Repository navigation

CKS: NPE when trying to remove a external node from a cks cluster #11581

New issue
New issue
Open
Bug
Open
CKS: NPE when trying to remove a external node from a cks cluster#11581
Bug
Assignees
Labels
component:cks
Milestone
4.22.3

Description

kiranchavala
opened on Sep 4, 2025

problem

CKS: NPE when trying to remove a external node from a cks cluster

versions

ACS 4.20.x

The steps to reproduce the bug

  1. Register a external cks template

https://download.cloudstack.org/testing/custom_templates/ubuntu/22.04/22.04/cks-ubuntu-2204-kvm.qcow2.bz2

  1. Launch a cks cluster

  2. Launch a Ubuntu vm with the template mentioned above

  3. Add the management server public key, once the Ubuntu VM boots up

  4. Add the Ubuntu vm as external node to the cks cluster

  1. CKS cluster will be in importing state

  2. The external node will go be in not-ready state , due to disk issue

Login to the external node and check the cloud-init-output.log

unpacking registry.k8s.io/etcd:3.5.21-0 (sha256:d58c035df557080a27387d687092e3fc2b64c6d0e3162dc51453a115f847d121)...time="2025-09-04T09:32:05Z" level=info msg="apply failure, attempting cleanup" error="failed to extract layer sha256:edcdf51bd97dae2c7c6a75ab21cf445d5997888402357f5cc36e7582543431ac: write /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/45/fs/usr/local/bin/etcd-3.4.18: no space left on device: unknown" key="extract-939483347-ZepJ sha256:c6230a0bcc0db1264e316a45b18c0a8dfab3c4818a4245035770b4e58967e035"
time="2025-09-04T09:32:05Z" level=warning msg="extraction snapshot removal failed" error="write /var/lib/containerd/io.containerd.metadata.v1.bolt/meta.db: no space left on device: unknown" key="extract-939483347-ZepJ sha256:c6230a0bcc0db1264e316a45b18c0a8dfab3c4818a4245035770b4e58967e035"
ctr: failed to extract layer sha256:edcdf51bd97dae2c7c6a75ab21cf445d5997888402357f5cc36e7582543431ac: write /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/45/fs/usr/local/bin/etcd-3.4.18: no space left on device: unknown
ctr: failed to ingest "blobs/sha256/0038afa1c30b6e7c6ed64ebbb3593756f0a5328da72cf3304be62b27cb40139a": failed to open writer: mkdir /var/lib/containerd/io.containerd.content.v1.content/ingest/f2e067fbba1183a1b7465f8eb36511a50660708c165d166d396d63d79880c233: no space left on device: unknown
ctr: failed to ingest "blobs/sha256/0038afa1c30b6e7c6ed64ebbb3593756f0a5328da72cf3304be62b27cb40139a": failed to open writer: mkdir /var/lib/containerd/io.containerd.content.v1.content/ingest/f2e067fbba1183a1b7465f8eb36511a50660708c165d166d396d63d79880c233: no space left on device: unknown
Loading docker image /mnt/k8sdisk//docker/etcd:3.5.21-0.tar failed!
ctr: failed to ingest "blobs/sha256/0bab15eea81d0fe6ab56ebf5fba14e02c4c1775a7f7436fbddd3505add4e18fa": failed to open writer: mkdir /var/lib/containerd/io.containerd.content.v1.content/ingest/e32451d5bcc4edc020e31aecb569918a2e63aca4585fb9735e3e19ad02b818a0: no space left on device: unknown
ctr: failed to ingest "blobs/sha256/0bab15eea81d0fe6ab56ebf5fba14e02c4c1775a7f7436fbddd3505add4e18fa": failed to open writer: mkdir /var/lib/containerd/io.containerd.content.v1.content/ingest/e32451d5bcc4edc020e31aecb569918a2e63aca4585fb9735e3e19ad02b818a0: no space left on device: unknown
ctr: failed to ingest "blobs/sha256/0bab15eea81d0fe6ab56ebf5fba14e02c4c1775a7f7436fbddd3505add4e18fa": failed to open writer: mkdir /var/lib/containerd/io.containerd.content.v1.content/ingest/e32451d5bcc4edc020e31aecb569918a2e63aca4585fb9735e3e19ad02b818a0: no space left on device: unknown

  1. Stop the CKS cluster, in oder to remove the external node

  2. CKS cluster goes into stop state , but the addition of the external node job still carries on

[root@ref-trl-9383-k-Mol8-kiran-chavala-mgmt1 ~]# tail -f     /var/log/cloudstack/management/management-server.log |grep job-295
2025-09-04 10:07:11,732 DEBUG [c.c.k.c.u.KubernetesClusterUtil] (API-Job-Executor-52:[ctx-09df8212, job-295, ctx-993d72f7]) (logid:00b69bc8) Checking ready nodes for the Kubernetes cluster KubernetesCluster {"id":9,"name":"isolated","uuid":"8663f80d-3ce7-47f4-b852-4eb145453e0a"} with total 3 provisioned nodes
2025-09-04 10:07:12,529 DEBUG [c.c.k.c.u.KubernetesClusterUtil] (API-Job-Executor-52:[ctx-09df8212, job-295, ctx-993d72f7]) (logid:00b69bc8) Kubernetes cluster KubernetesCluster {"id":9,"name":"isolated","uuid":"8663f80d-3ce7-47f4-b852-4eb145453e0a"} has total 3 provisioned nodes while 2 ready now
2025-09-04 10:07:16,880 WARN  [o.a.c.f.j.i.AsyncJobMonitor] (Timer-0:[ctx-2caef2bd]) (logid:299149e7) Task (job-295) has been pending for 2208 seconds
2025-09-04 10:07:27,530 DEBUG [c.c.k.c.u.KubernetesClusterUtil] (API-Job-Executor-52:[ctx-09df8212, job-295, ctx-993d72f7]) (logid:00b69bc8) Checking ready nodes for the Kubernetes cluster KubernetesCluster {"id":9,"name":"isolated","uuid":"8663f80d-3ce7-47f4-b852-4eb145453e0a"} with total 3 provisioned nodes
2025-09-04 10:07:28,209 DEBUG [c.c.k.c.u.KubernetesClusterUtil] (API-Job-Executor-52:[ctx-09df8212, job-295, ctx-993d72f7]) (logid:00b69bc8) Kubernetes cluster KubernetesCluster {"id":9,"name":"isolated","uuid":"8663f80d-3ce7-47f4-b852-4eb145453e0a"} has total 3 provisioned nodes while 2 ready now
  1. Destroy the external node and start the cks cluster

Exception observed

  1. The cks cluster remains in alert state

  2. Destroy the cks cluster

Exception

2025-09-04 11:01:47,260 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-64:[ctx-bc12e86e, job-333]) (logid:2e067630) Unexpected exception while executing org.apache.cloudstack.api.command.user.kubernetes.cluster.DeleteKubernetesClusterCmd java.lang.NullPointerException: Cannot invoke "com.cloud.vm.VMInstanceVO.getBackupOfferingId()" because "vm" is null
	at com.cloud.kubernetes.cluster.KubernetesClusterManagerImpl.checkIfVmsAssociatedWithBackupOffering(KubernetesClusterManagerImpl.java:2010)
	at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterDestroyWorker.destroy(KubernetesClusterDestroyWorker.java:267)
	at com.cloud.kubernetes.cluster.KubernetesClusterManagerImpl.destroyKubernetesCluster(KubernetesClusterManagerImpl.java:2384)
	at com.cloud.kubernetes.cluster.KubernetesClusterManagerImpl.destroyKubernetesCluster(KubernetesClusterManagerImpl.java:2392)
	at com.cloud.kubernetes.cluster.KubernetesClusterManagerImpl.deleteKubernetesCluster(KubernetesClusterManagerImpl.java:1969)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
	at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:109)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
	at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
	at jdk.proxy3/jdk.proxy3.$Proxy534.deleteKubernetesCluster(Unknown Source)
	at org.apache.cloudstack.api.command.user.kubernetes.cluster.DeleteKubernetesClusterCmd.execute(DeleteKubernetesClusterCmd.java:95)
	at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:173)
	at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:110)

...

What to do about it?

Workaround

Deploy the external ubuntu node with root disk size greater than 20 gb

Need to fix the NPE as the cks cluster remains in alert state

Activity

  1. kiranchavala
    added
    component:cks
    on Sep 4, 2025
  2. kiranchavala
    added the
    Bug
    issue type on Sep 4, 2025
  3. DaanHoogland
    added this to the 4.20.2 milestone on Sep 8, 2025
  4. DaanHoogland
    added this to Apache CloudStack BugFest - Issueson Sep 8, 2025
  5. DaanHoogland
    moved this to Todo in Apache CloudStack BugFest - Issueson Sep 18, 2025
  6. weizhouapache
    modified the milestones: 4.20.2, 4.20.3 on Oct 23, 2025
  7. nvazquez
    self-assigned this
    on Nov 3, 2025
  8. weizhouapache
    moved this from Todo to Dev In Progress in Apache CloudStack BugFest - Issueson Jan 12, 2026

  9. weizhouapache commented on Jan 12, 2026

    weizhouapache
    on Jan 12, 2026
    Member

    @kiranchavala
    it looks like the feature "Add nodes as workers to an existing CKS cluster" was introduced by #9102 in ACS 4.21.0
    can you confirm the ACS version?

  10. weizhouapache
    self-assigned this
    on Jan 12, 2026
  11. weizhouapache
    moved this from Dev In Progress to ready for Review in Apache CloudStack BugFest - Issueson Jan 12, 2026
  12. DaanHoogland
    linked a pull request that will close this issueCKS: fix NPE when remove a failed external node #12407on Jan 12, 2026
  13. vishesh92
    moved this from ready for Review to ready for Testing in Apache CloudStack BugFest - Issueson Jan 14, 2026
  14. vladimirpetrov
    self-assigned this
    on Jan 15, 2026
  15. DaanHoogland
    modified the milestones: 4.20.3, 4.22.1 on Jan 16, 2026
  16. sureshanaparti
    added this to Apache CloudStack 4.22.1on Feb 3, 2026
  17. sureshanaparti
    moved this to In Progress in Apache CloudStack 4.22.1on Feb 3, 2026
  18. DaanHoogland
    moved this from ready for Testing to Todo in Apache CloudStack BugFest - Issueson Mar 5, 2026

  19. kiranchavala commented on Mar 17, 2026

    kiranchavala
    on Mar 17, 2026
    MemberAuthor

    @nvazquez @Pearl1594

    Can you take a look at this

  20. kiranchavala
    assigned
    Pearl1594
    on Mar 17, 2026
  21. sureshanaparti
    removed this from Apache CloudStack 4.22.1on Apr 24, 2026
  22. sureshanaparti
    modified the milestones: 4.22.1, 4.22.2 on Apr 24, 2026
  23. weizhouapache
    removed their assignment
    on Oct 7, 2026
  24. nvazquez
    modified the milestones: 4.22.2, 4.22.3 on Nov 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

component:cks

Type

Bug

Projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions

Back | FazBrowse Home | New Git URL