Cloudstack wizard is getting stuck storage traffic on version 4.20

[verendrakalsi]

Ubuntu 22.04.5 LTS
Cloustack version :- 4.20

we have install cloudstack agent on 4 hosts and 1 hosts with cloudstack mgmt. After installing when we are trying to run the cloudstack mgmt wizard, it is getting stuck on the storage traffic section.

The Section creates the zone, pod, cluster, add the host, & Primary and secondary storage but after that nothing and it keep spinnig on the wizard at storage traffic section.

[verendrakalsi]

its getting stuck here.

[verendrakalsi]

looks like some issue with monuting the secondary storage
Failed to create netfs mount: 10.10.10.12:/KVMSecondary1/template/tmpl/1/3 org.libvirt.LibvirtException: internal error: Child process (/usr/bin/mount -o nodev,nosuid,noexec 10.10.10.12:/KVMSecondary1/template/tmpl/1/3 /mnt/d702e2f4-ecde-318b-b04e-a3cf1bab9afd) unexpected exit status 32: mount.nfs: mounting 10.10.10.12:/KVMSecondary1/template/tmpl/1/3 failed, reason given by server: No such file or directory

[HeinLUKO]

@verendrakalsi hello bro how did u solve this issue ? now i'm facing the same issue with separated secondary storage network traffic .

This is from /var/log/cloudstack/agent/agent.log

2026-07-21 17:00:57,619 ERROR [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-5:[]) (logid:7e967a47) org.libvirt.LibvirtException: internal error: Child process (/usr/bin/mount -o nodev,nosuid,noexec 10.144.9.99:/kvm02/secondary/template/tmpl/1/3 /mnt/47f2ec3c-17be-3cc8-aa3b-ceaad37f06ac) unexpected exit status 32: mount.nfs: mounting 10.144.9.99:/kvm02/secondary/template/tmpl/1/3 failed, reason given by server: No such file or directory

This is from /var/log/cloudstack/agent/agent.err

libvirt: Storage Driver error : Storage pool not found: no storage pool with matching uuid '47f2ec3c-17be-3cc8-aa3b-ceaad37f06ac'
libvirt: error : internal error: Child process (/usr/bin/mount -o nodev,nosuid,noexec 10.144.9.99:/kvm02/secondary/template/tmpl/1/3 /mnt/47f2ec3c-17be-3cc8-aa3b-ceaad37f06ac) unexpected exit status 32: mount.nfs: mounting 10.144.9.99:/kvm02/secondary/template/tmpl/1/3 failed, reason given by server: No such file or directory
libvirt: QEMU Driver error : Domain not found: no domain with matching name 's-3-VM'
libvirt: error : no connection driver available for lxc:///
libvirt: QEMU Driver error : Domain not found: no domain with matching name 's-3-VM'
libvirt: QEMU Driver error : Domain not found: no domain with matching name 's-3-VM'
libvirt: QEMU Driver error : Domain not found: no domain with matching name 's-3-VM'
libvirt: QEMU Driver error : Domain not found: no domain with matching name 's-3-VM'
libvirt: QEMU Driver error : Domain not found: no domain with matching name 's-3-VM'
libvirt: QEMU Driver error : Domain not found: no domain with matching name 'v-4-VM'
libvirt: error : no connection driver available for lxc:///
libvirt: QEMU Driver error : Domain not found: no domain with matching name 'v-4-VM'
libvirt: QEMU Driver error : Domain not found: no domain with matching name 'v-4-VM'
libvirt: QEMU Driver error : Domain not found: no domain with matching name 'v-4-VM'
libvirt: QEMU Driver error : Domain not found: no domain with matching name 'v-4-VM'
libvirt: QEMU Driver error : Domain not found: no domain with matching name 'v-4-VM'
libvirt: QEMU Driver error : Domain not found: no domain with matching name 'v-4-VM'
libvirt: error : no connection driver available for lxc:///
libvirt: QEMU Driver error : Domain not found: no domain with matching name 'v-4-VM'
libvirt: QEMU Driver error : Domain not found: no domain with matching name 'v-4-VM'
libvirt: QEMU Driver error : Domain not found: no domain with matching name 'v-4-VM'
libvirt: QEMU Driver error : Domain not found: no domain with matching name 'v-4-VM'
libvirt: QEMU Driver error : Domain not found: no domain with matching name 'v-4-VM'

libvirt: Storage Driver error : Storage pool not found: no storage pool with matching uuid '47f2ec3c-17be-3cc8-aa3b-ceaad37f06ac'
libvirt: error : internal error: Child process (/usr/bin/mount -o nodev,nosuid,noexec 10.144.9.99:/kvm02/secondary/template/tmpl/1/3 /mnt/47f2ec3c-17be-3cc8-aa3b-ceaad37f06ac) unexpected exit status 32: mount.nfs: mounting 10.144.9.99:/kvm02/secondary/template/tmpl/1/3 failed, reason given by server: No such file or directory
libvirt: QEMU Driver error : Domain not found: no domain with matching name 's-3-VM'
libvirt: error : no connection driver available for lxc:///
libvirt: QEMU Driver error : Domain not found: no domain with matching name 's-3-VM'
libvirt: QEMU Driver error : Domain not found: no domain with matching name 's-3-VM'
libvirt: QEMU Driver error : Domain not found: no domain with matching name 's-3-VM'
libvirt: QEMU Driver error : Domain not found: no domain with matching name 's-3-VM'
libvirt: QEMU Driver error : Domain not found: no domain with matching name 's-3-VM'
libvirt: QEMU Driver error : Domain not found: no domain with matching name 'v-4-VM'
libvirt: error : no connection driver available for lxc:///
libvirt: QEMU Driver error : Domain not found: no domain with matching name 'v-4-VM'
libvirt: QEMU Driver error : Domain not found: no domain with matching name 'v-4-VM'
libvirt: QEMU Driver error : Domain not found: no domain with matching name 'v-4-VM'
libvirt: QEMU Driver error : Domain not found: no domain with matching name 'v-4-VM'
libvirt: QEMU Driver error : Domain not found: no domain with matching name 'v-4-VM'
libvirt: QEMU Driver error : Domain not found: no domain with matching name 'v-4-VM'
libvirt: error : no connection driver available for lxc:///
libvirt: QEMU Driver error : Domain not found: no domain with matching name 'v-4-VM'
libvirt: QEMU Driver error : Domain not found: no domain with matching name 'v-4-VM'
libvirt: QEMU Driver error : Domain not found: no domain with matching name 'v-4-VM'
libvirt: QEMU Driver error : Domain not found: no domain with matching name 'v-4-VM'
libvirt: QEMU Driver error : Domain not found: no domain with matching name 'v-4-VM'

This is checking from NFS server

root@nfs:# ls -al /kvm01/secondary/
total 4
drwxrwxrwt 5 root root 54 Jul 21 15:06 .
drwxr-xr-x 4 root root 4096 Jul 21 14:21 ..
drwxrwxrwx 2 root root 6 Jul 21 15:06 snapshots
drwxr-xr-x 3 root root 18 Jul 21 15:02 template
drwxrwxrwx 2 root root 6 Jul 21 15:06 volumes
root@nfs:# ls -al /kvm02/secondary/
total 8
drwxr-xr-x 2 root root 4096 Jul 21 12:17 .
drwxr-xr-x 4 root root 4096 Jul 21 12:17 ..
root@nfs:~#

[Pearl1594]

@verendrakalsi Does /KVMSecondary1/ path exist on the secondary store: 10.10.10.12?

[verendrakalsi]

yes, it does. now this issue is resolved.

but now i am getting this error

2025-03-10 20:32:02,961 ERROR [cloud.agent.AgentShell] (main:[]) (logid:) Unable to start agent: javax.naming.ConfigurationException: Unable to configure LibvirtComputingResource
at com.cloud.agent.Agent.(Agent.java:196)
at com.cloud.agent.AgentShell.launchNewAgent(AgentShell.java:453)
at com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:432)
at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:416)
at com.cloud.agent.AgentShell.start(AgentShell.java:512)
at com.cloud.agent.AgentShell.main(AgentShell.java:542)

[Pearl1594]

Can you please share the entire stacktrace. Thanks

[DaanHoogland]

@verendrakalsi , the stacktrace as a "Cause", this is what @Pearl1594 is interested in.

Also can you add the log lines leading up to this stacktrace. Especially the ones containing LibvirtComputingResource or AgentShell?

[verendrakalsi]

this is what you are looking for ?

2025-03-11 14:11:17,825 INFO [cloud.agent.AgentShell] (main:[]) (logid:) Agent started
2025-03-11 14:11:17,830 INFO [cloud.agent.AgentShell] (main:[]) (logid:) Implementation Version is 4.20.0.0
2025-03-11 14:11:17,831 INFO [cloud.agent.AgentShell] (main:[]) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties
2025-03-11 14:11:17,870 INFO [cloud.agent.AgentShell] (main:[]) (logid:) Defaulting to using properties file for storage
2025-03-11 14:11:17,871 INFO [cloud.agent.AgentShell] (main:[]) (logid:) Defaulting to the constant time backoff algorithm
2025-03-11 14:11:17,895 INFO [cloud.utils.LogUtils] (main:[]) (logid:) log4j configuration found at /etc/cloudstack/agent/log4j-cloud.xml
2025-03-11 14:11:17,895 INFO [cloud.agent.AgentShell] (main:[]) (logid:) Using default Java settings for IPv6 preference for agent connection
2025-03-11 14:11:17,948 INFO [cloud.agent.Agent] (main:[]) (logid:) id is 0
2025-03-11 14:11:17,953 WARN [kvm.resource.LibvirtComputingResource] (main:[]) (logid:) Incorrect details for private Nic during initialization of ServerResourceBase
2025-03-11 14:11:17,953 ERROR [cloud.agent.AgentShell] (main:[]) (logid:) Unable to start agent: javax.naming.ConfigurationException: Unable to configure LibvirtComputingResource
at com.cloud.agent.Agent.(Agent.java:196)
at com.cloud.agent.AgentShell.launchNewAgent(AgentShell.java:453)
at com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:432)
at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:416)
at com.cloud.agent.AgentShell.start(AgentShell.java:512)
at com.cloud.agent.AgentShell.main(AgentShell.java:542)

[verendrakalsi]

2025-03-11T20:53:36,178 WARN [cloud.agent.Agent] (agentRequest-Handler-1:[]) Caught: com.cloud.utils.exception.CloudRuntimeException: Failed to get root directory from secondary storage URL [nfs://10.10.10.12/KVMSecondary1], using NFS version [null], due to [Unable to mount 10.10.10.12:/KVMSecondary1 at /mnt/SecStorage/bd7fe518-f806-3eee-92f9-0819535406ae due to mount.nfs: Connection timed out].
at org.apache.cloudstack.storage.resource.NfsSecondaryStorageResource.getRootDir(NfsSecondaryStorageResource.java:2675)
at org.apache.cloudstack.storage.resource.NfsSecondaryStorageResource.execute(NfsSecondaryStorageResource.java:2407)
at org.apache.cloudstack.storage.resource.NfsSecondaryStorageResource.executeRequest(NfsSecondaryStorageResource.java:298)
at com.cloud.storage.resource.PremiumSecondaryStorageResource.defaultAction(PremiumSecondaryStorageResource.java:62)
at com.cloud.storage.resource.PremiumSecondaryStorageResource.executeRequest(PremiumSecondaryStorageResource.java:58)
at com.cloud.agent.Agent.processRequest(Agent.java:686)
at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1109)
at com.cloud.utils.nio.Task.call(Task.java:83)
at com.cloud.utils.nio.Task.call(Task.java:29)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: com.cloud.utils.exception.CloudRuntimeException: Unable to mount 10.10.10.12:/KVMSecondary1 at /mnt/SecStorage/bd7fe518-f806-3eee-92f9-0819535406ae due to mount.nfs: Connection timed out
at org.apache.cloudstack.storage.resource.NfsSecondaryStorageResource.attemptMount(NfsSecondaryStorageResource.java:3081)
at org.apache.cloudstack.storage.resource.NfsSecondaryStorageResource.mount(NfsSecondaryStorageResource.java:3032)
at org.apache.cloudstack.storage.resource.NfsSecondaryStorageResource.mountUri(NfsSecondaryStorageResource.java:3020)
at org.apache.cloudstack.storage.resource.NfsSecondaryStorageResource.getRootDir(NfsSecondaryStorageResource.java:2671)
... 12 more

2025-03-11T20:54:36,351 INFO [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:[]) Determined host 10.10.10.12 corresponds to IP 10.10.10.12

[DaanHoogland]

@verendrakalsi , there still seems to be an issue with your NFS mount ; "Unable to mount 10.10.10.12:/KVMSecondary1 at /mnt/SecStorage/bd7fe518-f806-3eee-92f9-0819535406ae due to mount.nfs: Connection timed out"

[verendrakalsi]

But i do see secondary storage mounted to the management server and i can access it

10.10.10.12:/KVMSecondary1 251G 454M 250G 1% /var/lib/cloudstack/mnt/253023765254217.61fa6fda
root@host:/home/kvmadmin# cd /var/lib/cloudstack/mnt/
root@host:/var/lib/cloudstack/mnt# ls
202872835190222.29e9619d 253023765254217.61fa6fda
root@host:/var/lib/cloudstack/mnt# cd 253023765254217.61fa6fda/
root@host:/var/lib/cloudstack/mnt/253023765254217.61fa6fda# ls
snapshots template volumes
root@host:/var/lib/cloudstack/mnt/253023765254217.61fa6fda#

[verendrakalsi]

Also now in ssystem VMs we are getting this error
/mnt/SecStorage/ca7ab41c-ce2a-3350-a7e9-88a4e20bc784/template/tmpl/1/4/dnld7827224589018529675tmp_, bytes=(0 bytes) 0, error=No route to host (Host unreachable), pct=0

[weizhouapache]

can you log into ssvm and check if the secondary storage can be mounted ?

what are the IPs and routes in SSVM ?

[verendrakalsi]

i can see the secondary storage attached to the ssvm

root@s-9-VM:~# df -h
Filesystem Size Used Avail Use% Mounted on
udev 214M 0 214M 0% /dev
tmpfs 93M 492K 92M 1% /run
/dev/vda6 4.2G 1.3G 2.9G 32% /
tmpfs 231M 0 231M 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
/dev/vda1 366M 46M 321M 13% /boot
10.43.197.12:/KVMSecondary1 251G 455M 250G 1% /mnt/SecStorage/ca7ab41c-ce2a-3350-a7e9-88a4e20bc784
tmpfs 47M 0 47M 0% /run/user/0

[verendrakalsi]

default via 10.43.197.1 dev eth2
10.43.197.0/24 dev eth1 proto kernel scope link src 10.43.197.147
10.43.197.0/24 dev eth2 proto kernel scope link src 10.43.197.171
10.63.8.52 via 10.43.197.1 dev eth1
169.254.0.0/16 dev eth0 proto kernel scope link src 169.254.105.132

[weizhouapache]

I wonder why the ssvm tried to mount

10.10.10.12:/KVMSecondary1 

which has different IP as the secondary storage already mounted

10.43.197.12:/KVMSecondary1 251G 455M 250G 1% /mnt/SecStorage/ca7ab41c-ce2a-3350-a7e9-88a4e20bc784

[verendrakalsi]

irrespective of local or URL.

for load, we are directly getting that error. But URL shows as if it is doing something but in the logs we get error

[weizhouapache]

irrespective of local or URL.

for load, we are directly getting that error. But URL shows as if it is doing something but in the logs we get error

for upload from local, if you have not configured SSL certificate and domains, you can change global setting use.https.to.upload to false, restart mgmt server and destroy ssvm, then retry

for upload via URL, you can find the information in /var/log/cloud.log in ssvm

[verendrakalsi]

Those setting is already done, but we still have same issue.

[weizhouapache]

Those setting is already done, but we still have same issue.

can you please check if the public IP of ssvm is accessible from your local server ?

I tested ok.

[DaanHoogland]

@verendrakalsi , did you make any progress on this?

[DaanHoogland]

@verendrakalsi , assiming this is resolved. please comment if you still have an issue.