8000 OCP `4.18.10` bundle not able to start again after a stop · Issue #1050 · crc-org/snc · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

OCP 4.18.10 bundle not able to start again after a stop #1050

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
anjannath opened this issue Apr 28, 2025 · 6 comments
Open

OCP 4.18.10 bundle not able to start again after a stop #1050

anjannath opened this issue Apr 28, 2025 · 6 comments
Assignees
@praveenkumar

Comments

@anjannath
Copy link
Member

When using the bundle with crc SSH doesn't become available when we crc start again after the initial start and crc stop, in the Console logs there are error messages saying Failed to find a suitable Node IP
@albfan
Copy link
albfan commented May 6, 2025 •
edited
Loading

Can confirm with 4.18.12 is still the same.

Check with 4.18.2 and it works correctly

windows:

DEBU retry loop: attempt 71
DEBU Running SSH command: exit 0
DEBU Using ssh private keys: [C:\Users\crcqe\.crc\machines\crc\id_ed25519 C:\Users\crcqe\.crc\cache\crc_hyperv_4.18.12_amd64\id_ecdsa_crc]
DEBU SSH command results: err: ssh: handshake failed: read tcp 127.0.0.1:55329->127.0.0.1:2222: wsarecv: An existing connection was forcibly closed by the remote host., output:  
DEBU error: Temporary error: ssh command error:
command : exit 0
err     : ssh: handshake failed: read tcp 127.0.0.1:55329->127.0.0.1:2222: wsarecv: An existing connection was forcibly closed by the remote host.
 - sleeping 1s
DEBU RetryAfter timeout after 72 tries
DEBU Running '(Get-ItemProperty -Path "HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion" -Name ReleaseId).ReleaseId' 
DEBU Running '(Get-ItemProperty -Path "HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion").EditionID'

linux:

 - sleeping 1s 
DEBU retry loop: attempt 31                       
DEBU Running SSH command: exit 0                  
DEBU Using ssh private keys: [/home/cloud-user/.crc/machines/crc/id_ed25519 /home/cloud-user/.crc/cache/crc_libvirt_4.18.12_amd64/id_ecdsa_crc] 
DEBU SSH command results: err: ssh: handshake failed: read tcp 127.0.0.1:48104->127.0.0.1:2222: read: connection reset by peer, output:  
DEBU error: Temporary error: ssh command error:
command : exit 0
err     : ssh: handshake failed: read tcp 127.0.0.1:48104->127.0.0.1:2222: read: connection reset by peer
 - sleeping 1s

@albfan
Copy link
albfan commented May 6, 2025

I was unable to make it fail on mac

@albfan
Copy link
albfan commented May 7, 2025

Today I was unable to make windows and linux to fail.

Tried again on mac and see error only once. stop/start and see error twice:

DEBU Running /usr/local/crc/vfkit --cpus 4 --memory 10752 --bootloader efi,variable-store=/Users/crcqe/.crc/machines/crc/efistore.nvram,create --device virtio-serial,logFilePath=/Users/crcqe/.crc/machines/crc/vfkit.log --device virtio-fs,sharedDir=/Users/crcqe,mountTag=dir0 --device virtio-rng --device virtio-blk,path=/Users/crcqe/.crc/machines/crc/crc.img --device virtio-net,unixSocketPath=/Users/crcqe/.crc/crc-unixgram.sock,mac=5a:94:ef:e4:0c:ee --timesync vsockPort=1234 
DEBU time="2025-05-07T10:46:19+02:00" level=info msg="&{4 10752    {[efi variable-store=/Users/crcqe/.crc/machines/crc/efistore.nvram create] true} vsockPort=1234 [virtio-serial,logFilePath=/Users/crcqe/.crc/machines/crc/vfkit.log virtio-fs,sharedDir=/Users/crcqe,mountTag=dir0 virtio-rng virtio-blk,path=/Users/crcqe/.crc/machines/crc/crc.img virtio-net,unixSocketPath=/Users/crcqe/.crc/crc-unixgram.sock,mac=5a:94:ef:e4:0c:ee] none://  false  {[] false}}" 
DEBU time="2025-05-07T10:46:19+02:00" level=info msg="boot parameters: &{EFIVariableStorePath:/Users/crcqe/.crc/machines/crc/efistore.nvram CreateVariableStore:true}" 
DEBU time="2025-05-07T10:46:19+02:00" level=info  
DEBU time="2025-05-07T10:46:19+02:00" level=info msg="virtual machine parameters:" 
DEBU time="2025-05-07T10:46:19+02:00" level=info msg="\tvCPUs: 4" 
DEBU time="2025-05-07T10:46:19+02:00" level=info msg="\tmemory: 10752 MiB" 
DEBU time="2025-05-07T10:46:19+02:00" level=info  
DEBU time="2025-05-07T10:46:19+02:00" level=info msg="Adding virtio-serial device (logFile: /Users/crcqe/.crc/machines/crc/vfkit.log)" 
DEBU time="2025-05-07T10:46:19+02:00" level=info msg="Adding virtio-fs device" 
DEBU time="2025-05-07T10:46:19+02:00" level=info msg="Adding virtio-rng device" 
DEBU time="2025-05-07T10:46:19+02:00" level=info msg="Adding virtio-blk device (imagePath: /Users/crcqe/.crc/machines/crc/crc.img)" 
DEBU time="2025-05-07T10:46:19+02:00" level=info msg="Adding virtio-net device (nat: false macAddress: [5a:94:ef:e4:0c:ee])" 
DEBU time="2025-05-07T10:46:19+02:00" level=info msg="Using unix socket /Users/crcqe/.crc/crc-unixgram.sock" 
DEBU time="2025-05-07T10:46:19+02:00" level=info msg="local: /Users/crcqe/.crc/vfkit-17fa3-efd5.sock remote: /Users/crcqe/.crc/crc-unixgram.sock" 
DEBU time="2025-05-07T10:46:19+02:00" level=info msg="Adding virtio-vsock device" 
DEBU time="2025-05-07T10:46:19+02:00" level=info msg="virtual machine is running" 
DEBU time="2025-05-07T10:46:19+02:00" level=info msg="Setting up host/guest time synchronization" 
DEBU time="2025-05-07T10:46:19+02:00" level=info msg="waiting for VM to stop" 
DEBU Waiting for machine to be running, this may take a few minutes... 
DEBU retry loop: attempt 0                        
DEBU Machine is up and running!                   
INFO CRC instance is running with IP 127.0.0.1    
DEBU Waiting until ssh is available               
DEBU retry loop: attempt 0                        
DEBU Running SSH command: exit 0                  
DEBU Using ssh private keys: [/Users/crcqe/.crc/machines/crc/id_ed25519 /Users/crcqe/.crc/cache/crc_vfkit_4.18.12_amd64/id_ecdsa_crc] 
DEBU SSH command results: err: ssh: handshake failed: read tcp 127.0.0.1:59676->127.0.0.1:2222: read: connection reset by peer, output:  
DEBU error: Temporary error: ssh command error:
command : exit 0
err     : ssh: handshake failed: read tcp 127.0.0.1:59676->127.0.0.1:2222: read: connection reset by peer
 - sleeping 1s 
DEBU retry loop: attempt 1                        
DEBU Running SSH command: exit 0                  
DEBU Using ssh private keys: [/Users/crcqe/.crc/machines/crc/id_ed25519 /Users/crcqe/.crc/cache/crc_vfkit_4.18.12_amd64/id_ecdsa_crc] 
DEBU SSH command results: err: ssh: handshake failed: read tcp 127.0.0.1:59677->127.0.0.1:2222: read: connection reset by peer, output:  
DEBU error: Temporary error: ssh command error:
command : exit 0
err     : ssh: handshake failed: read tcp 127.0.0.1:59677->127.0.0.1:2222: read: connection reset by peer
 - sleeping 1s 
DEBU retry loop: attempt 2                        
DEBU Running SSH command: exit 0                  
DEBU Using ssh private keys: [/Users/crcqe/.crc/machines/crc/id_ed25519 /Users/crcqe/.crc/cache/crc_vfkit_4.18.12_amd64/id_ecdsa_crc] 
DEBU SSH command results: err: <nil>, output:     
INFO CRC VM is running                            
DEBU Using root access: disable core user password 
DEBU Running SSH command: sudo passwd --lock core 
DEBU SSH command results: err: <nil>, output: Locking password for user core.
passwd: Success

second time:

% crc start -b crc_vfkit_4.18.12_amd64.crcbundle -p tmp/pull-secret --log-level debug
DEBU CRC version: 25.05.07+83a415                 
DEBU OpenShift version: 4.18.2                    
DEBU MicroShift version: 4.18.2                   
DEBU Running 'crc start'                          
DEBU Total memory of system is 34359738368 bytes  
WARN Using crc_vfkit_4.18.12_amd64.crcbundle bundle, but crc_vfkit_4.18.2_amd64.crcbundle is expected for this release 
DEBU Unable to find out if a new version is available: Invalid Semantic Version 
INFO Using bundle path crc_vfkit_4.18.12_amd64.crcbundle 
INFO Checking if running macOS version >= 13.x    
DEBU Running 'sw_vers -productVersion'            
DEBU sw_vers -productVersion is: 15.1             
INFO Checking if running as non-root              
INFO Checking if crc-admin-helper executable is cached 
INFO Checking if running on a supported CPU architecture 
DEBU GOARCH is amd64 GOOS is darwin               
INFO Checking if crc executable symlink exists    
INFO Checking minimum RAM requirements            
DEBU Total memory of system is 34359738368 bytes  
INFO Check if Podman binary exists in: /Users/crcqe/.crc/bin/oc 
INFO Checking if running emulated on Apple silicon 
INFO Checking if vfkit is installed               
INFO Checking if old launchd config for tray and/or daemon exists 
INFO Checking if crc daemon plist file is present and loaded 
DEBU Running 'bash -c launchctl list | grep com.redhat.crc.daemon | awk '{print $1}'' 
INFO Checking SSH port availability               
INFO Loading bundle: crc_vfkit_4.18.12_amd64...   
INFO Starting CRC VM for openshift 4.18.12...     
DEBU Updating CRC VM configuration                
DEBU Running /usr/local/crc/vfkit --cpus 4 --memory 10752 --bootloader efi,variable-store=/Users/crcqe/.crc/machines/crc/efistore.nvram --device virtio-serial,logFilePath=/Users/crcqe/.crc/machines/crc/vfkit.log --device virtio-fs,sharedDir=/Users/crcqe,mountTag=dir0 --device virtio-rng --device virtio-blk,path=/Users/crcqe/.crc/machines/crc/crc.img --device virtio-net,unixSocketPath=/Users/crcqe/.crc/crc-unixgram.sock,mac=5a:94:ef:e4:0c:ee --timesync vsockPort=1234 
DEBU time="2025-05-07T10:53:51+02:00" level=info msg="&{4 10752    {[efi variable-store=/Users/crcqe/.crc/machines/crc/efistore.nvram] true} vsockPort=1234 [virtio-serial,logFilePath=/Users/crcqe/.crc/machines/crc/vfkit.log virtio-fs,sharedDir=/Users/crcqe,mountTag=dir0 virtio-rng virtio-blk,path=/Users/crcqe/.crc/machines/crc/crc.img virtio-net,unixSocketPath=/Users/crcqe/.crc/crc-unixgram.sock,mac=5a:94:ef:e4:0c:ee] none://  false  {[] false}}" 
DEBU time="2025-05-07T10:53:51+02:00" level=info msg="boot parameters: &{EFIVariableStorePath:/Users/crcqe/.crc/machines/crc/efistore.nvram CreateVariableStore:false}" 
DEBU time="2025-05-07T10:53:51+02:00" level=info  
DEBU time="2025-05-07T10:53:51+02:00" level=info msg="virtual machine parameters:" 
DEBU time="2025-05-07T10:53:51+02:00" level=info msg="\tvCPUs: 4" 
DEBU time="2025-05-07T10:53:51+02:00" level=info msg="\tmemory: 10752 MiB" 
DEBU time="2025-05-07T10:53:51+02:00" level=info  
DEBU time="2025-05-07T10:53:51+02:00" level=info msg="Adding virtio-serial device (logFile: /Users/crcqe/.crc/machines/crc/vfkit.log)" 
DEBU time="2025-05-07T10:53:51+02:00" level=info msg="Adding virtio-fs device" 
DEBU time="2025-05-07T10:53:51+02:00" level=info msg="Adding virtio-rng device" 
DEBU time="2025-05-07T10:53:51+02:00" level=info msg="Adding virtio-blk device (imagePath: /Users/crcqe/.crc/machines/crc/crc.img)" 
DEBU time="2025-05-07T10:53:51+02:00" level=info msg="Adding virtio-net device (nat: false macAddress: [5a:94:ef:e4:0c:ee])" 
DEBU time="2025-05-07T10:53:51+02:00" level=info msg="Using unix socket /Users/crcqe/.crc/crc-unixgram.sock" 
DEBU time="2025-05-07T10:53:51+02:00" level=info msg="local: /Users/crcqe/.crc/vfkit-17fca-5395.sock remote: /Users/crcqe/.crc/crc-unixgram.sock" 
DEBU time="2025-05-07T10:53:51+02:00" level=info msg="Adding virtio-vsock device" 
DEBU time="2025-05-07T10:53:51+02:00" level=info msg="virtual machine is running" 
DEBU time="2025-05-07T10:53:51+02:00" level=info msg="Setting up host/guest time synchronization" 
DEBU time="2025-05-07T10:53:51+02:00" level=info msg="waiting for VM to stop" 
DEBU Waiting for machine to be running, this may take a few minutes... 
DEBU retry loop: attempt 0                        
DEBU Machine is up and running!                   
INFO CRC instance is running with IP 127.0.0.1    
DEBU Waiting until ssh is available               
DEBU retry loop: attempt 0                        
DEBU Running SSH command: exit 0                  
DEBU Using ssh private keys: [/Users/crcqe/.crc/machines/crc/id_ed25519 /Users/crcqe/.crc/cache/crc_vfkit_4.18.12_amd64/id_ecdsa_crc] 
DEBU SSH command results: err: ssh: handshake failed: read tcp 127.0.0.1:60272->127.0.0.1:2222: read: connection reset by peer, output:  
DEBU error: Temporary error: ssh command error:
command : exit 0
err     : ssh: handshake failed: read tcp 127.0.0.1:60272->127.0.0.1:2222: read: connection reset by peer
 - sleeping 1s 
DEBU retry loop: attempt 1     
8000
                   
DEBU Running SSH command: exit 0                  
DEBU Using ssh private keys: [/Users/crcqe/.crc/machines/crc/id_ed25519 /Users/crcqe/.crc/cache/crc_vfkit_4.18.12_amd64/id_ecdsa_crc] 
DEBU SSH command results: err: ssh: handshake failed: read tcp 127.0.0.1:60273->127.0.0.1:2222: read: connection reset by peer, output:  
DEBU error: Temporary error: ssh command error:
command : exit 0
err     : ssh: handshake failed: read tcp 127.0.0.1:60273->127.0.0.1:2222: read: connection reset by peer
 - sleeping 1s 
DEBU retry loop: attempt 2                        
DEBU Running SSH command: exit 0                  
DEBU Using ssh private keys: [/Users/crcqe/.crc/machines/crc/id_ed25519 /Users/crcqe/.crc/cache/crc_vfkit_4.18.12_amd64/id_ecdsa_crc] 
DEBU SSH command results: err: <nil>, output:     
INFO CRC VM is running                            
DEBU Using root access: disable core user password

on a third run, mac shows again only 1 ssh handshake fail.

All platforms show for several failed attempts (maybe unrelated):

 - sleeping 1s 
DEBU retry loop: attempt 27                       
DEBU Running SSH command: host -R 3 foo.apps-crc.testing 
DEBU SSH command results: err: Process exited with status 1, output: foo.apps-crc.testing has address 192.168.127.2
Host foo.apps-crc.testing not found: 3(NXDOMAIN) 
DEBU error: Temporary error: ssh command error:
command : host -R 3 foo.apps-crc.testing
err     : Process exited with status 1
 - sleeping 1s

so we can see this as some still unknown concatenation of steps leading to this errors, but not happening always

@albfan
Copy link
albfan commented May 8, 2025

Can we add extra log info on starting that ssh service so we can track if there's alreayd one running or is not able to start for some reason?

@cfergeau
Copy link
Contributor

On macOS, crc-org/crc#4735 was fixed recently, but I don’t know if this is related to the issue you are seeing here.
When err : ssh: handshake failed: read tcp 127.0.0.1:60273->127.0.0.1:2222: read: connection reset by peer occurs, I would make sure the daemon is still running, this could indicate an issue with the usermode networking stack. But this can also mean the VM failed to boot.

@praveenkumar
Copy link
Member

mac issue is now merged on crc side so this should be closed?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@praveenkumar praveenkumar

Labels
None yet
Projects
Project planning: crc
Status: No status
Milestone
No milestone
Development

No branches or pull requests
4 participants
@albfan @praveenkumar @cfergeau @anjannath
0