Axis integration broken #46815
Assignees
Labels
Comments
|
axis documentation |
|
Here is a full debug log. The previous log did not show response from the camera. |
|
I've been working in HASS all day loading/reloading. I got the integration to load once, but on the next HASS reload the integration stopped again. |
|
That there are empty lines after [axis.vapix] indicates something is indeed wrong. Could you try using the library directly to see if you get any additional information would be helpful, I will try some more out this evening at home. python3 -m axis 192.168.0.90 root pass --params -D |
|
I'm not able to duplicate the problem this morning. I will try again tonight. Some problems can be created if restarting HASS while logged into the camera WebUI. When this occurred the add-on crashes, but will successfully start after closing the WebUI. I think the camera can only handle one stream at a time. But this does not explain the issues I've had for the past 3 or 4 days. I will test more later. |
|
Well its good that its working again, try some more combinations out if you can figure out whats causing the issue |
|
It happened again after a server reboot. The integration would not work for 20 minutes, so I ran the python lib directly and didn't see anything wonky. So I reloaded HASS and it started working. Here is the output from the python script, but as I said it looks ok. |
|
And it doesnt automatically try to set it up at a later time? Thats what I'd expect. The Script and the integration does the same calls so I wonder why the behavior between them differs, I wonder if its the session manager in home assistant (which is the only real difference but shouldnt affect either) |
|
For yesterday, yes it tried to set up repeatedly. But it only started working after restarting Core. This morning the camera was still working, until I restarted Core. So far it has tried to setup 11 times with no success (log file attached). Next I tested the python lib directly which shows some timeout errors (see attachment). I think camera is taking too long to respond? |
|
It now it's working again. All I did was update HASS OS to 5.12 and reload...... |
|
I wonder if it's related to starting order of integrations, since it is random |
|
I realise that I have been given incorrect information so have temporarily removed my comments so that I can go through and check everything again before reposting the correct information |
|
OK so updated information. My config
ISSUE I removed it and tried to re-add it back in, which failed. I was able to manually add it back in. I did this by comparing the config of other axis cameras in "core.config_entries" & "core.device_registry", an old backup copy of both files, and copying the style from one of the other cameras. E.G. Logs With "version" set to 2, image is shown and the entities work fine. With "version" set to 3, no image is shown and the entities report as unavailable. Log when I have removed the camera via the GUI, rebooted Core, then try adding it back in. It fails with the following log. Log from running |
|
Thorough investigation! I wonder why it doesn't work if it has v3 in the config entry. If it is v2 it will do a migration of entity unique ids since the MAC address format have been standardized in the Axis integration. 🤔 From what I can see it looks like it could be a timeout, but it is only 5 seconds between first and last log item, and the timeout is 15 seconds. Im confused |
|
Mine has the v3 entry as well. What I'm struggling with is the inconsistency of the problem. Some days I'm unable to get the integration to work. Other days I can't make it fail. Today is one of those days where it won't fail. |
|
Can you try disabling your other integrations and see if it goes to a more consistent behavior? |
|
What does the hass log say in regards to config entry not ready? The error is passed to it, so it would be good to see what it says |
|
I don't think I've seen that in the logs. Additionally I haven't seen any errors in the past 2 days. In that time I've reloaded HASS Core roughly 20 times (this morning 10 times for sure). In that time I've added HACS and one HACS integration. |
|
That's the kind of issue I don't like. That means it can come back whenever |
|
The camera worked great for 6 days, but the problem returned after applying the latest HASS update. I'm seeing a lot of "HTTP/1.1 401 Unauthorized" related to the camera IP address, followed by "HTTP/1.1 200 OK". I wonder if the integration is spamming the camera with too many requests. |
|
Update. The errors continued for 3 hours last night, and then cleared up in the middle of the night. The camera has been working for 4 hours now. |
401 are common because all requests try first as basic, a 401 signals to try digest authentication which will then succeed |
This is crazy confusing |
|
I have the same issue with my Axis M3027 camera running the latest firmware. I have some another point of view to provide, due to the fact that I have two HA instances running on my network because I'm slowly migrating from core-0.114.4 in a non-supported Synology HassOs version (from synocommunity) and another one running as an official virtual machine on my synology nas, with the latest HassOS. The issue is present only in the newest homeassistant, while in the old one the same camera works without issues. I did these tests:
By looking at the logs, this is what I see. As a sidenote, if it can be helpful, I installed the "new" homeassistant with an older version (mid-january) and the camera was working there. I then updated this homeassistant to some more recent version (in february) and, since then, is has stopped working. I then removed this camera integration to re-add it, hoping it would fix the issue, but it just became impossible to be added again. I'm no expert in Axis cameras, but it looks to be related to note: the log is not sequentially "correct" in time, because I got it from the ui in different moments, due to the fact that I had to "copy paste" different chunks, having a lot of cameras. If it helps, I can add a more complete log next time. |
|
That API discovery is not available is an expected situation and 404 errors are handled appropriately https://github.com/Kane610/axis/blob/f8d17078b9b2ac6f49f2a5713586c84657557775/axis/vapix.py#L127 Is it only older devices running firmwares ~6.50 and older that are misbehaving? There was a major refactoring of the Axis library introduced with 2021.2, I did test a lot but of course something could have been missed. So if you don't see any issues in 2021.1 but do in 2021.2, we have something to go on |
|
@Kane610 on my network, only older devices with firmware 6.50 are not working... All the others are doing great. I cannot speak for 2021.1 because I don't have it anywhere, but for sure in 0.114.4 all cameras are still working correctly. I could try to downgrade my 2021.3 to 2021.1 and see if something changes, if it's useful! |
|
Yes please try a the last few releases up to the last one last year. I think that will help a lot So a matrix between hass releases and behaviors of devices with different firmware versions |
|
Did a test on 2021.1.0 and 2021.1.5, I was able to add the AXIS M3027 camera with the integration. It works! I don't have time this evening to test all versions from this to the actual, but I can do it in the next days. If you have some that may be cooler to test first, please advise, to optimize the test :) |
|
Ok, so the same device doesn't work properly on later releases of hass? If thats the case I have to get ahold of one of those devices to pin point the exact issue. |
|
I can confirm that since 2021.2.0 cameras with firmware 6.50 cease to work: up to 2021.1.5 everything is ok! |
|
I wouldn't say they don't work. Mine is working right now. But if I reboot my HASS (or do an upgrade), it might not work for hours or days... and then start working again. I might try an upgrade right now, and report back the results. |
|
So I upgraded to 2021.3.4 and the camera worked immediately. Then for giggles I rebooted the machine, and now the camera won't work. Lol. UPDATE.... after waiting 7 minutes with no results, I restarted HASS and the camera started working immediately. This is consistent with my experiences since I started this thread. You never know if it will work or not, and results are not repeatable/predictable. Once I get the camera working, it works well until a reboot/reload... and then it may or may not work. |
|
My AXIS M3027 can't be added to homeassistant after 2021.2.0 . It just says "fail to connect". If I roll back to 2021.1.5, I can add it and everything works... My case looks different :( I haven't tried adding the camera in 2021.1.5 and then upgrading to 2021.2.0+ . I've seen that it just went to "unavailable", but I haven't verified if @michaelbaldy 's behavior applies also for my case. @michaelbaldy , if you remove your camera integration from homeassistant, are you able to add it back in? |
|
I've had that happen too. Keep reloading/rebooting. :) |
|
One thing does seem to make it worse.. If I reload/reboot HASS while logged into the camera's webUI, I'm guaranteed a failure, and it takes a very long time to get things working again. I'm caught in that loop right now. Reload hass 1: fail |
|
|
|
Reload hass 8: fail |
|
I've gotten a hold of an M3027 and gonna try to reproduce this issue. Any special configuration of the device I should know about? Or you have more or less just set the username and password? |
|
A pretty vanilla configured M3027 doesn't show any issue with being paired nor reconnected on a reboot. Could it be a time out issue? If you look at the debug log prints, is it spanning longer than 15 seconds perhaps? |
|
I just have the motion trigger app installed, but I could remove it and see if it works or not without! By looking at my logs (published here above), it seems to fail in ~13s! I could also try to "vanillize" my M3027, to give you some feedback on when it stops working. I'll get back to you in the WE! |
|
Thanks, it seems to work solid here. It would be nice to compare the exact output of the logs to see what differ. |
|
My M3027 is setup running their built-in motion application 4.2-3, with motion events saving video to a network share. The cameras been working fine, so here is what I did to make the camera fail:
Now that it has failed, I can review the error log in detail. Each error comes 6 seconds after communication starts. The amount of data retrieved before errors will vary. Here is the last data collected before each error: Error 1 at root.Brand.WebURL=http://www.axis.com/ Error 2 at root.IOPort.I1.Output.PulseTime=0 Error 3 at root.IOPort.I1.Output.PulseTime=0 Error 4 at root.Properties.VirtualInput.VirtualInput=yes Error 5 at root.Output.NbrOfOutputs=1 Error 6 at root.IOPort.I1.Output.PulseTime=0 Error 7 at root.Output.NbrOfOutputs=1 Error 8 at root.Properties.VirtualInput.VirtualInput=yes Error 9 at root.StreamProfile.S1.Parameters=resolution=1920x720... |
|
I do think the camera is getting overwhelmed by 7 simultaneous HTTP commands. I suspect it might work better if each HTTP request was delayed until the prior request completed. 2021-04-01 08:22:07 DEBUG (MainThread) [axis.vapix] http://192.168.0.21:80/axis-cgi/param.cgi?action=list&group=root.Properties {} |
|
I also think there should be a long delay between ERROR and starting again. I don't think camera has time to recover from SPAM before the next attempt. Maybe increase it from 20 seconds to 60 seconds. I think running video motion detection on the camera is CPU intense, and too many HTTP requests just puts it over the edge. |
|
I turned off VMD and HASS immediately connected. |
|
I think we're getting somewhere. With VMD disabled, I rebooted HASS 6 times, and the camera worked each time. With VMD enabled, I've rebooted HASS 3 times, and the camera fails. Then I turned off VMD, so that HASS can connect. Then I turn VMD back on and everything is OK :) Definitely too many http requests coming through, in combination with onboard VMD, overwhelming the CPU. |
|
I enabled both motion detection apps, and it definitely becomes slower but I still don't see your issue. This is running on dev environment on my macbook pro, maybe you have a less powerful unit? |
|
@Kane610 have you tried removing and re-adding the camera with the motion detection apps enabled? I'm running HA on a recent Synology NAS (don't remember right now the model, but it can handle the transcoding of two 4k videos) and it's currently running two separate instances of HA: one in docker and one in virtual machine. The one that runs in docker has no issues, because is an older version. The one that runs in virtual machine, works up to 2021.1.5. It's not running on a macbook pro, but I don't think it's a performance issue on the server side. If I have time, this evening I'll wipe out the current configuration of my M3027 and see if something works. |
|
Indeed that actually did trigger it, great! Now I can delve into it :) After it failed I removed the Video motion detection v3 app and I could add it |
|
I did some tests:
This is my log when I failed with motion detect v4: |
|
Glad you guys are able to duplicate the issue. I don't think it's a problem with my PC. I'm running an I3-6100. CPU load is 6%. Memory load is 9%. Plenty of hard drive space. |
|
Update: I've completely reset the camera to a full vanilla version, with just a custom password. I was able to add it perfectly and was also able to see its internal sensors. Before resetting the camera, the "daynight" sensor wasn't available, even with the motion sensor disabled. The feed is now smooth and continuous. As soon as I have some time, I'll repeat some tests with motion detection enabled. @michaelbaldy yeah I also think that the server performances are not affecting the issue! edit: I add a screenshot of my 3026 camera, that has not been reset. As visible, all the internal sensors are not available. |
|
So I'm progressing, there is a lower level time out hitting which seems to be the cause. It is only 5 seconds so I will increase that to 15 seconds |
21 tasks
|
PR is up, thank you for your excellent support ❤️ Please verify the beta |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
The problem
I've been using the Axis integration for two months. After a reboot a few days ago the integration no longer connects to the camera. I get a "failed to connect" error on the flow setup page. I've tried deleting the integration and rebuilding. I'm not sure if this a bug, or just an error on my system. I have included a copy of the debug log for the axis integration.
The camera is Axis M3027 running firmware 6.50.5.3.
I'm running the official HassOS on a NUC.
The only other integrations/addons that I'm running are:
What is version of Home Assistant Core has the issue?
core-2021.2.3
What was the last working version of Home Assistant Core?
core-2021.1.5
What type of installation are you running?
Home Assistant OS
Integration causing the issue
Axis
Link to integration documentation on our website
https://www.home-assistant.io/integrations/axis/
Example YAML snippet
# Put your YAML below this lineAnything in the logs that might be useful for us?
# Put your logs below this line 2021-02-20 07:29:40 DEBUG (MainThread) [axis.api] {} 2021-02-20 07:29:40 DEBUG (MainThread) [axis.vapix] http://192.168.0.21:80/axis-cgi/apidiscovery.cgi {'json': {'method': 'getApiList', 'apiVersion': '1.0', 'context': 'Axis library'}} 2021-02-20 07:29:40 DEBUG (MainThread) [axis.vapix] <Response [404 Not Found]>, 404 Client Error: Not Found for url: http://192.168.0.21:80/axis-cgi/apidiscovery.cgi For more information check: https://httpstatuses.com/404 2021-02-20 07:29:40 DEBUG (MainThread) [axis.api] '' 2021-02-20 07:29:40 DEBUG (MainThread) [axis.vapix] http://192.168.0.21:80/axis-cgi/param.cgi?action=list&group=root.Properties {} 2021-02-20 07:29:40 DEBUG (MainThread) [axis.vapix] http://192.168.0.21:80/axis-cgi/param.cgi?action=list&group=root.PTZ {} 2021-02-20 07:29:40 DEBUG (MainThread) [axis.vapix] http://192.168.0.21:80/axis-cgi/param.cgi?action=list&group=root.Brand {} 2021-02-20 07:29:40 DEBUG (MainThread) [axis.vapix] http://192.168.0.21:80/axis-cgi/param.cgi?action=list&group=root.StreamProfile {} 2021-02-20 07:29:40 DEBUG (MainThread) [axis.vapix] http://192.168.0.21:80/axis-cgi/param.cgi?action=list&group=root.Input {} 2021-02-20 07:29:40 DEBUG (MainThread) [axis.vapix] http://192.168.0.21:80/axis-cgi/param.cgi?action=list&group=root.IOPort {} 2021-02-20 07:29:40 DEBUG (MainThread) [axis.vapix] http://192.168.0.21:80/axis-cgi/param.cgi?action=list&group=root.Output {} 2021-02-20 07:29:43 DEBUG (MainThread) [axis.vapix] Response: root.StreamProfile.MaxGroups=26 root.StreamProfile.S0.Description= root.StreamProfile.S0.Name=1 root.StreamProfile.S0.Parameters=resolution=2592x1944&compression=30&mirror=0&textsize=medium&textcolor=white&textbackgroundcolor=black&textposition=top&text=0&clock=1&date=1&fps=0&maxframesize=0&videocodec=jpeg from 192.168.0.21 2021-02-20 07:29:44 DEBUG (MainThread) [axis.vapix] Response: root.Brand.Brand=AXIS root.Brand.ProdFullName=AXIS M3027 Network Camera root.Brand.ProdNbr=M3027 root.Brand.ProdShortName=AXIS M3027 root.Brand.ProdType=Network Camera root.Brand.ProdVariant= root.Brand.WebURL=http://www.axis.com/ from 192.168.0.21 2021-02-20 07:29:45 DEBUG (MainThread) [axis.vapix] Response: root.Output.NbrOfOutputs=1 from 192.168.0.21 2021-02-20 07:29:45 DEBUG (MainThread) [axis.vapix] Response: root.Input.NbrOfInputs=1 from 192.168.0.21 2021-02-20 07:29:45 DEBUG (MainThread) [axis.vapix] Response: root.IOPort.I0.Configurable=no root.IOPort.I0.Direction=input root.IOPort.I0.Input.Name=Input 1 root.IOPort.I0.Input.Trig=closed root.IOPort.I1.Configurable=no root.IOPort.I1.Direction=output root.IOPort.I1.Output.Active=closed root.IOPort.I1.Output.Button=none root.IOPort.I1.Output.DelayTime=0 root.IOPort.I1.Output.Mode=bistable root.IOPort.I1.Output.Name=Output 1 root.IOPort.I1.Output.PulseTime=0 from 192.168.0.21 2021-02-20 07:29:46 DEBUG (MainThread) [axis.vapix] 2021-02-20 07:29:46 ERROR (MainThread) [homeassistant.components.axis] Error connecting to the Axis device at 192.168.0.21 2021-02-20 07:29:46 DEBUG (MainThread) [axis.vapix]The text was updated successfully, but these errors were encountered: