8000 Memory issue · Issue #6165 · home-assistant/core · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Memory issue #6165

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
CCOSTAN opened this issue Feb 22, 2017 · 44 comments · Fixed by #6314 or #6321
Closed

Memory issue #6165

CCOSTAN opened this issue Feb 22, 2017 · 44 comments · Fixed by #6314 or #6321
Labels
bug integration: discovery integration: emulated_hue integration: hue

Comments

@CCOSTAN
Copy link
Contributor
CCOSTAN commented Feb 22, 2017 •
edited
Loading

Home Assistant release (hass --version):
0.38.4

Python release (python3 --version):
3.4.2

Component/platform:
HASS / RPI3.

Description of problem:
Getting memory issue when trying to restart. This might be a local issue on my pi but just popped up with the upgrade to 38.4

Traceback (if applicable):

Feb 22 14:48:57 Carlo-Pi hass[629]: Traceback (most recent call last):
Feb 22 14:48:57 Carlo-Pi hass[629]: File "/usr/lib/python3.4/asyncio/tasks.py", line 237, in _step
Feb 22 14:48:57 Carlo-Pi hass[629]: result = next(coro)
Feb 22 14:48:57 Carlo-Pi hass[629]: File "/srv/hass/hass_venv/lib/python3.4/site-packages/homeassistant/core.py", line 10
18, in _event_to_service_call
Feb 22 14:48:57 Carlo-Pi hass[629]: yield from service_handler.func(service_call)
Feb 22 14:48:57 Carlo-Pi hass[629]: File "/srv/hass/hass_venv/lib/python3.4/site-packages/homeassistant/components/__init
__.py", line 159, in async_handle_core_service
Feb 22 14:48:57 Carlo-Pi hass[629]: yield from conf_util.async_check_ha_config_file(hass)
Feb 22 14:48:57 Carlo-Pi hass[629]: File "/srv/hass/hass_venv/lib/python3.4/site-packages/homeassistant/config.py", line
471, in async_check_ha_config_file
Feb 22 14:48:57 Carlo-Pi hass[629]: stdout=asyncio.subprocess.PIPE, loop=hass.loop)
Feb 22 14:48:57 Carlo-Pi hass[629]: File "/usr/lib/python3.4/asyncio/subprocess.py", line 226, in create_subprocess_exec

Feb 22 14:48:57 Carlo-Pi hass[629]: stderr=stderr, **kwds)
Feb 22 14:48:57 Carlo-Pi hass[629]: File "/usr/lib/python3.4/asyncio/base_events.py", line 866, in subprocess_exec
Feb 22 14:48:57 Carlo-Pi hass[629]: bufsize, **kwargs)
Feb 22 14:48:57 Carlo-Pi hass[629]: File "/usr/lib/python3.4/asyncio/unix_events.py", line 173, in _make_subprocess_trans
port
Feb 22 14:48:57 Carlo-Pi hass[629]: extra=extra, **kwargs)
Feb 22 14:48:57 Carlo-Pi hass[629]: File "/usr/lib/python3.4/asyncio/base_subprocess.py", line 31, in __init__
Feb 22 14:48:57 Carlo-Pi hass[629]: stderr=stderr, bufsize=bufsize, **kwargs)
Feb 22 14:48:57 Carlo-Pi hass[629]: File "/usr/lib/python3.4/asyncio/unix_events.py", line 559, in _start
Feb 22 14:48:57 Carlo-Pi hass[629]: universal_newlines=False, bufsize=bufsize, **kwargs)
Feb 22 14:48:57 Carlo-Pi hass[629]: File "/usr/lib/python3.4/subprocess.py", line 859, in __init__
Feb 22 14:48:57 Carlo-Pi hass[629]: restore_signals, start_new_session)
Feb 22 14:48:57 Carlo-Pi hass[629]: File "/usr/lib/python3.4/subprocess.py", line 1395, in _execute_child
Feb 22 14:48:57 Carlo-Pi hass[629]: restore_signals, start_new_session, preexec_fn)
Feb 22 14:48:57 Carlo-Pi hass[629]: OSError: [Errno 12] Cannot allocate memory
@CCOSTAN
Copy link
Contributor Author
CCOSTAN commented Feb 22, 2017

I think these issues might have been localized. unable to reproduce on 0.39.0.dev

@CCOSTAN CCOSTAN closed this as completed Feb 22, 2017
@CCOSTAN CCOSTAN mentioned this issue Feb 22, 2017
Closed
@CCOSTAN
Copy link
Contributor Author
CCOSTAN commented Feb 22, 2017

image

HASS seems to be taking up 90% of my Pis memory now. :(

@CCOSTAN CCOSTAN reopened this Feb 22, 2017
@pvizeli
Copy link
Member
pvizeli commented Feb 22, 2017

Please add a list of components and platform. On my raspbeery I have use only 20%

@grivers3
Copy link

Same Issue. After shutting components off one by one, I found the emulated hue is my culprit. Also, there is no history when my emulated hue is active.
emulated_hue:
type: google_home
listen_port: 80
off_maps_to_on_domains:
- script
- scene
expose_by_default: false

/w Emulated Hue (Taken at 1 minute intervals)
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
24822 homeass+ 20 0 424524 95692 11476 S 99.1 12.7 1:31.66 hass
24822 homeass+ 20 0 438716 109888 11476 S 101.7 14.6 2:30.35 hass
24822 homeass+ 20 0 450224 121780 11476 S 99.7 16.2 3:29.69 hass
24822 homeass+ 20 0 460648 130824 11476 S 99.4 17.4 4:30.18 hass

w/o Emulated Hue (Taken at 1 minute intervals)
24506 homeass+ 20 0 282328 59596 11568 S 7.0 7.9 0:35.59 hass
24506 homeass+ 20 0 282328 59720 11568 S 2.3 7.9 0:41.68 hass
24506 homeass+ 20 0 389708 60708 11568 S 27.7 8.1 0:45.75 hass
24506 homeass+ 20 0 283212 60800 11568 S 1.0 8.1 0:48.51 hass

@grivers3
Copy link

History issue might be browser cache. Take with a grain of salt..

@CCOSTAN
Copy link
Contributor Author
CCOSTAN commented Feb 22, 2017

I rolled back to 0.38.3 and the issue went away. been running fine for a few hours now.

image

The issue for me seems to have been 0.38.4

@Cinntax
Copy link
Contributor
Cinntax commented Feb 23, 2017

I have also seen this issue, and tracked it down to the emulated_hue component. The ram usage just seems to grow and grow. I'm running on a normal server, so it took a while, but eventually ate up the RAM on my server + swap.

I'm not quite sure where to go to provide more detail- I turned logging up to debug and didn't see anything terribly useful in the logs while this was all happening.

@pvizeli
Copy link
Member
pvizeli commented Feb 23, 2017

Do you have aiohttp 1.3.3 running? I think that is the Memory issue they fix with that. Mybe you need to remove the dept folder with inlcude a older version?

@grivers3
Copy link

Deleted all aiohttp entries in deps and restarted. aiohttp-1.3.3.dist-info added to deps. Still having issue with the emulated hue.

@ctaloi
Copy link
ctaloi commented Feb 23, 2017

Confirming memory issue here since upgrading to 38.4

@pvizeli
Copy link
Member
pvizeli commented Feb 23, 2017

Okay from 0.38.3 to 0.38.4 have no change on aiohttp. Can you disable discovery? That have change the library

@ctaloi
Copy link
ctaloi commented Feb 23, 2017 via email

@Cinntax
Copy link
Contributor
Cinntax commented Feb 23, 2017

I agree- disabling discovery seemed to clear up the problem.

Also note that I tried to re-enable discovery, while at the time disabling my normal hue light component- thinking perhaps the issue was being caused by the hue component discovering the emulated_hue component. That did not make any difference - memory/CPU was still high.

@iambdud
Copy link
iambdud commented Feb 24, 2017

I'm seeing this issue in 0.38.3... and I already had the discovery component disabled... now that i think about it... i think the machine started eating memory when i told alexa to activate a scene via emulated_hue.

@deftdawg
Copy link
Contributor

Also seeing this in 0.38.4

This starts to appear in my logs after a while (like 2 hours)
hass[983]: OSError: [Errno 12] Cannot allocate memory

hass is using more than 12GB (of 16GB) of ram

@iambdud
Copy link
iambdud commented Feb 24, 2017 •
edited
Loading

So... I added back the "type" for my emulated_hue component and everything seems back to normal...

edit: I spoke too soon... it ran out of memory again.

@jer78
Copy link
jer78 commented Feb 24, 2017

Also have the memory issue since 0.38.4. Reverted back to 0.38.3 and everything is fine again.

@deftdawg
Copy link
Contributor

I set this up to recycle 0.38.4

#!/usr/bin/env bash

# Usage sudo ./ha-watchdog.sh
while [ true ]; do
  sudo service home-assistant@${SUDO_USER} restart
  echo "Watchdog restarted Home Assistant at $(date)"
  journalctl -f -u home-assistant@${SUDO_USER} | grep "Cannot allocate memory" | head -1
done

Watchdog restarted Home Assistant at Fri Feb 24 04:58:02 EST 2017
Feb 24 06:26:01 Chromebox hass[19432]: OSError: [Errno 12] Cannot allocate memory
Watchdog restarted Home Assistant at Fri Feb 24 08:09:18 EST 2017
Feb 24 09:35:37 Chromebox hass[2903]: OSError: [Errno 12] Cannot allocate memory
Watchdog restarted Home Assistant at Fri Feb 24 10:11:26 EST 2017
Feb 24 11:38:19 Chromebox hass[1462]: OSError: [Errno 12] Cannot allocate memory
Watchdog restarted Home Assistant at Fri Feb 24 12:13:53 EST 2017

Unfortunately, it doesn't seem to work 100%, I just had to manually cycle it a little while ago because the same message was appearing again.

I briefly looked at emulated_hue component tagged in 0.38.1 to 0.38.4, doesn't seem like anything changed between point releases (last update was 17 days ago), wonder what is causing this.

@acarlo79
Copy link
acarlo79 commented Feb 25, 2017 •
edited by balloob
Loading

Same issue here after the upgrade to 0.38.4, the only other action I performed was to chnage the emulated hue component from alexa to google_home.
Not sure yet if this is the culprit here:

17-02-25 00:00:10 ERROR (Thread-6) [homeassistant.core] Error doing job: Future exception was never retrieved
Traceback (most recent call last):
  File "/usr/lib/python3.4/concurrent/futures/thread.py", line 54, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/srv/hass/hass_venv/lib/python3.4/site-packages/homeassistant/components/updater.py", line 83, in <lambda>
    hass, lambda _: check_newest_version(hass, huuid),
  File "/srv/hass/hass_venv/lib/python3.4/site-packages/homeassistant/components/updater.py", line 91, in check_newest_version
    result = get_newest_version(huuid)
  File "/srv/hass/hass_venv/lib/python3.4/site-packages/homeassistant/components/updater.py", line 133, in get_newest_version
    import distro
  File "/home/hass/.homeassistant/deps/distro.py", line 1051, in <module>
    _distro = LinuxDistribution()
  File "/home/hass/.homeassistant/deps/distro.py", line 595, in __init__
    if include_lsb else {}
  File "/home/hass/.homeassistant/deps/distro.py", line 921, in _get_lsb_release_info
    stderr=subprocess.PIPE)
  File "/usr/lib/python3.4/subprocess.py", line 859, in __init__
    restore_signals, start_new_session)
  File "/usr/lib/python3.4/subprocess.py", line 1395, in _execute_child
    restore_signals, start_new_session, preexec_fn)
OSError: [Errno 12] Cannot allocate memory

@deftdawg
Copy link
Contributor

Seems to also be happening after update to 0.39.0

Feb 26 01:07:07 Chromebox hass[15729]: OSError: [Errno 12] Cannot allocate memory
Watchdog restarted Home Assistant at Sun Feb 26 01:17:14 EST 2017

@pvizeli
Copy link
Member
pvizeli commented Feb 26, 2017

To disable the updater does also not help? Only the discover (netdisco)?

@patrickeasters
Copy link
Contributor

I am also still seeing this in 0.39.0. I'll be rolling back to 0.38.3 again.

@patrickeasters patrickeasters mentioned this issue Feb 26, 2017
Closed
@danodemano danodemano mentioned this issue Feb 28, 2017
Closed
@tomusher tomusher mentioned this issue Feb 28, 2017
Merged
@vladm
Copy link
vladm commented Feb 28, 2017 •
edited
Loading

I'm on 39.1 under docker;
Commented out discovery, emulated_hue still seeing high CPU + 1.4gb ram usage.

@vladm
Copy link
vladm commented Feb 28, 2017 •
edited
Loading

okay, I've tracked down my high-cpu problem to using two hikvision binary_sensors defined as the following:

binary_sensor:
- platform: hikvision
  host: xx.xx.xx.10
  port: 80
  ssl: False
  username: aaa
  password: bbb

- platform: hikvision
  host: xx.xx.xx.20
  port: 80
  ssl: False
  username: aaa
  password: bbb

with both sensors defined in configuration, python process constantly takes 70% CPU. As long as I comment out the second sensor, everything goes back to normal.

@mezz64

@mezz64
Copy link
Contributor
mezz64 commented Feb 28, 2017

Interesting. I also have two cameras setup with hikvision binary with a docker install and don't have any high CPU or memory problems. I don't use emulated_hue though. Can you replicate this with a bare HASS instance and just the hikvision sensors defined?

@vladm
Copy link
vladm commented Feb 28, 2017

@mezz64 - confirmed. Removed everything, leaving only sections "homeassistant" with basic settings, "frontend", "http" and "binary_sensor" defined as above.

Should I create a new issue as it seems a separate problem?

This is 0.39.1 official docker image.

@deftdawg
Copy link
Contributor

Kind of hard for me to see what changed between 38.3 and 38.4+ on github, so today I fired up sourcetree and pull down the repo...

It looks like the only thing that changed was bumping netdisco from 0.8.2 to 0.8.3 and a bunch of UI changes in js/html files...

Looking at netdisco repo, support was added for Philips hue upnp and some other stuff in that release. My guess is whatever that code is doing it is causing crap ton of cpu / memory consumption...

To test this theory, I rolled back netdisco by doing the following:

cd ~/.homeassistant/deps
mv netdisco ../netdisco.0.8.3  # get it out of the deps directory
cp -r ../deps.bkp/netdisco .  # copy in previous version of the dep
# restart hass

So far my 2x CPUs have dropped from both hovering around 80% cpu to around 20% each. And memory seems to be stable at couple hundred MBs. I'll continue to watch it... but if others are experiencing this you may want to try rolling back netdisco too.

@mezz64
Copy link
Contributor
mezz64 commented Feb 28, 2017

@vladm You can create a seperate issue if you wish. I'm fairly certain something else is going here though. Was your "deps" folder empty? I was running on dev, but switched to the latest release container just now to verify from my end an am still not seeing anything significant... 5% cpu and about 150MB of memory being used. Anyone else with memory/cpu issues using hikvision binary sensors?

@tomusher
Copy link

@jumpkick I've submitted a PR to the netdisco repo which at least resolves the issue with the discovery component (home-assistant-libs/netdisco#99)

@deftdawg deftdawg mentioned this issue Feb 28, 2017
Merged
@deftdawg
Copy link
Contributor

@mezz64 -- if you deps/netdisco folder has philips_hue_nupnp.py in it then it's 0.8.3, if not you have 0.8.2 or earlier and you won't see the issue.

@tomusher cool, I just put a patch to roll it back... Doesn't matter to me which way we go as long as this bug gets crushed out of my system 😄

@mezz64
Copy link
Contributor
mezz64 commented Feb 28, 2017

@jumpkick I did/do not personally experience the issue with 0.8.3 netdisco. I was just trying to figure out if there is something going wrong in my hikvision component that was mentioned by @vladm as a possible source of the problem. I can't replicate on my end so it's hard to troubleshoot.

@deftdawg
Copy link
Contributor

@mezz64 @vladm hikvision did not change between 0.38.3 and 0.38.4, so it's not related to this issue, should spin up a different issue for that.

@vladm vladm mentioned this issue Feb 28, 2017
Closed
@kipwittchen
Copy link

Rolling back netdisco to 0.8.2 seems to have fixed my mqtt/memory leak issue! I'll check the log/see if mysensors are still reporting properly in the morning.

balloob pushed a commit that referenced this issue Mar 1, 2017
@deftdawg @balloob
Rollback netdisco to 0.8.2 to resolve #6165 (#6314)
a0256e1 
* Rollback netdisco to 0.8.2 to resolve #6165

* Rollback netdisco to 0.8.2 to resolve #6165
@balloob balloob mentioned this issue Mar 1, 2017
Merged
2 tasks
@balloob balloob reopened this Mar 1, 2017
@balloob
Copy link
Member
balloob commented Mar 1, 2017

I have a proper Netdisco fix here: #6321

Would love to get some people testing this.

@iambdud
Copy link
iambdud commented Mar 1, 2017 •
edited
Loading

Would love to test because this issue is driving me bonkers... but fairly new to this. I'm using docker. How can I help? Should i just pull the dev image in docker?

@jnewland
Copy link
Contributor
jnewland commented Mar 1, 2017

Would love to get some people testing this.

Memory usage is low and stable with #6321 deployed. 👌

@kipwittchen
Copy link

I ran this command to install netdisco 0.9.0:
pip install netdisco==0.9.0 --target ~/.homeassistant/deps

So far, so good!

@balloob balloob mentioned this issue Mar 11, 2017
Merged
@home-assistant home-assistant locked and limited conversation to collaborators Jun 2, 2017
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
bug integration: discovery integration: emulated_hue integration: hue
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Rollback netdisco to 0.8.2 to resolve #6165
Discovery fix home-assistant/core
0