Provide api to see which integrations are being loaded #48274

bdraco · 2021-03-24T04:42:40Z

Needs #48269 since device trackers are not being added to the components

Proposed change

This is much easier to review without whitespace changes due to the context manager addition

Goals

Show the integration that is taking the most time to startup during startup
Show how longer each integration took in total on the config info page to call out slow or misbehaving integrations so Home Assistant doesn't take the blame for slow ones (esp custom ones)

2021-03-26 08:59:01 INFO (MainThread) [homeassistant.bootstrap] Integration setup times: {'logger': 0.001837, 'onboarding': 0.002468, 'search': 0.002993, 'lock': 0.004147, 'webhook': 0.004786, 'frontend': 0.005408, 'alexa': 0.00748, 'stream': 0.009616, 'http': 0.011046, 'image': 0.012741, 'cover': 0.0138, 'system_log': 0.017852, 'device_automation': 0.019021, 'config': 0.044177, 'sensor': 0.045179, 'fan': 0.051967, 'cloud': 0.053339, 'my': 0.054587, 'recorder': 0.055922, 'vacuum': 0.080255, 'api': 0.081303, 'auth': 0.088869, 'light': 0.091841, 'default_config': 0.093249, 'rest_command': 0.100522, 'lovelace': 0.103624, 'dhcp': 0.105445, 'websocket_api': 0.110111, 'script': 0.113537, 'emulated_hue': 0.120521, 'binary_sensor': 0.145037, 'weather': 0.197711, 'media_player': 0.204634, 'switch': 0.241131, 'cast': 0.313861, 'person': 0.341786, 'homekit': 0.344398, 'ssdp': 0.444936, 'homekit_controller': 0.670873, 'doorbird': 0.714473, 'climate': 0.740771, 'history': 0.783979, 'blueprint': 0.806934, 'media_source': 0.869515, 'sun': 0.876625, 'shell_command': 0.880883, 'threading': 0.881798, 'remote': 0.885051, 'map': 0.913215, 'trace': 0.937068, 'autelis': 0.937501, 'filter': 0.958175, 'enphase_envoy': 0.961878, 'somfy': 0.96544, 'automation': 0.967931, 'homeassistant': 1.117395, 'sonos': 1.231818, 'systemmonitor': 1.297367, 'zeroconf': 1.421767, 'logbook': 1.541961, 'system_health': 1.574957, 'flume': 1.658237, 'scene': 1.676498, 'profiler': 1.678715, 'tag': 1.729499, 'upnp': 1.809989, 'heos': 1.91246, 'unifi': 1.951867, 'timer': 1.978993, 'camera': 2.00041, 'ffmpeg': 2.001187, 'denonavr': 2.070011, 'mobile_app': 2.25135, 'template': 2.261124, 'nut': 2.53838, 'apple_tv': 2.602957, 'counter': 2.717594, 'input_datetime': 2.730172, 'tts': 2.778089, 'updater': 2.853858, 'openweathermap': 3.065485, 'samsungtv': 3.203091, 'group': 3.27649, 'harmony': 3.696712, 'rachio': 4.201295, 'ipp': 4.290456, 'roku': 4.433694, 'input_text': 4.929488, 'synology_dsm': 4.958669, 'zone': 4.987181, 'input_number': 4.996274, 'bond': 5.327438, 'input_boolean': 5.853331, 'screenlogic': 5.921358, 'notify': 5.929316, 'lutron_caseta': 6.010032, 'input_select': 6.024482, 'rest': 6.607305, 'sense': 6.637527, 'smartir': 6.648064, 'august': 6.659398, 'faa_delays': 6.986708, 'icloud3': 7.141761, 'device_tracker': 7.340055, 'unifiprotect': 7.423218, 'roomba': 7.626561, 'senseme': 7.818574, 'hacs': 8.534149, 'tesla': 9.586424, 'emonitor': 9.692194, 'isy994': 11.400433, 'ping': 15.259365, 'esphome': 21.798107, 'broadlink': 32.379421}

This has already helped identify some integrations that could be sped up:

ping: Prevent ping integration from delaying startup #43869
senseme: Use synchronization primitives to avoid sleeping during first update bdraco/aiosenseme#10
do not load disabled/ignored integrations: Do not load ignored or disabled integrations at startup #48355
small optimizations of adding entities: Small speed up to adding entities #48353
group creating storage collection entities: Optimize storage collection entity operations with asyncio.gather #48352
broadlink storage loads: Lazy load broadlink storage #48391
bond setup concurrency Speed up bond setup with gather #48454

Type of change

Dependency upgrade
Bugfix (non-breaking change which fixes an issue)
New integration (thank you!)
New feature (which adds functionality to an existing integration)
Breaking change (fix/feature causing existing functionality to break)
Code quality improvements to existing code or addition of tests

Example entry for `configuration.yaml`:

# Example configuration.yaml

Additional information

This PR fixes or closes issue: fixes #
This PR is related to issue:
Link to documentation pull request:

Checklist

The code change is tested and works locally.
Local tests pass. Your PR cannot be merged unless tests pass
There is no commented out code in this PR.
I have followed the development checklist
The code has been formatted using Black (black --fast homeassistant tests)
Tests have been added to verify that the new code works.

If user exposed functionality or configuration variables are added/changed:

Documentation added/updated for www.home-assistant.io

If the code communicates with devices, web services, or third-party tools:

The manifest file has all fields filled out correctly.
Updated and included derived files by running: python3 -m script.hassfest.
New or updated dependencies have been added to requirements_all.txt.
Updated by running python3 -m script.gen_requirements_all.
Untested files have been added to .coveragerc.

The integration reached or maintains the following Integration Quality Scale:

No score or internal
🥈 Silver
🥇 Gold
🏆 Platinum

To help with the load of incoming pull requests:

I have reviewed two other open pull requests in this repository.

bdraco · 2021-03-24T04:47:30Z

I think I should make DATA_SETUP_STARTED a context manager so we can add it more places to increase what can be tracked.

With async_start_setup(hass, names):

Pushes into DATA_SETUP_STARTED, pop on finish

On finish into DATA_SETUP_COMPLETED so we can see on config info page

Places we need it:

Setup integration -- DATA_SETUP_STARTED already populates integration
Entity platform (platform.integration)
device tracker legacy (device_tracker.integration)
notify platforms (notify.integration)

bdraco · 2021-03-24T19:58:44Z

2021-03-24 19:57:35 DEBUG (MainThread) [homeassistant.bootstrap] Setup times: {'logger': datetime.timedelta(microseconds=1423), 'http': datetime.timedelta(microseconds=17148), 'system_log': datetime.timedelta(microseconds=1087), 'recorder': datetime.timedelta(microseconds=83353), 'webhook': datetime.timedelta(microseconds=75709), 'websocket_api': datetime.timedelta(microseconds=74605), 'auth': datetime.timedelta(microseconds=63292), 'device_automation': datetime.timedelta(microseconds=61863), 'api': datetime.timedelta(microseconds=45899), 'alexa': datetime.timedelta(microseconds=3386), 'search': datetime.timedelta(microseconds=15968), 'config': datetime.timedelta(microseconds=38842), 'lovelace': datetime.timedelta(microseconds=86972), 'image': datetime.timedelta(microseconds=7629), 'person': datetime.timedelta(microseconds=313228), 'cloud': datetime.timedelta(microseconds=49667), 'onboarding': datetime.timedelta(microseconds=2139), 'frontend': datetime.timedelta(microseconds=4019), 'blueprint': datetime.timedelta(microseconds=819283), 'shell_command': datetime.timedelta(microseconds=818958), 'trace': datetime.timedelta(microseconds=817333), 'updater': datetime.timedelta(seconds=2, microseconds=709575), 'remote': datetime.timedelta(microseconds=793298), 'rest_command': datetime.timedelta(microseconds=777519), 'my': datetime.timedelta(microseconds=775199), 'threading': datetime.timedelta(microseconds=761732), 'autelis': datetime.timedelta(microseconds=759321), 'climate': datetime.timedelta(microseconds=747819), 'sun': datetime.timedelta(microseconds=743480), 'map': datetime.timedelta(microseconds=711164), 'history': datetime.timedelta(microseconds=654846), 'dhcp': datetime.timedelta(microseconds=44424), 'binary_sensor': datetime.timedelta(microseconds=47389), 'homeassistant': datetime.timedelta(seconds=1, microseconds=239579), 'emulated_hue': datetime.timedelta(microseconds=575971), 'system_health': datetime.timedelta(seconds=1, microseconds=455119), 'logbook': datetime.timedelta(seconds=1, microseconds=395573), 'profiler': datetime.timedelta(seconds=1, microseconds=322999), 'media_source': datetime.timedelta(seconds=1, microseconds=320622), 'scene': datetime.timedelta(seconds=1, microseconds=300893), 'input_datetime': datetime.timedelta(seconds=1, microseconds=585562), 'camera': datetime.timedelta(seconds=1, microseconds=537392), 'timer': datetime.timedelta(seconds=1, microseconds=529638), 'ffmpeg': datetime.timedelta(seconds=1, microseconds=518741), 'counter': datetime.timedelta(seconds=1, microseconds=498190), 'tag': datetime.timedelta(seconds=1, microseconds=109656), 'zeroconf': datetime.timedelta(seconds=1, microseconds=348315), 'switch': datetime.timedelta(microseconds=465537), 'flume': datetime.timedelta(seconds=2, microseconds=206408), 'openweathermap': datetime.timedelta(seconds=4, microseconds=26078), 'apple_tv': datetime.timedelta(seconds=3, microseconds=81848), 'somfy': datetime.timedelta(seconds=1, microseconds=33030), 'group': datetime.timedelta(seconds=2, microseconds=981550), 'ping': datetime.timedelta(seconds=32, microseconds=226043), 'ssdp': datetime.timedelta(microseconds=403898), 'zone': datetime.timedelta(seconds=3, microseconds=633598), 'media_player': datetime.timedelta(microseconds=501019), 'nut': datetime.timedelta(seconds=2, microseconds=71835), 'sonos': datetime.timedelta(seconds=1, microseconds=558537), 'samsungtv': datetime.timedelta(seconds=4, microseconds=222867), 'weather': datetime.timedelta(microseconds=34404), 'input_number': datetime.timedelta(seconds=4, microseconds=81606), 'input_text': datetime.timedelta(seconds=4, microseconds=40730), 'input_boolean': datetime.timedelta(seconds=4, microseconds=9207), 'upnp': datetime.timedelta(seconds=1, microseconds=641862), 'screenlogic': datetime.timedelta(seconds=5, microseconds=448981), 'doorbird': datetime.timedelta(microseconds=664446), 'tts': datetime.timedelta(seconds=2, microseconds=736614), 'roomba': datetime.timedelta(seconds=6, microseconds=832777), 'sensor': datetime.timedelta(microseconds=39270), 'denonavr': datetime.timedelta(seconds=1, microseconds=671809), 'vacuum': datetime.timedelta(microseconds=37323), 'template': datetime.timedelta(seconds=1, microseconds=295748), 'unifi': datetime.timedelta(seconds=1, microseconds=982090), 'homekit': datetime.timedelta(microseconds=376393), 'esphome': datetime.timedelta(seconds=7, microseconds=742603), 'senseme': datetime.timedelta(seconds=8, microseconds=999276), 'smartir': datetime.timedelta(seconds=7, microseconds=200950), 'systemmonitor': datetime.timedelta(microseconds=872095), 'fan': datetime.timedelta(microseconds=33225), 'rachio': datetime.timedelta(seconds=3, microseconds=238924), 'homekit_controller': datetime.timedelta(microseconds=721161), 'heos': datetime.timedelta(seconds=2, microseconds=51075), 'automation': datetime.timedelta(seconds=1, microseconds=23583), 'light': datetime.timedelta(microseconds=341107), 'input_select': datetime.timedelta(seconds=4, microseconds=113446), 'cover': datetime.timedelta(microseconds=51583), 'faa_delays': datetime.timedelta(seconds=4, microseconds=889299), 'filter': datetime.timedelta(microseconds=597656), 'enphase_envoy': datetime.timedelta(microseconds=604108), 'script': datetime.timedelta(microseconds=50113), 'rest': datetime.timedelta(seconds=4, microseconds=648672), 'cast': datetime.timedelta(microseconds=266041), 'roku': datetime.timedelta(seconds=2, microseconds=426973), 'notify': datetime.timedelta(seconds=5, microseconds=46318), 'mobile_app': datetime.timedelta(seconds=1, microseconds=157026), 'stream': datetime.timedelta(microseconds=11007), 'default_config': datetime.timedelta(microseconds=71706), 'bond': datetime.timedelta(seconds=3, microseconds=772802), 'unifiprotect': datetime.timedelta(seconds=5, microseconds=950214), 'broadlink': datetime.timedelta(seconds=6, microseconds=301112), 'ipp': datetime.timedelta(seconds=4, microseconds=118290), 'august': datetime.timedelta(seconds=5, microseconds=115461), 'lutron_caseta': datetime.timedelta(seconds=4, microseconds=469076), 'lock': datetime.timedelta(microseconds=3285), 'harmony': datetime.timedelta(seconds=2, microseconds=752673), 'sense': datetime.timedelta(seconds=5, microseconds=612323), 'synology_dsm': datetime.timedelta(seconds=4, microseconds=269406), 'tesla': datetime.timedelta(seconds=6, microseconds=867953), 'hacs': datetime.timedelta(seconds=6, microseconds=562388), 'icloud3': datetime.timedelta(seconds=6, microseconds=528373), 'isy994': datetime.timedelta(seconds=11, microseconds=701235), 'device_tracker': datetime.timedelta(seconds=13, microseconds=54139)}

bdraco · 2021-03-29T23:44:26Z

rebase to fix conflict only

homeassistant/bootstrap.py

homeassistant/components/websocket_api/commands.py

MartinHjelmare

Looks good!

Would be good with some more eyes and thoughts.

homeassistant/bootstrap.py

homeassistant/components/websocket_api/commands.py

Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>

homeassistant/components/websocket_api/commands.py

bdraco · 2021-04-05T08:09:14Z

Thanks. Retested 👍

Frontend PR adjusted to the new name.

homeassistant added the cla-signed label Mar 24, 2021

probot-home-assistant bot added core integration: websocket_api labels Mar 24, 2021

bdraco changed the title ~~Provide api to see which integrations are being loaded.~~ Provide api to see which integrations are being loaded Mar 24, 2021

bdraco mentioned this pull request Mar 25, 2021

Display cumulative setup seconds on the config info page home-assistant/frontend#8720

Merged

9 tasks

bdraco force-pushed the startup_visibility branch from f5dd39b to 21c7d99 Compare March 27, 2021 14:51

bdraco marked this pull request as ready for review March 29, 2021 02:05 8000

bdraco force-pushed the startup_visibility branch 2 times, most recently from 8ea169e to 815bd1a Compare March 29, 2021 23:44

frenck reviewed Mar 30, 2021

View reviewed changes

homeassistant/bootstrap.py Outdated Show resolved Hide resolved

bdraco commented Mar 30, 2021

View reviewed changes

homeassistant/bootstrap.py Outdated Show resolved Hide resolved

bdraco force-pushed the startup_visibility branch from 5751cc2 to d7ab25e Compare April 2, 2021 23:22

bdraco mentioned this pull request Apr 3, 2021

Startup is still blocked (just later on) hacs/integration#2005

Closed

bdraco added 14 commits April 4, 2021 10:24

dispatch every second

da84b07

debug

e2fc9dd

Make setup time a context manager

c1f47bd

fix

c3fa709

handle multiple ping sensors

599b6c9

reduce

0ef0de3

fix

6473bfc

add to the right place

516d769

add to the right place

92fc5d6

log

326e455

merge

69b0c1f

add api

f96077c

adjust

1ff8f3f

adjust

c576e20

bdraco added 6 commits April 4, 2021 10:25

sort

fd3c671

coverage

1f23ae4

8000
coverage

0396044

fix merge

de4da02

remove extra line from conflict resolution error

40a5476

Update homeassistant/bootstrap.py

12c99cf

bdraco force-pushed the startup_visibility branch from d7ab25e to 12c99cf Compare April 4, 2021 20:25

remove debug

5998872

MartinHjelmare reviewed Apr 4, 2021

View reviewed changes

homeassistant/components/websocket_api/commands.py Outdated Show resolved Hide resolved

bdraco commented Apr 4, 2021

View reviewed changes

homeassistant/components/websocket_api/commands.py Outdated Show resolved Hide resolved

Update homeassistant/components/websocket_api/commands.py

e3c2bbe

MartinHjelmare approved these changes Apr 4, 2021

View reviewed changes

MartinHjelmare requested a review from balloob April 4, 2021 22:15

balloob reviewed Apr 5, 2021

View reviewed changes

homeassistant/bootstrap.py Outdated Show resolved Hide resolved

balloob reviewed Apr 5, 2021

View reviewed changes

homeassistant/bootstrap.py Show resolved Hide resolved

balloob reviewed Apr 5, 2021

View reviewed changes

homeassistant/components/websocket_api/commands.py Outdated Show resolved Hide resolved

balloob reviewed Apr 5, 2021

View reviewed changes

homeassistant/components/websocket_api/commands.py Outdated Show resolved Hide resolved

balloob approved these changes Apr 5, 2021

View reviewed changes

bdraco and others added 4 commits April 4, 2021 21:14

Update homeassistant/components/websocket_api/commands.py

3634b17

Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>

remove handle_unsubscribe_bootstrap_integrations

54139ed

drop remaining

952d697

adjust test to match change in name

13b721e

bdraco commented Apr 5, 2021

View reviewed changes

homeassistant/components/websocket_api/commands.py Outdated Show resolved Hide resolved

bdraco commented Apr 5, 2021

View reviewed changes

homeassistant/components/websocket_api/commands.py Outdated Show resolved Hide resolved

bdraco added 2 commits April 4, 2021 21:23

Update homeassistant/components/websocket_api/commands.py

da811ba

Update homeassistant/components/websocket_api/commands.py

f00aa02

bdraco merged commit 12e3bc8 into home-assistant:dev Apr 5, 2021

bdraco deleted the startup_visibility branch April 5, 2021 08:11

github-actions bot locked and limited conversation to collaborators Apr 6, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Provide api to see which integrations are being loaded #48274

Provide api to see which integrations are being loaded #48274

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Provide api to see which integrations are being loaded #48274

Provide api to see which integrations are being loaded #48274

Uh oh!

Conversation

Uh oh!

Proposed change

Type of change

Example entry for configuration.yaml:

Additional information

Checklist

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Example entry for `configuration.yaml`: