8000 Separate engine builds to build, test archive components · Issue #81855 · flutter/flutter · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Separate engine builds to build, test archive components #81855

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
godofredoc opened this issue May 4, 2021 · 15 comments
Closed

Separate engine builds to build, test archive components #81855

godofredoc opened this issue May 4, 2021 · 15 comments
Assignees
@godofredoc
Labels
engine flutter/engine repository. See also e: labels. P1 High-priority issues at the top of the work list platform-mac Building on or for macOS specifically

Comments

@godofredoc
Copy link
Contributor

We are planning to split the Mac iOS builds into multiple smaller pieces to optimize resource usage and be able to build/test/archive in less than 30 mins.

In order to start splitting the build it would be great if we can get a dependency graph to help us identify how can we split the gn, ninja, test, archive steps of the recipe.
@godofredoc
Copy link
Contributor Author

\cc @zanderso

@godofredoc
Copy link
Contributor Author

@gw280 is this something you can help us with?

@gw280
Copy link
Contributor
gw280 commented May 4, 2021

Definitely. When are you going to need this by?

@godofredoc
Copy link
Contributor Author

I'm planning to start working on this on Thu | Fri this week.

@gw280
Copy link
Contributor
gw280 commented May 4, 2021

OK, I'll get this sorted out today/tomorrow.

@jmagman jmagman mentioned this issue May 4, 2021
Closed
@TahaTesser TahaTesser added passed first triage engine flutter/engine repository. See also e: labels. platform-mac Building on or for macOS specifically labels May 5, 2021
@chinmaygarde chinmaygarde added the P1 High-priority issues at the top of the work list label May 10, 2021
This was referenced May 11, 2021
Closed
Closed
@godofredoc
Copy link
Contributor Author

@gw280 thanks for creating the dependency graph. To follow up on this topic we will need to create three different recipes:

Note: I'll refer to a single combination of build flags as flavor. eg ios_debug, ios_debug_arm

  • manager, will read the build config, trigger subbuilds, wait for completion, collect artifacts, trigger tests and finally create the bundle.
  • builder, will build a single flavor.
  • tester, will use the artifacts generated by a builder to run tests.

So far this is the configuration used by the builder recipe:

[
   {
         "gn" : ["--ios", "--runtime-mode", "debug", "--simulator", "--no-lto"],
          "ninja": {"config": "ios_debug_sim", "targets": ["ios_test_flutter"]}
   }
]

The next step is to create the artifacts packager, to optimize the execution time we'll need to package only what is required.

@gw280 can you please help us identify which directories from out are needed for each flavor?

@zanderso
Copy link
Member

The next step is to create the artifacts packager, to optimize the execution time we'll need to package only what is required.

@gw280 can you please help us identify which directories from out are needed for each flavor?

I suspect it's easier to identify the directories that aren't needed, like obj/ and exe.unstripped/. I'm not sure there would be other unneeded ones at the top level beyond those two, but if we need to cut down on space, we can pick out the exact things under gen/ that are needed.

@godofredoc
Copy link
Contributor Author

Removing those two directories are probably a good enough optimization, I'll start with that.

@gw280
Copy link
Contributor
gw280 commented May 24, 2021

So for packaging itself, I think we all need is the .framework files in out/`:

https://github.com/flutter/engine/blob/master/sky/tools/create_ios_framework.py#L32

In particular, this is the actual packaging script and it looks for Flutter.framework only.

For running tests, ultimately we just need the *_unittests, *_tests and *_benchmarks executables and their library and asset dependencies. This is going to be harder to enumerate because the assets and dependencies will differ with each binary and could change when people update the tests etc. In this case I'd say go with @zanderso's suggestion of just removing the directories we are pretty sure aren't needed. obj should be fine, gen is needed because some of the assets like kernel_blob.bin for shell_unittests (for example) are needed from there.

@godofredoc
Copy link
Contributor Author

I wonder if we can use gn to list the runtime deps and package them in the most efficient way possible, I've seen that approach used by different teams.

@godofredoc godofredoc mentioned this issue Jun 24, 2021
Merged
9 tasks
@godofredoc godofredoc assigned godofredoc and unassigned gw280 Aug 4, 2021
@godofredoc godofredoc mentioned this issue Aug 4, 2021
Merged
8 tasks
godofredoc added a commit to flutter/engine that referenced this issue Aug 4, 2021
@godofredoc
Add configuration for windows android aot. (#27877)
f7f57a7 
This is a configuration required to separate builds from tests in the
engine repo.

Bug: flutter/flutter#81855
@godofredoc godofredoc mentioned this issue Aug 9, 2021
Merged
8 tasks
godofredoc added a commit to flutter/engine that referenced this issue Aug 10, 2021
@godofredoc
Engine build configuration for web engine. (#27971)
5541f1d 
This is part of the implementation of build/test separation.

Bug: flutter/flutter#81855
@godofredoc godofredoc mentioned this issue Aug 12, 2021
Merged
8 tasks
godofredoc added a commit to flutter/engine that referenced this issue Aug 12, 2021
@godofredoc
Update builder configurations to apply the new generators format. (#2…
8bf3cc6 
…8038)

Web engine required more complex generator configurations which forced
us change the generator values from list to a dict.

Bug: flutter/flutter#81855
@godofredoc godofredoc mentioned this issue Aug 20, 2021
Closed
8 tasks
naudzghebre pushed a commit to naudzghebre/engine that referenced this issue Sep 2, 2021
@godofredoc
Add configuration for windows android aot. (flutter#27877)
4ad99d6 
This is a configuration required to separate builds from tests in the
engine repo.

Bug: flutter/flutter#81855
@godofredoc godofredoc mentioned this issue Sep 29, 2021
Merged
8 tasks
This was referenced Sep 9, 2022
Merged
Merged
Merged
Merged
Merged
This was referenced Sep 17, 2022
Merged
Merged
Merged
This was referenced Sep 28, 2022
Merged
Merged
@godofredoc
Copy link
Contributor Author

https://flutter-review.googlesource.com/c/infra/+/34500 adding prod drone builders.

This was referenced Sep 30, 2022
Merged
Merged
Merged
@godofredoc godofredoc mentioned this issue Oct 26, 2022
Merged
8 tasks
@godofredoc godofredoc mentioned this issue Nov 3, 2022
Merged
8 tasks
auto-submit bot pushed a commit to flutter/engine that referenced this issue Nov 3, 2022
8000
@godofredoc
Move the two remaining engine v2 builds to prod. (#37274)
ed31c3d 
* Move the two remaining engine v2 builds to prod.

This is also removing the environment variable as it is not needed
anymore.

Bug: flutter/flutter#81855

* Remove remaining env variables.
@godofredoc godofredoc mentioned this issue Nov 14, 2022
Merged
8 tasks
auto-submit bot pushed a commit to flutter/engine that referenced this issue Nov 14, 2022
@godofredoc @zanderso
Archive windows gen_snapshot.exe. (#35414)
965f87d 
* Archive windows gen_snapshot.exe.

This is required to enable engine_v2 recipes for windows platform.

Bug: flutter/flutter#81855

* Add dep to generate gen_snapshot.exe.

* Use target platform name rather than host.

* Use prebuilt_arch.

* Use platform_name instead of is_win.

* Remove android check.

* Fix output name.

* Separate platform and cpu in name.

* Remove failing line.

* Use dart_target_arch rather than target_cpu in windows.

* Add dart_target_arch to scope.

* Add support for android_cpu.

* Fix android_cpu variable name.

* Rebase to ToT

* Try to use target cpu directly.

* Use host os instead of is_win.

* Build engine artifacts for win.

* Update build/archives/BUILD.gn

Co-authored-by: Zachary Anderson <zanderso@users.noreply.github.com>

* Update build/archives/BUILD.gn

Co-authored-by: Zachary Anderson <zanderso@users.noreply.github.com>

* Update build/archives/BUILD.gn

Co-authored-by: Zachary Anderson <zanderso@users.noreply.github.com>

Co-authored-by: Zachary Anderson <zanderso@users.noreply.github.com>
auto-submit bot pushed a commit to flutter/engine that referenced this issue Nov 15, 2022
@godofredoc
Add archives to android aot engine. (#37601)
6dc4a6a 
This enables archives generation of gen_snapshot for windows AOT.

Bug: flutter/flutter#81855
schwa423 pushed a commit to schwa423/engine that referenced this issue Nov 16, 2022
@godofredoc @schwa423
Move the two remaining engine v2 builds to prod. (flutter#37274)
6a57828 
* Move the two remaining engine v2 builds to prod.

This is also removing the environment variable as it is not needed
anymore.

Bug: flutter/flutter#81855

* Remove remaining env variables.
schwa423 pushed a commit to schwa423/engine that referenced this issue Nov 16, 2022
@godofredoc @zanderso @schwa423
Archive windows gen_snapshot.exe. (flutter#35414)
fb8dd16 
* Archive windows gen_snapshot.exe.

This is required to enable engine_v2 recipes for windows platform.

Bug: flutter/flutter#81855

* Add dep to generate gen_snapshot.exe.

* Use target platform name rather than host.

* Use prebuilt_arch.

* Use platform_name instead of is_win.

* Remove android check.

* Fix output name.

* Separate platform and cpu in name.

* Remove failing line.

* Use dart_target_arch rather than target_cpu in windows.

* Add dart_target_arch to scope.

* Add support for android_cpu.

* Fix android_cpu variable name.

* Rebase to ToT

* Try to use target cpu directly.

* Use host os instead of is_win.

* Build engine artifacts for win.

* Update build/archives/BUILD.gn

Co-authored-by: Zachary Anderson <zanderso@users.noreply.github.com>

* Update build/archives/BUILD.gn

Co-authored-by: Zachary Anderson <zanderso@users.noreply.github.com>

* Update build/archives/BUILD.gn

Co-authored-by: Zachary Anderson <zanderso@users.noreply.github.com>

Co-authored-by: Zachary Anderson <zanderso@users.noreply.github.com>
schwa423 pushed a commit to schwa423/engine that referenced this issue Nov 16, 2022
@godofredoc @schwa423
Add archives to android aot engine. (flutter#37601)
86fa120 
This enables archives generation of gen_snapshot for windows AOT.

Bug: flutter/flutter#81855
@godofredoc godofredoc mentioned this issue Nov 16, 2022
Merged
8 tasks
auto-submit bot pushed a commit to flutter/engine that referenced this issue Nov 16, 2022
@godofredoc
Move windows engine_v2 to prod. (#37680)
4f3f456 
Windows engine_v2 are now generating the expected artifacts and ready to
be moved to the production environment.

Bug: flutter/flutter#81855
@whesse whesse mentioned this issue Dec 12, 2022
Closed
@whesse
Copy link
Contributor
whesse commented Dec 12, 2022

An initial implementation of a tester recipe has been created, that runs Flutter tests on the engine builds uploaded by the engine_v2 recipes. Further work will be done on deciding whether this recipe can be combined with the existing flutter/flutter_drone recipe that is currently used to run tests, using an alternate way of downloading artifacts.

Follow this on the tracking issue: #116906

@github-actions
Copy link
github-actions bot commented Jun 1, 2023

This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Jun 1, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@godofredoc godofredoc

Labels
engine flutter/engine repository. See also e: labels. P1 High-priority issues at the top of the work list platform-mac Building on or for macOS specifically
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@chinmaygarde @gw280 @whesse @zanderso @TahaTesser @godofredoc
0