8000 OBS build support by dcermak · Pull Request #2067 · packit/packit · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

OBS build support #2067

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Feb 1, 2025
Merged

OBS build support #2067

merged 6 commits into from
Feb 1, 2025

Conversation

dcermak
Copy link
Contributor
@dcermak dcermak commented Sep 6, 2023

This is a very much WIP support for building rpms in the open build service. I have so far tested it only on packit via packit build in-obs --targets fedora-rawhide-x86_64,fedora-rawhide-aarch64,opensuse-tumbleweed-x86_64,opensuse-tumbleweed-aarch64 which created this package: https://build.opensuse.org/package/show/home:dancermak:packit/packit (it is unresolvable because my obs wrapper is not packaged).

TODO:

  • Write new tests or update the old ones to cover new functionality.
  • Update doc-strings where appropriate.
  • Update or write new documentation in packit/packit.dev.
  • Implement waiting for package builds
  • Find the right obs abstraction library (currently it's using my wrapper py-obs but that one is not packaged anywhere, however osc is rather inconvenient to use as a lib). cc @dmach
  • probably much more

RELEASE NOTES BEGIN

Packit now supports building packages in the open build service

RELEASE NOTES END

@softwarefactory-project-zuul
Copy link
Contributor

Build failed.
https://softwarefactory-project.io/zuul/t/packit-service/buildset/584251747064409e8eb1e7dac74f4ebe

pre-commit FAILURE in 1m 31s
packit-tests-rpm TIMED_OUT in 30m 27s
packit-tests-pip-deps TIMED_OUT in 30m 29s
packit-tests-git-main TIMED_OUT in 30m 30s
packit-tests-pip-deps-sess-rec TIMED_OUT in 30m 26s
packit-tests-git-main-sess-rec TIMED_OUT in 30m 27s
reverse-dep-packit-service-tests TIMED_OUT in 30m 29s

@softwarefactory-project-zuul
Copy link
Contributor

Build failed.
https://softwarefactory-project.io/zuul/t/packit-service/buildset/2e100090b44c4f849b008d044fbf3d81

pre-commit FAILURE in 1m 42s
packit-tests-rpm TIMED_OUT in 30m 24s
packit-tests-pip-deps TIMED_OUT in 30m 20s
packit-tests-git-main TIMED_OUT in 30m 23s
packit-tests-pip-deps-sess-rec TIMED_OUT in 30m 25s
packit-tests-git-main-sess-rec TIMED_OUT in 30m 23s
reverse-dep-packit-service-tests TIMED_OUT in 30m 24s

Copy link
Contributor
@LecrisUT LecrisUT left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just some quick notes. I will look into it in more detail on the weekend

@LecrisUT
Copy link
Contributor
LecrisUT commented Sep 6, 2023

To hook to the packit-service, I think this part needs to be abstracted.

@LecrisUT
Copy link
Contributor
LecrisUT commented Sep 6, 2023

Oh yeah, you will not be able to build in copr because there is no internet access and all of the tests go through testing-farm 😟. That is a conundrum. Just to run the tests, maybe you can add that package here with a temporary commit.

@softwarefactory-project-zuul
Copy link
Contributor

Build failed.
https://softwarefactory-project.io/zuul/t/packit-service/buildset/f354f75dc4704dc9a409feeeb58e2e7b

✔️ pre-commit SUCCESS in 1m 31s
packit-tests-rpm TIMED_OUT in 30m 19s
packit-tests-pip-deps TIMED_OUT in 30m 27s
packit-tests-git-main TIMED_OUT in 30m 22s
packit-tests-pip-deps-sess-rec TIMED_OUT in 30m 22s
packit-tests-git-main-sess-rec TIMED_OUT in 30m 22s
reverse-dep-packit-service-tests TIMED_OUT in 30m 27s

@dcermak
Copy link
Contributor Author
dcermak commented Sep 8, 2023

Please don't spend too much time on reviewing this yet, I will migrate away from using py-obs to osc, but that will change the looks of the code drastically.

@softwarefactory-project-zuul
Copy link
Contributor

Build failed.
https://softwarefactory-project.io/zuul/t/packit-service/buildset/315fd68c95f74a65b3572021f2e32aa7

pre-commit FAILURE in 1m 28s
packit-tests-rpm FAILURE in 2m 19s
packit-tests-pip-deps FAILURE in 2m 07s
packit-tests-git-main FAILURE in 2m 28s
✔️ packit-tests-pip-deps-sess-rec SUCCESS in 2m 36s
✔️ packit-tests-git-main-sess-rec SUCCESS in 2m 51s
✔️ reverse-dep-packit-service-tests SUCCESS in 3m 41s


logger = logging.getLogger(__name__)

conf.get_config()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be called in the click cli function, instead of globally. Can make a simple wrapper if you want to expose it outside the CLI.

@dcermak dcermak force-pushed the obs-build-support branch 2 times, most recently from 8775e44 to f0fe546 Compare October 13, 2023 10:00
@dcermak dcermak marked this pull request as ready for review October 13, 2023 10:01
@softwarefactory-project-zuul
Copy link
Contributor

Build failed.
https://softwarefactory-project.io/zuul/t/packit-service/buildset/c3ab5ef6021e45678c82b76e85730de7

✔️ pre-commit SUCCESS in 1m 47s
packit-tests-rpm FAILURE in 2m 27s
packit-tests-pip-deps FAILURE in 2m 26s
packit-tests-git-main FAILURE in 2m 46s
✔️ packit-tests-pip-deps-sess-rec SUCCESS in 2m 58s
✔️ packit-tests-git-main-sess-rec SUCCESS in 3m 16s
✔️ reverse-dep-packit-service-tests SUCCESS in 3m 46s

@softwarefactory-project-zuul
Copy link
Contributor

Build failed.
https://softwarefactory-project.io/zuul/t/packit-service/buildset/2f6ef5eacb68409c99d05f47b95609ab

pre-commit FAILURE in 1m 42s
✔️ packit-tests-rpm SUCCESS in 17m 58s
✔️ packit-tests-pip-deps SUCCESS in 18m 23s
✔️ packit-tests-git-main SUCCESS in 18m 27s
✔️ packit-tests-pip-deps-sess-rec SUCCESS in 2m 51s
✔️ packit-tests-git-main-sess-rec SUCCESS in 3m 02s
✔️ reverse-dep-packit-service-tests SUCCESS in 3m 39s

Copy link
Member
@lachmanfrantisek lachmanfrantisek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot, Dan!

I expected this to be much more XML code..;)

Comment on lines 272 to 273
# wait for the build result
core.get_results(_API_URL, prj_name, pkg_name, printJoin="", wait=True)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we make the waiting optional (e.g. via option) so it's easier to re-use from Packit service where the reporting would be done differently?

Also, do you think we can move more code to obs_build.py for the same reason?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we make the waiting optional (e.g. via option) so it's easier to re-use from Packit service where the reporting would be done differently?

I have made waiting optional.

Also, do you think we can move more code to obs_build.py for the same reason?

What do you mean by that exactly?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have made waiting optional.

Nice! Thanks!

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, do you think we can move more code to obs_build.py for the same reason?

From Packit Service, we use just the Packit API => if we put some logic just into the CLI handling, it can't be reused from the service.

@lachmanfrantisek
Copy link
Member

@majamassarini I think this is worth taking a look at when working on #2099

Here, there is yet another secondary path we are working with (same as we clone dist-git with Fedora-based jobs.)
We should maybe revisit how we handle the jobs with multiple paths involved.

@majamassarini
Copy link
Member

Here, there is yet another secondary path we are working with (same as we clone dist-git with Fedora-based jobs.) We should maybe revisit how we handle the jobs with multiple paths involved.

I am not sure I understand what you mean for multiple paths.
But I can tell you that I see it in this way.
In the OBS use case, I think that packit needs a PathEntry to initialize the api.dg object (and probably the name is wrong here?), instead in the #2099 use case I have to find a way to iniziale the api.up object.
Our working directories are usually wrapped around a LocalProject instance, both dg and up have one LocalProject reference. So we should teach our LocalProject instance how it can obtain a project from the OBS service and in a similar way, for #2099, we should teach the LocalProject how to get the package sources from the Specfile Source directive.

The problem is that LocalProject thinks that the working dir is a Git repo, and in both cases we should abstract somehow the differences. In the OBS code we should go through the osc.core object, if I get it correctly, and in the #2099 use case we should probably do much more hacks.

This should allow us to use the packit jobs, almost as they are, with this new resources.
I don't know if this is understandable and you agree with me?

@lachmanfrantisek
Copy link
Member

Thanks for taking a look @majamassarini ! You are right that it's getting really complicated since it touches the basic design decisions from the early days when we thought we should always work with two git repositories -- one representing upstream (or source-git) and one downstream.

I agree we can avoid a huge architecture redesign for now and fix those two use cases separately. (As Dan is doing.)

@dcermak
Copy link
Contributor Author
dcermak commented Oct 24, 2023

@majamassarini @lachmanfrantisek I am not sure if I understood #2099, but afaik there is no project out there that is actually using OBS as its upstream version control system (except for a few obs specific services). Literally everyone else under the sun is nowadays using git or svn or hg, but not obs. Hence I do not think that this is a path worth exploring, as the 3 projects actually being upstream on OBS are not worth the investment.

Additionally, OBS' version control system is very peculiar. There are no branches, subdirectories are not supported and then there is the highly confusing concept of _link files, which are a "floating merge" (your current repo is merged with the HEAD of another repo, i.e. the current HEAD of your repo is not constant and can change in time 🤪). I have worked with it longer than I would like to admit and probably longer than is useful for my sanity. I'd therefore suggest to really touch the version control system as little as possible.

@dcermak dcermak force-pushed the obs-build-support branch 2 times, most recently from 3b462d7 to c4ea913 Compare October 24, 2023 08:58
@softwarefactory-project-zuul
Copy link
Contributor

Build succeeded.
https://softwarefactory-project.io/zuul/t/packit-service/buildset/d90083c3ceab49b789fec82064c350d8

✔️ pre-commit SUCCESS in 1m 53s
✔️ packit-tests-rpm SUCCESS in 19m 08s
✔️ packit-tests-pip-deps SUCCESS in 19m 18s
✔️ packit-tests-git-main SUCCESS in 19m 18s
✔️ packit-tests-pip-deps-sess-rec SUCCESS in 2m 49s
✔️ packit-tests-git-main-sess-rec SUCCESS in 3m 07s
✔️ reverse-dep-packit-service-tests SUCCESS in 3m 45s

@majamassarini
Copy link
Member

@majamassarini @lachmanfrantisek I am not sure if I understood #2099, but afaik there is no project out there that is actually using OBS as its upstream version control system (except for a few obs specific services). Literally everyone else under the sun is nowadays using git or svn or hg, but not obs. Hence I do not think that this is a path worth exploring, as the 3 projects actually being upstream on OBS are not worth the investment.

The pointer to #2099 is probably misleading. Am I wrong or your downstream metadata are taken from the OBS service?
If yes than what I wanted to say is that right now, the Packit code, when works with both upstream or downstream sources, wraps them around a class that is called LocalProject but this class for the moment is able to play just with plain git repositories. And to make this changes and the #2099 changes be more close to the Packit code style we should probably change the LocalProject behaviour. But this is a big work and I also think that for now your code looks good, we can always refactor it later :)

@dcermak
Copy link
Contributor Author
dcermak commented Oct 25, 2023

@majamassarini @lachmanfrantisek I am not sure if I understood #2099, but afaik there is no project out there that is actually using OBS as its upstream version control system (except for a few obs specific services). Literally everyone else under the sun is nowadays using git or svn or hg, but not obs. Hence I do not think that this is a path worth exploring, as the 3 projects actually being upstream on OBS are not worth the investment.

The pointer to #2099 is probably misleading. Am I wrong or your downstream metadata are taken from the OBS service?

Currently I do not take any metadata from OBS. This is just adding support for building in OBS, nothing more.

If yes than what I wanted to say is that right now, the Packit code, when works with both upstream or downstream sources, wraps them around a class that is called LocalProject but this class for the moment is able to play just with plain git repositories. And to make this changes and the #2099 changes be more close to the Packit code style we should probably change the LocalProject behaviour. But this is a big work and I also think that for now your code looks good, we can always refactor it later :)

Sounds good to me 👍

@lachmanfrantisek
Copy link
Member

@lachmanfrantisek I am not sure if I understood #2099,

Sorry for the confusion. Packit has historically worked with two git repositories -- one representing upstream and one downstream. And we rely on this concept across the codebase. From the functional perspective, the #2099 is completely different to what you are working on. My point was that both issues destroy the concept of upstream-git + downstream-git. I just wanted to mention this so we know this is yet another use case that does not fit the original design.

Copy link
Contributor

Merge Failed.

This change or one of its cross-repo dependencies was unable to be automatically merged with the current state of its repository. Please rebase the change and upload a new patchset.
Warning:
Error merging github.com/packit/packit for 2067,90874899d0724bab7625234c017551aba6c7034d

Copy link
Contributor

Build failed.
https://softwarefactory-project.io/zuul/t/packit-service/buildset/652d83e1a83b42d381095814fcb5fbfc

✔️ pre-commit SUCCESS in 1m 59s
✔️ packit-tests-rpm SUCCESS in 21m 29s
✔️ packit-tests-pip-deps SUCCESS in 21m 25s
✔️ packit-tests-git-main SUCCESS in 21m 38s
packit-tests-pip-deps-sess-rec FAILURE in 2m 24s
packit-tests-git-main-sess-rec FAILURE in 2m 40s
✔️ reverse-dep-packit-service-tests SUCCESS in 3m 33s

Copy link
Member
@lbarcziova lbarcziova left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

mostly just suggestions, otherwise LGTM 🥳 thanks a lot!

do you need some help with the failing test?

Copy link
Contributor

Build failed.
https://softwarefactory-project.io/zuul/t/packit-service/buildset/8f9df0825f73448aabc2e49eebf08935

pre-commit FAILURE in 1m 52s
packit-tests-rpm FAILURE in 1m 45s
packit-tests-pip-deps FAILURE in 2m 37s
packit-tests-git-main FAILURE in 2m 06s
packit-tests-pip-deps-sess-rec FAILURE in 1m 50s
packit-tests-git-main-sess-rec FAILURE in 2m 05s
reverse-dep-packit-service-tests FAILURE in 3m 34s

@dcermak
Copy link
Contributor Author
dcermak commented Jan 24, 2025

do you need some help with the failing test?

Yes, that would be great. I am pretty certain it's due to opensuse-distro-aliases making a network call that needs to be mocked, but I don't know how

Copy link
Contributor

Build failed.
https://softwarefactory-project.io/zuul/t/packit-service/buildset/6557232028214e19b6683a9859436597

✔️ pre-commit SUCCESS in 1m 57s
✔️ packit-tests-rpm SUCCESS in 22m 01s
✔️ packit-tests-pip-deps SUCCESS in 22m 11s
✔️ packit-tests-git-main SUCCESS in 22m 16s
packit-tests-pip-deps-sess-rec FAILURE in 2m 23s
packit-tests-git-main-sess-rec FAILURE in 2m 40s
✔️ reverse-dep-packit-service-tests SUCCESS in 3m 23s

@dcermak dcermak requested a review from LecrisUT January 24, 2025 12:56
@dcermak
Copy link
Contributor Author
dcermak commented Jan 24, 2025

I hope that the CI will pass now 🤞

@dcermak dcermak force-pushed the obs-build-support branch 2 times, most recently from 06d8705 to 3ca02ca Compare January 24, 2025 13:16
Copy link
Contributor

Build succeeded.
https://softwarefactory-project.io/zuul/t/packit-service/buildset/6a8862e887b34751a236c641c0f2c54d

✔️ pre-commit SUCCESS in 1m 56s
✔️ packit-tests-rpm SUCCESS in 22m 15s
✔️ packit-tests-pip-deps SUCCESS in 22m 36s
✔️ packit-tests-git-main SUCCESS in 22m 29s
✔️ packit-tests-pip-deps-sess-rec SUCCESS in 2m 26s
✔️ packit-tests-git-main-sess-rec SUCCESS in 2m 37s
✔️ reverse-dep-packit-service-tests SUCCESS in 3m 29s

dcermak and others added 6 commits February 1, 2025 00:50
Add new dependency: opensuse-distro-aliases
and integrate it with packit.config.aliases.get_aliases
opensuse is different to all the other build targets as e.g. leap contains a
dash in the name and tumbleweed has no version
preliminary OBS building support for packit-cli

Co-authored-by: rxbryan <rxbryn@gmail.com>
Co-authored-by: Laura Barcziová <49026743+lbarcziova@users.noreply.github.com>
Copy link
Contributor

Build succeeded.
https://softwarefactory-project.io/zuul/t/packit-service/buildset/8b0e40b5a3794e17a614890dbdab98c6

✔️ pre-commit SUCCESS in 1m 56s
✔️ packit-tests-rpm SUCCESS in 22m 16s
✔️ packit-tests-pip-deps SUCCESS in 22m 14s
✔️ packit-tests-git-main SUCCESS in 22m 36s
✔️ packit-tests-pip-deps-sess-rec SUCCESS in 2m 26s
✔️ packit-tests-git-main-sess-rec SUCCESS in 2m 47s
✔️ reverse-dep-packit-service-tests SUCCESS in 3m 19s

@lachmanfrantisek
Copy link
Member

I am going to merge and release this -- if you have any more questions, feel free to improve this in the form of a separate issue or pull request.

Huge thanks @dcermak and @rxbryan 🚀

@lachmanfrantisek lachmanfrantisek added the mergeit Merge via Zuul label Feb 1, 2025
Copy link
Contributor

Build succeeded (gate pipeline).
https://softwarefactory-project.io/zuul/t/packit-service/buildset/bac9cd9694a3434ebfa844adbfbd3c27

✔️ pre-commit SUCCESS in 1m 57s

Copy link
Contributor

Pull request merge failed: Resource not accessible by integration, You may need to manually rebase your PR and retry.

@lachmanfrantisek
Copy link
Member

Merging manually since there is a GitHub action change...

@lachmanfrantisek lachmanfrantisek merged commit 9fe3ca7 into packit:main Feb 1, 2025
28 of 29 checks passed
nforro pushed a commit to nforro/packit that referenced this pull request Feb 11, 2025
RELEASE NOTES BEGIN

Packit now supports building packages in the Open Build Service

RELEASE NOTES END
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
mergeit Merge via Zuul
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants
0