8000 Proposal: Use prepare.yml instead of rendering Dockerfiles with the Docker driver · Issue #1642 · ansible/molecule · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Proposal: Use prepare.yml instead of rendering Dockerfiles with the Docker driver #1642

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
MarkusTeufelberger opened this issue Dec 30, 2018 · 24 comments · Fixed by #1649
Closed

Comments

@MarkusTeufelberger
Copy link
Contributor

Issue Type

  • Feature request

Desired Behavior

Similar to Vagrant it might be beneficial to let Ansible itself install Python/Bash/Sudo/??? to various containers in a (pre-rendered) prepare.yml playbook and instead using the images listed in "platforms" unmodified. The current approach (render https://github.com/ansible/molecule/blob/master/molecule/cookiecutter/scenario/driver/docker/%7B%7Bcookiecutter.molecule_directory%7D%7D/%7B%7Bcookiecutter.scenario_name%7D%7D/Dockerfile.j2 and build a custom image from that) seems a bit out of place, after all Vagrant boxes for example are also used unmodified instead of rendering out a Packer config and building the VM from scratch.

@ssbarnea
Copy link
Member
ssbarnea commented Jan 4, 2019

Sounds like a better approach as the bash file could easily grow insane.

PS. I cannot believe someone usedcurcly braches inside foldernames...

@MarkusTeufelberger
Copy link
Contributor Author

Since this is an architectural change, who is in charge now to tell me if this is worth investing the time to write a PR for?

@webknjaz merged a lot of PRs and committed quite a bit recently - are you responsible for architecture too?

@webknjaz
Copy link
Member
webknjaz commented Jan 7, 2019

@MarkusTeufelberger I've volunteered to help with Molecule after its acquisition, it's nowhere near my actual areas of responsibilities (which is Ansible Core Engine).
But in general yes, I kinda influence some of the decisions.

And yes, I like the idea of using Ansible to pre-populate container envs. Since Molecule is tightly coupled with Ansible and targets it unconditionally it's a way to go.
I think it's very similar to what https://github.com/ansible/ansible-container tried to do and what @TomasTomecek is trying to achieve in his project https://github.com/TomasTomecek/ansible-bender.

P.S. @ssbarnea curly braces are coming from cookiecutter project which uses jinja2 to render project templates. I cannot blame them though, it's a quite common tool for sharing project templates nowadays and I can see why they did it.

P.P.S. @MarkusTeufelberger while I give it a green light as of now there's no "official" roadmap on how we think it should look later, just keep that in mind.

cc @gundalow

webknjaz pushed a commit that referenced this issue Jan 7, 2019
PR #1649

Fedora Rawhide dropped the python2-dnf package and now has only
python3-dnf one.

By using a wildcard we avoid the installation failure as it will
install all found versions.

Signed-off-by: Sorin Sbarnea <ssbarnea@redhat.com>
Closes #1642
@MarkusTeufelberger
Copy link
Contributor Author

Please reopen, this is likely a typo (#1649 fixes #1645, not this one)

@MarkusTeufelberger
Copy link
Contributor Author

Please reopen, @webknjaz

@gundalow
Copy link
Contributor

I've added this to ansible/community#427 (comment) we will have out first meeting on Wednesday 23rd at 1900UTC, calendar invite is in the agenda ansible/community#427

@decentral1se decentral1se added this to the v.2.21 milestone Jan 31, 2019
ssbarnea added a commit to ssbarnea/molecule that referenced this issue Jan 31, 2019
PR ansible#1649

Fedora Rawhide dropped the python2-dnf package and now has only
python3-dnf one.

By using a wildcard we avoid the installation failure as it will
install all found versions.

Signed-off-by: Sorin Sbarnea <ssbarnea@redhat.com>
Closes ansible#1642
@decentral1se decentral1se removed this from the v.2.21 milestone Feb 27, 2019
@decentral1se
Copy link
Contributor

@webknjaz
Copy link
Member
webknjaz commented Mar 7, 2019

Let's also ping @TomasTomecek ^

@TomasTomecek
Copy link

Hello! Please let me know if you need any help.

@MarkusTeufelberger
Copy link
Contributor Author

I'd rather use ansible native tools than require buildah on the molecule host to be honest... but ansible-bender looks like a nice project!

@TomasTomecek
Copy link

Not sure what you mean by ansible-native tools. I don't think you need to use anything from bender directly: I'm glad to consult with you if you have any questions. Most of the relevant code is over here, I'm assuming that molecule has something very similar (and more advanced) as well.

(yes, bender support only buildah now, but I'm planning to add support for docker and podman in future)

@MarkusTeufelberger
Copy link
Contributor Author

I mean using a docker container as target host and using Ansible modules (e.g. raw and/or apt) to interact with it using the docker connection plugin, no special Python code or anything beyond Ansible, Docker and the docker Python library necessary.

@MarkusTeufelberger
Copy link
Contributor Author
MarkusTeufelberger commented Mar 7, 2019

To clarify: The desired output of a molecule run is either a deleted container (if it was successful) or a running container (if something went wrong or if only converge was requested). There's no real need other than potential time/bandwidth savings when setting up to build a special docker image. It could be done more like e.g. a gitlab-ci runner that fetches and runs an image from upstream, runs some commands in there and then that's it.

Currently molecule tends to clutter the local docker cache with containers it built itself. This is different than e.g. the Vagrant mode, where it will fetch an upstream "image" (called "box" in Vagrant terms), but uses this image unmodified to deploy an initialization playbook in there (to install python etc.). I'd like molecule to act similarly in Docker mode, so it never actually builds an image by itself. As far as I understand ansible-bender, it is a great tool to run something like docker build with something other than a Dockerfile. Still the desired output would be probably a Docker image, something that I want to avoid being created in the first place.

8000

@TomasTomecek
Copy link

As far as I understand ansible-bender, it is a great tool to run something like docker build with something other than a Dockerfile.

That's correct.

Still the desired output would be probably a Docker image, something that I want to avoid being created in the first place.

Got it, thanks for the explanation.

@decentral1se
Copy link
Contributor

Just to pitch in, whatever way we improve this, we'd have to cover the use case of #1810. I think users are looking for ways to customise what packages are installed without 'ejecting' and adding to the generated Dockerfile.j2.

I'd like molecule to act similarly in Docker mode, so it never actually builds an image by itself.

Ah, that's interesting. Taking a look at it, yes, Molecule always builds an image: if you specify a registry.url in your Driver configuration, it still builds an image. We could add a dont-build: True/False (bikeshed as necessary, default: False) to escape this functionality (not looked deeply, but this is probably not too hard ...). Maybe that's all a bit off-topic on this issue though ...

@MarkusTeufelberger
Copy link
Contributor Author

I would rip out the whole Dockerfile.j2 part and replace it with something close to https://github.com/ansible/molecule/blob/master/molecule/cookiecutter/scenario/driver/vagrant/%7B%7Bcookiecutter.molecule_directory%7D%7D/%7B%7Bcookiecutter.scenario_name%7D%7D/prepare.yml

If someone wants to customize something before the playbook is executed, they can customize this prepare.yml playbook and they don't have to re-build an image. In the case of #1810, a user really should make sure rsync is installed before running the synchronize module within the role he or she is writing. Alternatively all the files in there are just templates and defaults anyways, they can be changed.

@ssbarnea
Copy link
Member

I would personally prefer to avoid using cookiecutter templating and use a non templated version that does the job.

@MarkusTeufelberger
Copy link
Contributor Author

As long as it can be overwritten/extended, I'm totally fine with that. The more it looks like the vagrant driver and "just works", the better.

@ssbarnea ssbarnea added the docker label Feb 3, 2020
@ssbarnea
Copy link
Member

I will close this because current behavior is to user embedded prepare.yml and internal Dockerfile template, which is good for most cases. User can still customize it needed.
I think is a decent trade-off between ability to customize and avoiding a lot of copy/paste between scenarios.

@MarkusTeufelberger
Copy link
Contributor Author

I obviously disagree (hence this feature request), but fair enough if you don't want to go that route.

Maybe it would still be be worth documenting how to actually customize this (don't run docker build when running tests)? I am not sure that this is obvious to users.

@ssbarnea
Copy link
Member

Based on the changes being made to docker modules and ansible, I suspect it would be a nightmare to maintain prepare.yml files scattered on tens if not hundreads of repos. At the same time I kinda find hard to make molecule keep-up being a wrapper for calling these.

I am open to suggestions in this tricky area.

@MarkusTeufelberger
Copy link
Contributor Author

The prepare.yml part would just contain the steps (mostly installing packages) that are currently in the Dockerfile.j2 template and the internal docker build step would be removed from the internal create.yml file. I doubt that the prepare.yml contents would change much/at all in most cases, and the Dockerfile.j2 file also needs to be maintained as well already (I for example need to add support for installing packages on Arch Linux containers on every single Dockerfile template every time I want to use molecule... instead of being able to just use the package Ansible module).

Maybe I should just open a PR to make it a bit clearer, but I wanted to ask first if this is even desired or if docker/podman are expected and designed to be special cases in molecule with integrated templates, build steps on creation time and whatnot compared to e.g. vagrant or lxc/lxd.

@ssbarnea
Copy link
Member

I plan to attempt to combine the docker and podman into a single driver that can use any of the two backends, right afater the 3.0 release. This should allow users to switch from one to another without having to touch their scenarios. Still, I am not sure if I will succeed.

For arch-linux, why you are not maeing few PRs to add it to the implicit set of supported platforms? One first step would be to add it to https://github.com/pycontribs/pys and later to molecule.

@MarkusTeufelberger
Copy link
Contributor Author

Molecule support for Arch docker containers in the Dockerfile was added, then removed, then re-added and then removed again in the past few years already, I found it more sustainable to maintain my own files instead of relying on built-ins.

The repository you're referring to installs too many things in containers that are not related to Ansible and that might actually mask errors in roles (e.g. no task to install git in a role) and they are messing with the default location after install of /usr/bin/python[2/3], so I'm somewhat hesitant to use these images for actual tests. Also they seem to build from latest versions of containers, which is also a constant source of errors.

This change would allow me (and others) to choose an upstream base image in the molecule.yml file and still run Ansible playbooks instead of shell scripts in Dockerfiles against these to provision them. It would also eliminate the pollution of the local docker cache and likely make it faster to run.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants
0