Ansible deployment for production #171
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
provisioning/vars/mangaki.yml
Outdated
|
|
||
| google_ua_id: 'UA-63869890-1' | ||
|
|
||
| hash_paddle: 'lelouch_vi_britannia' |
There was a problem hiding this comment.
Quel est le statut de ces variables ? Est-ce qu'on ne pourrait pas avoir une erreur plutôt que des défaut publics ?
Par ailleurs, quel est le statut de cette ligne w.r.t le default("some_paddle") dans settings.prod.j2 ?
There was a problem hiding this comment.
En fait le hash_paddle n'est utilisé que pour: https://github.com/mangaki/mangaki/blob/master/mangaki/mangaki/views.py#L695
Je ne sais pas si on peut pas plutôt dériver ça de la SECRET_KEY, qu'en penses-tu?
There was a problem hiding this comment.
Mais sinon en effet, t'as raison, je me répète, ça sert à rien de mettre un default, mais je peux mettre une erreur en faisant mandatory et ne mettant pas cette variable ici.
There was a problem hiding this comment.
Vu comment le hash_paddle est utilisé, il s'agit probablement d'un secret partagé avec mandrill ( @jilljenn ?), donc je ne pense pas qu'on puisse changer cela pour le moment. Je suis pour virer cette valeur et le mettre en mandatory en effet (même choise pour secret_key).
There was a problem hiding this comment.
(En fait secret_key et hash_paddle devraient être dans le secret.yml)
There was a problem hiding this comment.
OK, possible. Excellent, je vais le faire.
There was a problem hiding this comment.
Je crois que c'était juste un secret partagé avec nous :D Dans les premières versions de la newsletter.
There was a problem hiding this comment.
Code mort à dégager, donc ?
5f08f58 to
cd9ca95
Compare
06de925 to
e916369
Compare
|
OK, @Elarnon — un coup de review? |
Closed
Codecov Report
@@ Coverage Diff @@
## master #171 +/- ##
==========================================
- Coverage 56.33% 55.86% -0.47%
==========================================
Files 13 13
Lines 703 716 +13
==========================================
+ Hits 396 400 +4
- Misses 307 316 +9
Continue to review full report at Codecov.
|
Elarnon
requested changes
Nov 12, 2016
Vagrantfile
Outdated
|
|
||
| config.vm.network "private_network", ip: "10.99.42.10" | ||
|
|
||
| config.vm.provision :shell, path: "provisioning/bootstrap.sh" | ||
| config.vm.provision "ansible" do |ansible| | ||
| ansible.playbook = "provisioning/site.yml" |
There was a problem hiding this comment.
Please be coherent in the use of spaces (rather than tabs).
| @@ -0,0 +1,12 @@ | |||
| [Unit] | |||
| Description=Re-generate certificates biweekly | |||
There was a problem hiding this comment.
Every other week. Biweekly is ambiguous.
| @@ -0,0 +1,15 @@ | |||
| # Initial configuration for Let's Encrypt. | |||
| # You should not use this configuration as the definitive one. | |||
There was a problem hiding this comment.
We are temporarily using it to get an initial certificate from Let's Encrypt, then it's replaced by the nginx_with_ssl role. The rationale behind this was : it was impossible to obtain a certificate using the "final configuration".
There was a problem hiding this comment.
Could we have instead two templates, one for the HTTP website and one for the HTTPS website. The HTTP one would be enabled first, then letsencrypt.sh would be used to generate certificates, then the HTTPS one would be enabled?
There was a problem hiding this comment.
It would be possible, but we have to agree on something:
Which module will enable the HTTPS configuration? Let's Encrypt or NGINX?
(Knowing that NGINX will copy the templates, enable the HTTP one)
My opinion is that we could let Let's Encrypt enable the HTTPS one, but we have created internal dependencies between these two modules, Let's Encrypt has to know where is the HTTPS configuration and suppose that NGINX is configured / installed, also it needs a handler to reload NGINX. (for HTTPS config; it could become just a simple "global variable").
There was a problem hiding this comment.
I think it's fine to have some dependency between the letsencrypt and nginx roles, since letsencrypt needs a working http server to generate certificates anyways. Note that currently the nginx role somewhat depends on the letsencrypt role since it contains stuff about the well-known directory. What I would do is have the nginx role handle letsencrypt certificate generation and:
- Place the HTTP and HTTPS config in the
sites-availabledirectory. If the variableuse_letsencryptis set, the HTTP config should contain the.well-knownsetup and the HTTPS config should contain paths to the letsencrypt certificates; otherwise the HTTPS config will use the snakeoil certificates. - Enable the HTTP config
- If the
use_letsencryptvariable is set, setup letsencrypt renewal and generate a certificate. - Enable the HTTPS config
What do you think?
| Description=Compute ranking daily | ||
| After=postgresql | ||
| Type=oneshot | ||
| ExecStart={{ command_to_compute_ranking }} |
There was a problem hiding this comment.
Can we get something like {{ manage.py }} ranking here?
| Description=Compute top directors daily | ||
| After=postgresql | ||
| Type=oneshot | ||
| ExecStart={{ command_to_compute_top_directors }} |
There was a problem hiding this comment.
Can we get something like {{ manage.pyp }} top directors here?
|
To test it, I spawned a quick machine on the cloud, used Ansible to test in real network conditions. |
|
I won't do immediately the changes for the Let's Encrypt, but I have it on my todo list. Let's try to get it done before the next two weeks. |
|
Looks impressive to me. |
|
Yay, it works! Except only a minor bug when:
It must be a special case of Ansible trying to group all tasks which Also, let's |
b32181e to
7cdc471
Compare
All remaining is unattended upgrades and email backend configuration. — Make Let's Encrypt renewal and setup works. — Add timers for ranking / top director. — Run Mangaki app server with supervisord. — Install lxml / numpy package to speed the pip install.
… dehydrated's version
cc46ffc to
5bfc977
Compare
jilljenn
pushed a commit
that referenced
this pull request
Feb 14, 2017
* Rough ansible provisioning * ansible(roles): add initial roles * ansible(templates): add settings * ansible(vars): add mangaki * ansible(playbook): add the playbook * ansible(gitignore): ignore the hosts inventory and secrets vars * ansible(readme): explain about secrets * WIP 2 * Ansible: WIPv3 All remaining is unattended upgrades and email backend configuration. — Make Let's Encrypt renewal and setup works. — Add timers for ranking / top director. — Run Mangaki app server with supervisord. — Install lxml / numpy package to speed the pip install. * ansible: remove old mangaki.yml vars * ansible(email): add external smtp server support * Address PR comments * ansible: Refactor LE and NGINX into a same role * ansible: improve apt cache management and remove useless steps * ansible(letsencrypt): Force-restart NGINX after its installation, pin dehydrated's version * ansible(readme): make it more useful * ansible(nginx): remove escape character ' from main.yml
jilljenn
added a commit
that referenced
this pull request
Feb 14, 2017
* This is a commit. * Upper and latter corrected * Written tests for mangaki * Should be okay now. * Fixed stuff: DRY principle applied. URL to work applied. All correct handlers created. * Added 500 base error view in case of DatabaseError. * Looks better that way. * management_commands: add a sketch of generate seed data command for every purpose * seed data generation: use the temp database More verbose output to understand how the process is moving Fix the argument parsing * experiment: generate seed data w/o database cloning * Make tests pass, and actually run them on CircleCI. (#223) Merging with approval from @RaitoBezarius. * Do not require discourse settings for avatar initialization (#229) * Clean up signal handling (#231) * Clean up signal handling - We no longer use a post_save signal for updating scores when a Suggestion is changed; rather this has its place directly in the Suggestion.save method. - Signals are connected in the ready() method of Mangaki's new application configuration class, as recommended by Django's documentation: https://docs.djangoproject.com/en/dev/topics/signals/#connecting-receiver-functions Note that we use a receivers module instead of a signals module to allow for the creation and import of custom signals without registering handlers if the need ever arises. - Profile creation is no longer tied to login with django-allauth but rather to the actual User model creation. This helps handling corner cases such as accounts created through the `manage.py createsuperuser` management command actually having a profile. * Update tests for automatic profile creation * Address PR comments * circle: get back to the project root folder after tests (#233) * Add tests for searching works (#232) * Upgrade Mangaki to Django 1.10 (#234) * Lint the last commit in the CI (#226) * Disable git lint until we can configure it properly * Add some tests ensuring views are not crashing (#237) * Fix typo (unreviewed) * train_test_split moved in sklearn 0.18 (#240) Fixes #239 * Move Mangaki into the root folder (#227) * The `mangaki` folder content has been moved to the root of the repository. * After pulling this commit, many files will be reorganized, backup your work tree before pulling. * Pay attention to your `settings.ini` and path-dependant code, though, they should not be affected by this change. * unreviewed: fix spacing in README * Revert "unreviewed: fix spacing in README" This reverts commit f9848a9. * Revert "Move Mangaki into the root folder (#227)" This reverts commit 74d8749. This breaks mangaki. * Store posters in a FileField (#235) * Rename Work.poster to Work.ext_poster * Be less aggressive with local poster handling - Do not remove the external poster URL when downloading it locally; it is still a piece of possibly valuable information (especially since it is the only way we currently have to link an anime do MAL!) - If there is no locally available poster, fall back to the external URL. * Make admin action for refreshing posters actually work * Ignore media/ directory for development environments * Add FileField for posters * Clean up poster retrieval Various fixes for poster retrieval: - `retrieve_poster` now uses the requests library and is moved to a method on the Work class. It downloads the known external poster by default. - The admin action for updating posters no longer tries to re-download an existing poster onto itself. * Add management command to bulk download posters * Make retrieve_poster update the external poster URL * Update seed data to replace Work.poster with Work.ext_poster * Merge migrations * Address PR comments * Fix path and README (#244) * Add line to admin and change news (#245) Update news * Code coverage is now browsable on the codecov website (#252) * coverage has to run into the proper folder and nosetests to know about where are the tests * circle: fix path to manage.py for the coverage run * Rewrite the AniDB utility (together with a test!) (#248) * Rewrite the AniDB utility (together with a test!) * Remove useless lines * Rename variables * Add style * Remove useless code * Remove unit tests for AniDB * Improve style * Add doctesting * Update and repair the Vagrantfile (#243) * vagrant: update and repair the Vagrantfile * vagrant: transfer the current user's keys into the machine * readme: update instructions * vagrant(bootstrap): install virtualenv as a user rather than globally * vagrant(size): warning about the size taken by the installation * Store posters in a FileField (#235) * Rename Work.poster to Work.ext_poster * Be less aggressive with local poster handling - Do not remove the external poster URL when downloading it locally; it is still a piece of possibly valuable information (especially since it is the only way we currently have to link an anime do MAL!) - If there is no locally available poster, fall back to the external URL. * Make admin action for refreshing posters actually work * Ignore media/ directory for development environments * Add FileField for posters * Clean up poster retrieval Various fixes for poster retrieval: - `retrieve_poster` now uses the requests library and is moved to a method on the Work class. It downloads the known external poster by default. - The admin action for updating posters no longer tries to re-download an existing poster onto itself. * Add management command to bulk download posters * Make retrieve_poster update the external poster URL * Update seed data to replace Work.poster with Work.ext_poster * Merge migrations * Address PR comments * Fix path and README (#244) * Add line to admin and change news (#245) Update news * script(bootstrap): remove the comment which got in the ALLOWED_HOSTS array… * Code coverage is now browsable on the codecov website (#252) * coverage has to run into the proper folder and nosetests to know about where are the tests * circle: fix path to manage.py for the coverage run * Rewrite the AniDB utility (together with a test!) (#248) * Rewrite the AniDB utility (together with a test!) * Remove useless lines * Rename variables * Add style * Remove useless code * Remove unit tests for AniDB * Improve style * Add doctesting * Add new WALS algorithm from TensorFlow (#246) * Add new WALS algorithm from TensorFlow * Add WALS file * Improve style * Minor cleanup around the codebase (#253) * Fix syntax error in `reco_list.html` Signed-off-by: Raito Bezarius <masterancpp@gmail.com> * Mutable arguments are dangerous Default to None, if it's none, replace them by empty arrays. Signed-off-by: Raito Bezarius <masterancpp@gmail.com> * {decode,encode}string are deprecated It's {decode,encode}bytes now. Signed-off-by: Raito Bezarius <masterancpp@gmail.com> * Reference local variable `now` properly before the loop Signed-off-by: Raito Bezarius <masterancpp@gmail.com> * Time to import time for the `retrieveposters` mgt command Signed-off-by: Raito Bezarius <masterancpp@gmail.com> * Reference the `nb_ratings` variable in the good scope. Signed-off-by: Raito Bezarius <masterancpp@gmail.com> * Remove unused imports from `zero.py` Signed-off-by: Raito Bezarius <masterancpp@gmail.com> * Remove unused imports Signed-off-by: Raito Bezarius <masterancpp@gmail.com> * Remove unused imports (import missing models for knn.py also) Signed-off-by: Raito Bezarius <masterancpp@gmail.com> * Import missing modules for NMF (otherwise, I don't see how it was working…) Signed-off-by: Raito Bezarius <masterancpp@gmail.com> * Requirements refactoring into folders Add matplotlib as requirement Signed-off-by: Raito Bezarius <masterancpp@gmail.com> * Remove more unused imports and add used imports * requirements: add production * cleanup: edit the README about requirements, remove old README in mangaki/ * readme: typo * hotfix: s/requirements-dev.txt/requirements/dev.txt (#258) * Ansible deployment for production (#171) * Rough ansible provisioning * ansible(roles): add initial roles * ansible(templates): add settings * ansible(vars): add mangaki * ansible(playbook): add the playbook * ansible(gitignore): ignore the hosts inventory and secrets vars * ansible(readme): explain about secrets * WIP 2 * Ansible: WIPv3 All remaining is unattended upgrades and email backend configuration. — Make Let's Encrypt renewal and setup works. — Add timers for ranking / top director. — Run Mangaki app server with supervisord. — Install lxml / numpy package to speed the pip install. * ansible: remove old mangaki.yml vars * ansible(email): add external smtp server support * Address PR comments * ansible: Refactor LE and NGINX into a same role * ansible: improve apt cache management and remove useless steps * ansible(letsencrypt): Force-restart NGINX after its installation, pin dehydrated's version * ansible(readme): make it more useful * ansible(nginx): remove escape character ' from main.yml * Remove useless notebooks (still ongoing) (#257) * Remove useless notebooks (still ongoing) * Cleanup notebooks * Notebook sur des exemples et tests de svd et dpp (#195) * création du premier notebook * Début du notebook sur le graphique * modifications mineures deu notebook, tests persos * chngments mineurs : mise en forme * continuation du notebook, recopiage du début d'une fonction compute_similarity_cosine * suppression de vieux fichiers * suite du notebook, 1ère fonction pour avoir une matrice de similarité * ajout d'une fonction coisine bis moins calculatoire, début des tests de la DPP * avancement du notebook (dpp, distance, comparaison), un peu (bcp) fouilli, ne pas lire * notebook _notebook stage_ : meilleure implémentation de cosine, début de jaccard, notebook _essai_ : essais, début dimaètre d'ordre r * un plus de tests, celui proposé par jj dans notebook _notebook stage_ et _essai, test_, début de la classe (juste une ébauche), création de matrice creuse pour utiliser directement ratings.csv et ne pas être obligé de faire tourner svd tout le temps * classe pour la dpp dans dpp.py et nouveau notebook _DPP_ pour tester * Add file for requirements for algorithms * Update notebooks with annotations * Modification de dpp.py Ajouts et modifications de classes et de fonctions nécessaires à la vérification, au test (comparaison) et à l'implémentation liés au dpp (determinantal point process) * Modifications de dpp.py Changements mineurs, ajout de jaccard (fonction toute faite), code plus lisible avec moins d'erreurs Utlisation directe de la BDD non encore faite mais bientôt en cours * Modifications de dpp.py et ajouts de notebooks brouillons Récupération des éléments depuis la base de données et non plus depuis ratings.csv Notebooks brouillons "Test classe" et "Test classe-Copy1" où des tests ont été faits. On peut y voir qu'un gros problème subsiste : des éléments sont ou deviennent des "nan" lors de certaines opérations * Modification de la future classe SimilarityMatrix de dpp.py Aucun changement notoire dans dpp.py Création de "test de Similarity[...]" pour créer, tester et modifier la classe Similarity. En cours : création de la matrice liée aux données brutes, en limitant les appels et les users et/ou les oeuvres n'ayant aucun rating * nouveau * Modifications de dpp.py Continuation et presque finition de dpp.py : la classe SimilarityMatrix a été en grande partie refaite. Les classes suivantes, à savoir MangakiUniform et MangakiDPP, et la fonction compare ont été modifiées en conséquence * Modifications de dpp.py et modification du notebook de test "test de similarity_matrix qui sera ds dpp.py.ipynb Suppression d'une fonction inutile dans dpp.py Modification du notebook : un test de dpp.py est fait vers la fin (voir le gros titre/heading "Test "final" de dpp.py" * Modifications de dpp.py Modifications suite aux remarques d'elarnon, dont principalement : -nouveau constructeur de SimilarityMatrix -code plus proche des recommandations PEP8 (utilisation de flake8 en vérification) -utilisation d'une matrice creuse et non plus d'une matrice qu'on remplit de zéros -utilisation de la fonction np.random.choice à la place de random.shuffle Il reste à créer la fonction compare voulue par elarnon notée compare2 en attendant * Modifications de dpp.py Modifications surtout de compare2 (qui remplacera compare) : -changement du diamètre d'ordre 1 pour le diamètre d'ordre 0 -changement du synospsis de la fonction, des arguments A faire : la tester, dans tous les cas * Save * Checkpoint * Dernières modifications dees fichiers liés aux algorithmes liés à la dpp : -dpp.py -buildmatrix.py qui contient une classe construisant principalement une matrice des ratings( users en ligne, works/items en colonne, ratings dans les cellules) à partir de la base de données ou d'un fichier csv Modification faire si buildmatrix.py est gardé, enlever la classe BuildMatrix dans dpp.py * Add pandas to requirements * Remove useless file * Determinantal Point Processes (#201) * implement dpp in mangaki * Dernière version de dpp.py et buildmatrix.py A l'air de bien marcher et vérifie la PEP8 * Modification de dpp.py Changements des noms de variable dans la fonction diameter_0 (variables sans accent et en anglais * Modification de buildmatrix.py Suppression d'une erreur d'inattention : un "rating" s'était incrusté à la place d'un "choice" ... * Intégration de dpp au site Commencement * avancement * Pas grand chose de nouveau : il faudrait prendre en conséquence le fait que l'on peut choisir d'avoir que des mangas ou que des animes avec la dpp * Avancement de l'intégration de dpp * problème d'url et vue pr savoir si l'on doit avoir le mode dpp ou pas * quelques changements * quelques changements * anciens fichiers * essai * Intégration de dpp au site, avec les recommandations cette fois Améliorable (fonctions presque "doubles" se ressemblant pr la version sans dpp et avec dpp * petites modifications mais non encore fini * master * rectifications en cours * changements * encore des rectifications (non fini encore) * pb : affichage anime/mangas seulement ne marche pas * rectifications * oublis * suppression d'une migration inutile (déjà présente ds master en fait) * retrait des recommandations pour dpp pour un utilisateur lambda * début rectificatifs * Rectifications PB avec la popularité normalement réglé (mais pas vraiment testé car j'ai 20 oeuvres en tout, c'est tout ^^). Si testé depuis une grosse seed :: augmenter le nombre d'oeuvres prises en compte ds popular (dpp ds models.py) et le nbre de points du sample de dpp (dans views.py) PB url avec les sort et les keywords dpp encore à faire * améliorations * essais * changements, encore des trucs à faires (les urls et voir dernières remarques elarnon) * amélioration des urls * L'histoire d'url pr dpp est réglé Modification des titres "DPP" en "Découvrir" * dernières (?) rectifications suite aux messages d'elarnon sauf erreur. Il faut encore cleaner le code, vérifier si PEP8 est bien respectée * PEP8 mieux respectée * rectifications * dernière version master * Changement des dépendances des migrations pour que ça marche (management command marche et la loaddata de la seed aussi (mais pas la big seed :/)) * Suppression d'un notebook inutile et n'ayant rien à faire là * rectifications, encore à vérifier en testant sur la version dev de mangaki * Rectifications suite aux remarques de raito * Variables temporaires supprimées dans ratingsmatrix.py car inutiles * typographie * new message error more accurate * Remove heavy files, add migrations * Fix tests * Fix test * Clean code * Add migration for tropes * Add request to server_error, remove handler500 from urls
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
#52 depends on this pull request.
In order to make it easier to manage Mangaki in production, we'll use Ansible to create reproducible deployments, here is the checklist: