Add management command to create organizations' encryption keys #2854
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
FrankApiyo
approved these changes
Jun 27, 2025
kelvin-muchiri
added a commit
that referenced
this pull request
Jul 7, 2025
* add management command to create org encryption key * update module docstrings * resolve lint error line too long
kelvin-muchiri
added a commit
that referenced
this pull request
Jul 9, 2025
* add management command to create org encryption key * update module docstrings * resolve lint error line too long
kelvin-muchiri
added a commit
that referenced
this pull request
Jul 9, 2025
* create KMSKey on creating org * decrypt submission automatically * enable kms encryption for XForm * remove briefcase.log * add test * fix lint errors * fix lint errors * install optional packages in ci * prevent forms with submissions from encryption * add utility method to disable form encryption * update xform version when encrypting xform * add functionality to disable XForm encryption * set up SSH agent in Github CI * resolve permission denied error in CI * Install optional packages when building security check Docker image * setup SSH agent when running security checks * resolve permission denied when runnin static-analysis * resolve permission denied when runnin static-analysis * Github CI clean up * Github CI clean up * Github CI clean up * Github CI clean up * add test * fix lint errors * fix lint errors * fix failing test * fix failing test * fix failing test * resolve decrypted media files deleted * refactor to match valigetta update * fix incorrect argument data type * suppress lint warning * add property is_kms_encrypted to XForm list * automatically encrypt published form * fix failing tests * add property is_kms_encrypted to XForm detail view * fix failing test * fix is_kms_encrypted not rendered * fix bug int object has no encrypted * create XFormVersion after encrypting XForm * raise exception when decrypting unencrypted Instance * add test * refactor tests * resolve cyclic dependency * resolve cyclic dependency * render active KMSKey in org details API response * add tests for is_instance_encrypted * rename symbol * add created_by field in KMSKey model * record user that created key when rotating key * return active KMS key expiry date in org detail * expire old key when rotating key * update doc string * return all active managed keys for an organization * enhance tests * update docstring * make grace period end date static * add test * only update expiry_date during manual rotation * refactor test * refactor code * add indices to KMSKey * set default grace period duration * resolve lint error no-else-return * update hash after encrypting xform * resolve version not updated after encrypting form * wait for transaction to complete * invalidate formList cache after encrypting/unencrypting XForm * fix cyclic dependency * add support for filtering data based on encryption status * update documentation * resolve lint error resolve Import "from onadata.libs.kms.tools import decrypt_instance" should be placed at the top of the module * resolve lint warning no-value-for-parameter * fix incorrect docs formatting * explicitly set active KMS key in org details * add endpoint for manual key rotation * disable rotating disabled key * add rotated_at, rotated_by KMSKey fields * create index for rotated_at field on model KMSKey * capture rotation errors during manual rotation * refactor manual rotation into pre-mature rotation * compute grace period end date during key rotation * add manual key rotation documentation * return key id when returning org keys * use translation strings * return new key_id after rotation * capture audit log manual key rotation * update rotate key audit log verb * refactor rotate key audit log * allow optional reason for key rotation * update documentation * update response from rotate key action * refactor code * do not encrypt XForm if KMSKey for organization not found * resolve lint warning raise-missing-from * add utility method for sending key rotation notification * add test * update docstring * update docstring * add tests for triger_key_rotation * add test * rename method * rename test case * add utility method for disabling keys * log exception when disabling expired keys * do not disable already disabled key * remove unnecessary not refresh_from_db * add async tasks for rotate_expired_keys, disable_expired_keys * refactor code * rename test case * feat: Add can-view and can-view-minor roles * feat: Update postgres: 13 -> 17 * feat: Update form metaperms to include can-view roles * cleanup: Fix issues raised by prospector * tests: Update some failing test cases * feat: Update CI workflows * test: Fix some failing test cases * feat: Create default meta_permissions when a form is created * feat: Add editor-no-download meta role * chore(pylint): Supress warnings related to imports * chore: Move cache invalidation to update_role_by_meta_xform_perms method * chore: Update documentation for permissions * chore: Remove unecessary .all() from metadata query * chore: fix typo * chore: Refactor XForm MetaData creation signal * resolve forms being encrypted twice during key rotation * wrap create_key within a transaction * wrap disable_key within a transaction * add test case * resolve cyclic dependency * address lint warning consider-using-min-builtin * suppress lint warning * resolve cyclic dependency * resolve cyclic imports * revert import refactor * refactor to resolve cyclic dependency * refactor code * invalidate organization cache after creating, rotating, disabling key * refactor tests * limit transaction scope * refactor code * refactor code * refactor code * add comment * add comment * refactor is_kms_encrypted property into field * exclude submissions pending decryption in data endpoint * uncomment tests * rename XForm is_kms_encrypted to is_managed * add is_automatic field to active key information * restrict active key info to organization admins * create XForm xform_meta_perms after XForm is created * refactor code * rename symbol * merge key info in the org details page merge active key and inactive key info explicitly add is_active key to mark the active key * use is_active to get organization active key * update documentation * add docstring * add support for override manual encryption * rename config * feat: Update XForm meta-perms post-save signal * feat: Move XForm meta-perms post_save signal to DataDictionary model * refactor: Clean up metadata creation post save signal * add async task to send key rotation reminder asynchronously * fix incorrect function argument * send notifcation after rotating key * refactor code * update key rotated message * abort decryption if KMSKey is disabled * handle decryption failure * fix key rotation notification sent if date > target date * exclude deleted attachments when generating attachments zip * remove unused imports * reject submission if encryption key disabled accept by default and reject if configuration is set to reject * add tests * refactor code * refactor code * upgrade valigetta package * disable orphan key if get public key fails * suppress lint warning too-many-locals * fix failing tests * fix failing tests * create key alias after creating KMS key * update test * update key KMS description * make encryption key alias unique * add support for KMS API client * upgrade valigetta * add API choice to KMSKey model * create key even if alias exists on KMS * save KMSAPIClient token to avoid unnecessary re-authentiction by different processes * add type hints * chore: Fix failing tests * feat: Add a new editor-no-view role based permission * chore: Fix failing tests * chore: Update documentation * refactor: Create xform_utils and fix cyclic imports * chore: Cleanup & add new test case * cleanup: Revert some test changes * feat: Ensure ReadOnlyRole is not affected by metaperms * feat: Remove editor-no-view metapermissions * chore: Remove unused import * feat: Fix failing tests for data viewset * feat: Ensure metaperms don't affect manager, owner and readonly role * chore: Fix failing test related to metaperms * chore(cleanup): Update tests and use queryset iterator * track the number of decrypted submissions * refactor, add tests * force update decrypted submission count on Instance delete * refactor code * DRY up * decrement decrypted submission count on Instance delete * remove unusued methdod * fix failing test * add test * add async task to commit cached decrypted submission count * ignore un-managed XForm when adjusting decrypted submission count * refactor to allow adjusting counter by delta * expose num_of_pending_decryption_submissions in retrieve XForm endpoint * pin async tasks to master * address lint errors * use @user_master decorator in place of context * fix failing tests * fix failing test * refactor code * fix failing tests * revert compose file updates * fix failing test * handle negative num_of_pending_decryption_submissions * include only processes of interest within transaction * i 7477 nclude only processes of interest within transaction * resolve lint errors * rename methods * update docstring * Use pre_save to decrement `num_of_decrypted_submissions` on Instance soft delete (#2844) * use pre_save to decr num_of_decrypted_submissions on Instance soft delete * rename signal handler * refactor code * update docstring * Send grace period expiry reminder (#2845) * add task for sending grace period expiry reminder notification * add support for multiple grace period reminders * enhance validation for setting KMS_GRACE_EXPIRY_REMINDER_DURATION * enhance comment * resolve lint error possibly-used-before-assignment * Merge decrement/increment counter asynchronous tasks (#2847) * refactor code * fix failing tests * decrement EntityList num_entities if Entity.soft_delete is by-passed decrement EntiyList num_entities even if deleted_at is set directly * refactor async tasks to use base task * feat: Apply metaperms for can view & download instead of can view * chore: Add test case and default xform metadata setting * chore: Fix profile not found issue * chore: Fix failing tests * increment XForm num_of_decrypted_submissions async * add comment * add tests for decrypt_instance_async * increase celery task max_retries to 5 * resolve lint warning * retry valigetta ConnectionException for decrypt instance async task (#2851) * Install valigetta (#2853) * install valigetta package * strip extras for pip-compile * install valigetta via https * Add management command to create organizations' encryption keys (#2854) * add management command to create org encryption key * update module docstrings * resolve lint error line too long --------- Co-authored-by: FrankApiyo <franklineapiyo@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changes / Features implemented
Add management command to create organizations' encryption keys