fix: neDB implementation issues #979

kriswest · 2025-04-15T11:17:19Z

resolves #948
resolves #947

Fixes a number of shortcomings in the file DB implementation, including:

Merge documents when updating user records instead of replacing (to match behaviour of mongo - was dropping gitAccount when record was updated)
Apply basic validation to repository records (to match behaviour of mongo)
Reduce username and email to lowercase consistently in DB clients
Add indexes and compaction settings (db file format is append only and needs periodic compaction)

Changes were also applied to the mongo DB classes:

Be more consistent about lowercasing user details (username and email)
Lowercase repo name consistently
- this change may be controversial (may need migration)
- however a number of functions already lowercase the repo name... e.g. addUserCanPush and addUserCanAuthorise, its just not being applied consistently

Finally, tests were updates:

existing was updated to ensure it uses unique usernames and emails as this is necessary after the indexes are added.
Many more DB tests were introduced. These only cover the fileDB currently, but could be run over mongo implementation if an instance is setup for testing and config applied, perhaps in a GH action

…anyway)

…ing-fixes

…file-based DB implementation

…n't be IRL)

…file-based DB implementation

…ion-issues

netlify · 2025-04-15T11:17:41Z

✅ Deploy Preview for endearing-brigadeiros-63f9d0 ready!

Name	Link
🔨 Latest commit	`37f1684`
🔍 Latest deploy log	https://app.netlify.com/projects/endearing-brigadeiros-63f9d0/deploys/6839cee3600b1300081cbd79
😎 Deploy Preview	https://deploy-preview-979--endearing-brigadeiros-63f9d0.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

codecov · 2025-04-15T11:20:45Z

Codecov Report

Attention: Patch coverage is 77.16535% with 29 lines in your changes missing coverage. Please review.

Project coverage is 56.01%. Comparing base (25dbd56) to head (37f1684).

Files with missing lines	Patch %	Lines
src/plugin.ts	6.25%	15 Missing ⚠️
src/service/passport/oidc.js	0.00%	4 Missing ⚠️
.../proxy/processors/push-action/checkAuthorEmails.ts	0.00%	2 Missing ⚠️
src/proxy/processors/push-action/parsePush.ts	33.33%	2 Missing ⚠️
src/db/file/repo.ts	96.42%	0 Missing and 1 partial ⚠️
src/proxy/actions/Step.ts	75.00%	1 Missing ⚠️
src/proxy/processors/pre-processor/parseAction.ts	50.00%	1 Missing ⚠️
src/proxy/processors/push-action/preReceive.ts	0.00%	1 Missing ⚠️
src/proxy/processors/push-action/pullRemote.ts	66.66%	1 Missing ⚠️
src/proxy/processors/push-action/scanDiff.ts	94.73%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #979      +/-   ##
==========================================
+ Coverage   48.70%   56.01%   +7.30%     
==========================================
  Files          53       53              
  Lines        2166     2196      +30     
  Branches      242      250       +8     
==========================================
+ Hits         1055     1230     +175     
+ Misses       1074      940     -134     
+ Partials       37       26      -11

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

…ab-dedicated.com/natwestgroup/DigitalX/Technology/EnterpriseEngineering/OSPO/finos/fork-git-proxy into 948-neDB-implementation-issues

…' into 948-neDB-implementation-issues

kriswest · 2025-04-16T17:09:07Z

This PR has one more push to come that will hopefully complete the code coverage. Should be reviewed and pushed tomorrow

kriswest · 2025-04-17T16:24:40Z

One last commit to push on this PR that should get coverage on /src/db/file up to near 100% - might get pushed after easter now.

kriswest · 2025-04-22T09:35:49Z

@JamieSlome @coopernetes @grovesy this PR should be ready to go now

JamieSlome

@kriswest - before I review pre-merge, can we confirm that the linting changes are aligned with the current config and expectation asserted by the project?

…ran it

kriswest · 2025-05-07T13:18:46Z

@JamieSlome I was set-up as per the project config, however the project config doesn't cover packages/git-proxy-cli! I've tweaked the format command so that it does apply and run it. I tried to add prettier to the child project directly but it's not working there, so I extended the parent project's format command to cover it instead.

This adds a bunch of files to the PR with minor changes - waiting on a review then will push

kriswest · 2025-05-07T13:46:08Z

@JamieSlome prettier was run on everything and should be applying to packages/git-proxy-cli. It looks like that was previously formatted with different settings / narrower width. Does it look right to you?

…into 948-neDB-implementation-issues

kriswest added 21 commits April 10, 2025 13:16

fix: remove incorrect default query from repos page (that is ignored …

4ef686f

…anyway)

fix: passthrough repository queries

232c466

test: use unique emails for users in tests and remove afterwards

ab362e4

Merge remote-tracking branch 'upstream/main' into 947-repository-list…

2e3282d

…ing-fixes

test: don't clean-up test-repo as cypress tests rely on it

12e7bfe

fix: add indexes, compaction and consistent lowercasing of inputs in …

200788c

…file-based DB implementation

fix: consistent lowercasing of inputs in mongoDB implementation

9769be6

fix: make repo name index non-unique (currently assumed unique but wo…

188d820

…n't be IRL)

test: use unique emails for users in tests and remove afterwards

f38c266

fix: add indexes, compaction and consistent lowercasing of inputs in …

4632e86

…file-based DB implementation

chore: porting changes from users.js

40a83b5

fix: bad merge

6f08207

test: don't clean-up test-repo as cypress tests rely on it

092ec13

fix: add indexes, compaction and consistent lowercasing of inputs in …

08a6c82

…file-based DB implementation

test: use unique emails for users in tests and remove afterwards

1966088

fix: add indexes, compaction and consistent lowercasing of inputs in …

8463c71

…file-based DB implementation

fix: consistent lowercasing of inputs in mongoDB implementation

fa5f270

chore: porting changes from users.js

4abbe28

fix: bad merge

8a73e5a

test: don't clean-up test-repo as cypress tests rely on it

7aaf03c

Merge branch '947-repository-listing-fixes' into 948-neDB-implementat…

5db446b

…ion-issues

kriswest changed the title ~~948 neDB implementation issues~~ fix: neDB implementation issues Apr 15, 2025

github-actions bot added the fix label Apr 15, 2025

kriswest added 5 commits April 15, 2025 13:23

test: basic db test coverage

008e4fa

fix: clean-up DB after CLI tests

fa234f8

Merge branch '948-neDB-implementation-issues' of https://natwest.gitl…

b67a81c

…ab-dedicated.com/natwestgroup/DigitalX/Technology/EnterpriseEngineering/OSPO/finos/fork-git-proxy into 948-neDB-implementation-issues

test: more DB test coverage

734fc30

test: more code coverage in DB and service/routes/repo

4ff22a3

This was referenced Apr 16, 2025

fix: push parsing drops commit messages unless they end in a newline #976

Open

fix: repository listing default query #975

Closed

fix: 946 associate commits by email #973

Open

kriswest added 2 commits April 16, 2025 18:03

Merge branch 'main' into 948-neDB-implementation-issues

520a9bc

Merge remote-tracking branch 'upstream/948-neDB-implementation-issues…

79a30ba

…' into 948-neDB-implementation-issues

test: more DB test coverage

fe64a91

coopernetes self-assigned this Apr 21, 2025

Merge branch 'main' into 948-neDB-implementation-issues

ec4516a

JamieSlome reviewed May 6, 2025

View reviewed changes

kriswest added 2 commits May 7, 2025 13:39

Merge branch 'main' into 948-neDB-implementation-issues

ddcc149

chore: extended prettier format command to package/git-proxy-cli and …

b5cd038

…ran it

kriswest and others added 2 commits May 19, 2025 16:14

Merge branch 'main' into 948-neDB-implementation-issues

12f20c9

Merge branch 'main' into 948-neDB-implementation-issues

c49ae45

kriswest mentioned this pull request May 23, 2025

Validate that a project doesn't already exist in GitProxy before creation (to avoid duplicates) #1028

Open

Merge branch 'main' into 948-neDB-implementation-issues

b434e43

kriswest mentioned this pull request May 30, 2025

feat: Create admin protected endpoint for creating users #981

Open

kriswest added 2 commits May 30, 2025 16:25

Merge branch 'main' into 948-neDB-implementation-issues

fe3b5a0

Merge remote-tracking branch 'github/948-neDB-implementation-issues' …

37f1684

…into 948-neDB-implementation-issues

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix: neDB implementation issues #979

fix: neDB implementation issues #979

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

fix: neDB implementation issues #979

Are you sure you want to change the base?

fix: neDB implementation issues #979

Conversation

Uh oh!

Uh oh!

Uh oh!

✅ Deploy Preview for endearing-brigadeiros-63f9d0 ready!

Uh oh!

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!