File-based DB support implementation is not usable out-of-the-box and deviates from the mongo implementation

Describe the bug
There appear to be a number of flaws in the file-based DB implementation that impede its use. These include some small deviations from the mongo implementation AND some deviations in behaviotr between @seald/nedb and mongo that should be handled in the implementation.

lowercasing of inputs (particularly of usernames and emails) is inconsistently applied and differs from mongo implementation. In places, inputs are lowercased before being sent to the database client and hence apply to both client implementations, however, the mongo clients applies lowercasing on its own in other situations. The database clients should consistently lowercase inputs where appropriate (in both record creation and query) to ensure consistent application. Example of deviation (there are many more cases):

Mongo:

git-proxy/src/db/mongo/repo.js

Lines 41 to 45 in 5d24d9d

    
           exports.addUserCanPush = async (name, user) => { 
        
             name = name.toLowerCase(); 
        
             const collection = await connect(cnName); 
        
             await collection.updateOne({ name: name }, { $push: { 'users.canPush': user } }); 
        
           };

File:

git-proxy/src/db/file/repo.js

Lines 54 to 73 in 5d24d9d

    
           exports.addUserCanPush = async (name, user) => { 
        
             return new Promise(async (resolve, reject) => { 
        
               const repo = await exports.getRepo(name); 
        
               if (repo.users.canPush.includes(user)) { 
        
                 resolve(null); 
        
                 return; 
        
               } 
        
               repo.users.canPush.push(user); 
        
               const options = { multi: false, upsert: false }; 
        
               db.update({ name: name }, repo, options, (err) => { 
        
                 if (err) { 
        
                   reject(err); 
        
                 } else { 
        
                   resolve(null); 
        
                 } 
        
               }); 
        
             }); 
        
           };

Queries are being ignored in user retrieval, causing all users to be retrieved and push parsing to fail. Query being ignored:

git-proxy/src/db/file/users.js

Lines 78 to 81 in 5d24d9d

    
           exports.getUsers = function (query) { 
        
             if (!query) query = {}; 
        
             return new Promise((resolve, reject) => { 
        
               db.find({}, (err, docs) => {

(note that query is not passed in on last nline)
Push parsing that requires DB filtering of users that is not happening:

git-proxy/src/proxy/processors/push-action/checkUserPushPermission.js

Lines 13 to 20 in 5d24d9d

    
           const list = await db.getUsers({ gitAccount: action.user }); 
        
           console.log(JSON.stringify(list)); 
        
           if (list.length == 1) { 
        
             user = list[0].username; 
        
             isUserAllowed = await db.isUserPushAllowed(repoName, user); 
        
           }

(note exactly one user must be found on line 17 - yet we are not actually filtering users)

The implementation does not merge records on update, but replaces them. This causes fields not supplied in the update to be dropped (e.g. if a user is updated on login via an SSO integration, then the Github user name is dropped as that is manually entered later).

git-proxy/src/db/file/users.js

Lines 65 to 76 in 5d24d9d

    
           exports.updateUser = function (user) { 
        
             return new Promise((resolve, reject) => { 
        
               const options = { multi: false, upsert: false }; 
        
               db.update({ username: user.username }, user, options, (err) => { 
        
                 if (err) { 
        
                   reject(err); 
        
                 } else { 
        
                   resolve(null); 
        
                 } 
        
               }); 
        
             }); 
        
           };

(this looks correct by @seald/nedb does not actually merge records when called like this - it finds the existing record and replaces it with the user passed in. This needs rewriting to either use the $set keyword or to manually retrieve the existing doc and merge into before calling db.update.

neDB implementations should be set to compact their data files periodically, unless they are restarted regularly (compacts on read) - neDB uses an append-only file format so it will create many duplicate lines, See https://github.com/seald/nedb?tab=readme-ov-file#persistence. Add calls to set-up compaction to avoid infinite growth and very redundant data files.
No indexes are applied, which will result in performance losses as the dataset grows. Add calls to create indexes supporting common queries - these calls should be made on every start-up. See: https://github.com/seald/nedb?tab=readme-ov-file#indexing for details
Expected behavior
The behaviour of the file-based DB support should match that of the mongo implementation and it should work out-of-the-box.

Additional context
I'll raise a PR to address these issues in the implementation.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	exports.addUserCanPush = async (name, user) => {
	name = name.toLowerCase();
	const collection = await connect(cnName);
	await collection.updateOne({ name: name }, { $push: { 'users.canPush': user } });
	};

	exports.addUserCanPush = async (name, user) => {
	return new Promise(async (resolve, reject) => {
	const repo = await exports.getRepo(name);

	if (repo.users.canPush.includes(user)) {
	resolve(null);
	return;
	}
	repo.users.canPush.push(user);

	const options = { multi: false, upsert: false };
	db.update({ name: name }, repo, options, (err) => {
	if (err) {
	reject(err);
	} else {
	resolve(null);
	}
	});
	});
	};

	exports.getUsers = function (query) {
	if (!query) query = {};
	return new Promise((resolve, reject) => {
	db.find({}, (err, docs) => {

	const list = await db.getUsers({ gitAccount: action.user });

	console.log(JSON.stringify(list));

	if (list.length == 1) {
	user = list[0].username;
	isUserAllowed = await db.isUserPushAllowed(repoName, user);
	}

	exports.updateUser = function (user) {
	return new Promise((resolve, reject) => {
	const options = { multi: false, upsert: false };
	db.update({ username: user.username }, user, options, (err) => {
	if (err) {
	reject(err);
	} else {
	resolve(null);
	}
	});
	});
	};

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions