8000 Refactor scheduler by shimonewman · Pull Request #4641 · appwrite/appwrite · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Refactor scheduler #4641

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 77 commits into from
Nov 16, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
77 commits
Select commit Hold shift + click to select a range
bc631d9
ticker
shimonewman Nov 4, 2022
ed505fb
queue
shimonewman Nov 6, 2022
0e99198
queue
shimonewman Nov 6, 2022
ef7b53f
queue
shimonewman Nov 6, 2022
6f9dcae
queue
shimonewman Nov 7, 2022
9ed6ed6
queue
shimonewman Nov 7, 2022
4409217
queue
shimonewman Nov 7, 2022
3768912
replacing offset with pagination
shimonewman Nov 7, 2022
711d806
replacing offset with pagination
shimonewman Nov 7, 2022
3a651ad
replacing offset with pagination
shimonewman Nov 7, 2022
9d641b9
tidy up
shimonewman Nov 8, 2022
1761b77
function worker
shimonewman Nov 9, 2022
aef565c
addressing comments
shimonewman Nov 10, 2022
745ae7a
Merge branch 'feat-db-pools' of github.com:appwrite/appwrite into ref…
shimonewman Nov 10, 2022
3dc619f
addressing comments
shimonewman Nov 10, 2022
9e3b9b8
addressing comments
shimonewman Nov 11, 2022
134dba0
Fix startup errors
Meldiron Nov 12, 2022
250ea93
some fixes
shimonewman Nov 13, 2022
5e7de81
Improve schedule accuracy + simplify
Meldiron Nov 14, 2022
83e19a3
minor changes
shimonewman Nov 14, 2022
9626d86
Merge remote-tracking branch 'origin/refactor-scheduler' into refacto…
Meldiron Nov 14, 2022
c2e8fc5
Increase timer delay
Meldiron Nov 14, 2022
de8122c
Share coroutines for executions
Meldiron Nov 14, 2022
ff06e4b
Move schedule delete TODO
Meldiron Nov 15, 2022
30e39f0
Merge branch 'feat-db-pools' into refactor-scheduler-improvements
Meldiron Nov 15, 2022
22effdd
Refactor schedule task to new syntax
Meldiron Nov 15, 2022
b031e13
Improve reclaim of pools in schedule
Meldiron Nov 15, 2022
c13589c
Bug fix
Meldiron Nov 15, 2022
1084631
Fux reclaim logic in CLI; Prevent early executions
Meldiron Nov 15, 2022
9539e27
Linter fix
Meldiron Nov 15, 2022
98fbe18
Merge branch 'feat-db-pools' of https://github.com/appwrite/appwrite …
christyjacob4 Nov 15, 2022
174f57c
Merge branch 'refactor-scheduler' of https://github.com/appwrite/appw…
christyjacob4 Nov 15, 2022
727338c
Imrpove pools relciam logic
Meldiron Nov 15, 2022
4a126ef
Merge branch 'refactor-scheduler-improvements' of https://github.com/…
Meldiron Nov 15, 2022
ea5bd51
Linter fix
Meldiron Nov 15, 2022
bb3c99c
Rename file step 1
Meldiron Nov 15, 2022
451c633
Rename file step 2
Meldiron Nov 15, 2022
bfe8b98
Attempt to reuse db connections
Meldiron Nov 15, 2022
b6e4ec8
Add TODO
Meldiron Nov 15, 2022
c07953a
Linter fix
Meldiron Nov 15, 2022
e042959
Add TODO
Meldiron Nov 15, 2022
6be62fb
Merge pull request #4668 from appwrite/refactor-scheduler-improvements
christyjacob4 Nov 15, 2022
390fbc5
minor changes
shimonewman Nov 15, 2022
8bcf349
addressing some comments
shimonewman Nov 15, 2022
3682c1d
addressing some comments
shimonewman Nov 15, 2022
0d79f19
addressing some comments
shimonewman Nov 15, 2022
11915e1
addressing some comments
shimonewman Nov 15, 2022
797c3f2
addressing some comments
shimonewman Nov 15, 2022
eb47894
Merge branch 'refactor-scheduler' into refactor-functions-worker
Meldiron Nov 15, 2022
855c3d3
Re-implement removed stuff during merge
Meldiron Nov 15, 2022
280a44e
Fix bugs during QA
Meldiron Nov 15, 2022
44a82de
Refactor func event triggering
Meldiron Nov 15, 2022
6bf370a
QA bug fixes
Meldiron Nov 15, 2022
4a92db4
feat: check async execution
christyjacob4 Nov 16, 2022
3bd3b8d
feat: check async execution
christyjacob4 Nov 16, 2022
586fd46
feat: check async execution
christyjacob4 Nov 16, 2022
5abe9ad
feat: refactor execute function to a resource
christyjacob4 Nov 16, 2022
8b0a78b
fix: function events and linter
christyjacob4 Nov 16, 2022
e976bd3
fix: event triggers
christyjacob4 Nov 16, 2022
dbb05c4
fix: event triggers
christyjacob4 Nov 16, 2022
4e3bcd6
Authorization::skip added
shimonewman Nov 16, 2022
6b88f66
removed pools injection in functions controller
shimonewman Nov 16, 2022
ac70537
resetting workers-number
shimonewman Nov 16, 2022
0d61a7f
resetting workers-number
shimonewman Nov 16, 2022
44875e2
QA review changes
Meldiron Nov 16, 2022
9e4a656
Simplify worker
Meldiron Nov 16, 2022
390f407
Linter fixes
Meldiron Nov 16, 2022
83639ea
Merge branch 'feat-db-pools' into refactor-functions-worker
Meldiron Nov 16, 2022
43adf75
Fix lockfile after merge
Meldiron Nov 16, 2022
fdbb5b9
QA bug fixing
Meldiron Nov 16, 2022
3e325af
Revert unnessessary changes
Meldiron Nov 16, 2022
3b6174b
Bug fixes
Meldiron Nov 16, 2022
8b000af
Merge branch 'feat-db-pools' into refactor-scheduler
Meldiron Nov 16, 2022
284150a
Merge branch 'refactor-scheduler' into refactor-functions-worker
Meldiron Nov 16, 2022
e37f702
PR review changes
Meldiron Nov 16, 2022
5f8d3ff
More PR reviews
Meldiron Nov 16, 2022
0c759e9
Merge pull request #4659 from appwrite/refactor-functions-worker
christyjacob4 Nov 16, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .env
Original file line number Diff line number Diff line change
Expand Up @@ -87,5 +87,6 @@ _APP_USAGE_DATABASE_INTERVAL=15
_APP_USAGE_STATS=enabled
_APP_LOGGING_PROVIDER=
_APP_LOGGING_CONFIG=
_APP_REGION=default
_APP_DOCKER_HUB_USERNAME=
_APP_DOCKER_HUB_PASSWORD=
_APP_DOCKER_HUB_PASSWORD=
22 changes: 18 additions & 4 deletions app/cli.php
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
require_once __DIR__ . '/init.php';
require_once __DIR__ . '/controllers/general.php';

use Appwrite\Event\Func;
use Appwrite\Platform\Appwrite;
use Utopia\CLI\CLI;
use Utopia\Database\Validator\Authorization;
Expand Down Expand Up @@ -56,20 +57,29 @@
}, ['pools', 'cache']);

CLI::setResource('getProjectDB', function (Group $pools, Database $dbForConsole, $cache) {
$getProjectDB = function (Document $project) use ($pools, $dbForConsole, $cache) {
$databases = []; // TODO: @Meldiron This should probably be responsibility of utopia-php/pools

$getProjectDB = function (Document $project) use ($pools, $dbForConsole, $cache, &$databases) {
if ($project->isEmpty() || $project->getId() === 'console') {
return $dbForConsole;
}

$databaseName = $project->getAttribute('database');

if (isset($databases[$databaseName])) {
return $databases[$databaseName];
}

$dbAdapter = $pools
->get($project->getAttribute('database'))
->get($databaseName)
->pop()
->getResource()
;
->getResource();

$database = new Database($dbAdapter, $cache);
$database->setNamespace('_' . $project->getInternalId());

$databases[$databaseName] = $database;

return $database;
};

Expand Down Expand Up @@ -100,6 +110,10 @@
return $database;
}, ['register']);

CLI::setResource('queueForFunctions', function (Group $pools) {
return new Func($pools->get('queue')->pop()->getResource());
}, ['pools']);

CLI::setResource('logError', function (Registry $register) {
return function (Throwable $error, string $namespace, string $action) use ($register) {
$logger = $register->get('logger');
Expand Down
142 changes: 109 additions & 33 deletions app/config/collections.php
Original file line number Diff line number Diff line change
Expand Up @@ -751,6 +751,107 @@
],
],

'schedules' => [
'$collection' => ID::custom(Database::METADATA),
'$id' => ID::custom('schedules'),
'name' => 'schedules',
'attributes' => [
[
'$id' => ID::custom('resourceType'),
'type' => Database::VAR_STRING,
'format' => '',
'size' => 100,
'signed' => true,
'required' => false,
'default' => null,
'array' => false,
'filters' => [],
],
[
'$id' => ID::custom('resourceId'),
'type' => Database::VAR_STRING,
'format' => '',
'size' => Database::LENGTH_KEY,
'signed' => true,
'required' => false,
'default' => null,
'array' => false,
'filters' => [],
],
[
'$id' => ID::custom('resourceUpdatedAt'),
'type' => Database::VAR_DATETIME,
'format' => '',
'size' => 0,
'signed' => false,
'required' => false,
'default' => null,
'array' => false,
'filters' => ['datetime'],
],
[
'$id' => ID::custom('projectId'),
'type' => Database::VAR_STRING,
'format' => '',
'size' => Database::LENGTH_KEY,
'signed' => true,
'required' => false,
'default' => null,
'array' => false,
'filters' => [],
],
[
'$id' => ID::custom('schedule'),
'type' => Database::VAR_STRING,
'format' => '',
'size' => 100,
'signed' => true,
'required' => false,
'default' => null,
'array' => false,
'filters' => [],
],
[
'$id' => ID::custom('active'),
'type' => Database::VAR_BOOLEAN,
'signed' => true,
'size' => 0,
'format' => '',
'filters' => [],
'required' => false,
'default' => null,
'array' => false,
],
[
'$id' => ID::custom('region'),
'type' => Database::VAR_STRING,
'format' => '',
'size' => 10,
'signed' => true,
'required' => true,
'default' => null,
'array' => false,
'filters' => [],
],
],
'indexes' => [
[
'$id' => ID::custom('_key_region_resourceType_resourceUpdatedAt'),
'type' => Database::INDEX_KEY,
'attributes' => ['region', 'resourceType','resourceUpdatedAt'],
'lengths' => [],
'orders' => [],
],
[
'$id' => ID::custom('_key_region_resourceType_projectId_resourceId'),
'type' => Database::INDEX_KEY,
'attributes' => ['region', 'resourceType', 'projectId', 'resourceId'],
'lengths' => [],
'orders' => [],
],
],
],

'platforms' => [
'$collection' => ID::custom(Database::METADATA),
'$id' => ID::custom('platforms'),
Expand Down Expand Up @@ -2149,40 +2250,29 @@
'filters' => [],
],
[
'$id' => ID::custom('schedule'),
'$id' => ID::custom('scheduleId'),
'type' => Database::VAR_STRING,
'format' => '',
'size' => 128,
'size' => Database::LENGTH_KEY,
'signed' => true,
'required' => false,
'default' => null,
'array' => false,
'filters' => [],
],
[
'$id' => ID::custom('scheduleUpdatedAt'), // Used to fix duplicate executions bug. Can be removed once new queue library is used
'type' => Database::VAR_DATETIME,
'format' => '',
'size' => 0,
'signed' => false,
'required' => false,
'default' => null,
'array' => false,
'filters' => ['datetime'],
],
[
'$id' => ID::custom('schedulePrevious'),
'type' => Database::VAR_DATETIME,
'$id' => ID::custom('schedule'),
'type' => Database::VAR_STRING,
'format' => '',
'size' => 0,
'signed' => false,
'size' => 128,
'signed' => true,
'required' => false,
'default' => null,
'array' => false,
'filters' => ['datetime'],
'filters' => [],
],
[
'$id' => ID::custom('scheduleNext'),
'$id' => ID::custom('scheduleUpdatedAt'),
'type' => Database::VAR_DATETIME,
'format' => '',
'size' => 0,
Expand Down Expand Up @@ -2258,20 +2348,6 @@
'lengths' => [],
'orders' => [Database::ORDER_ASC],
],
[
'$id' => ID::custom('_key_scheduleNext'),
'type' => Database::INDEX_KEY,
'attributes' => ['scheduleNext'],
'lengths' => [],
'orders' => [Database::ORDER_ASC],
],
[
'$id' => ID::custom('_key_schedulePrevious'),
'type' => Database::INDEX_KEY,
'attributes' => ['schedulePrevious'],
'lengths' => [],
'orders' => [Database::ORDER_ASC],
],
[
'$id' => ID::custom('_key_timeout'),
'type' => Database::INDEX_KEY,
Expand Down
Loading
0