Refine master-master mode support #42

Totktonada · 2020-10-23T22:56:48Z

Initially the module leant on box.cfg.replication_source (box.cfg.replication for tarantool-1.7.6+) to don't start task processing on a replica. Here the word replica means an instance with at least one configured upstream.

Then something in the world changes and it becomes usual to use bidirectional replication with tarantool. Most of times it is not a real master-master: all writes are going to a single instance at given moment of the time. But it allows to skip replication reconfiguration step when switching an instance that aimed to process writes. This way decreases a time for the switching and decreases probability of mistakes during replica set reconfiguration. box.cfg{read_only = true|false} is often used to protect instances, where writes should not be performed.

In this confguration the expirationd module should process tasks only on the writeable instance and so checking of a box.cfg.replication is not sufficient. The option force was added with this idea in the mind. It let a user control where to start task processing and where don't. However it obligates an application developer to externally stop tasks processing on an instance that goes to read-only and start it on an instance that goes to be writeable. It is not quite obvious.

This ideal solution would be ability to configure expirationd to obtain information about an instance state. Say, 'stop processing when box.info.ro becomes true and start it when it becomes false'.

I propose the following API:

local expirationd = require('expirationd')

expirationd.start(name, space_id, is_tuple_expired, {
    <...>
    when = <function>,  -- process tuples only when this condition is true;
                        -- this may be a user-defined function or a predefined one:
                        -- * `expirationd.has_no_upstreams` (it is default)
                        -- * `expirationd.is_writeable` (usually good for master-master setup)
    force = <boolean>,  -- process tuples despite `opts.when` value
})

--
-- Built-in functions to use in `opts.when`.
--
-- Implementations are listed here to make the idea more clear.
--

expirationd.has_no_upstreams = function()
    -- `replication_source` is the old name of `replication` parameter (prior to 1.7.6).
    -- `box.cfg{replication = {}}` may be used to withdraw from upstreams.
    local upstreams = box.cfg.replication or box.cfg.replication_source
    return upstreams == nil or (type(upstreams) == 'table' and next(upstreams) == nil)
end

expirationd.is_writeable = function()
    -- Not sure whether `box.info.ro` can be `nil` during box configuration.
    -- Consider `nil` as 'read-only' just in case.
    return box.info.ro == false
end

The discussion was started in PR #37 by @filonenko-mikhail, then was stale for a long time. I filed the issue to discuss and finally decide what would be most convenient way to configure the module in a multi-instance environment.

The text was updated successfully, but these errors were encountered:

Totktonada · 2020-10-24T01:50:15Z

As I see, the condition whether to proceed with tuples is verified in two cases:

Before traverse a space (the whole space!).
When a worker fiber is respawned (after an error in is_tuple_expired() or process_expired_tuple()).

expirationd/expirationd.lua

Lines 162 to 164 in a700bd1

    
           if (box.cfg.replication_source == nil and box.cfg.replication == nil) or task.force then 
        
               task.do_worker_iteration(task) 
        
           end

I propose to check the opts.when function result after each tuples_per_iteration tuples for the sake of flexibility. This way we'll able to gracefully stop tuples processing, not only when an expiration operation start to raising errors.

Totktonada · 2022-05-31T22:02:17Z

I would change my proposal a bit now. I don't see much sense in introducing options here. Let's just lean on box.info.ro (not box.cfg.read_only!) instead of box.info.replication. It is technically incompatible change, but at the same time it looks safe enough for real usages.

Details:

Lean on box.info.ro. Stop tasks processing in the read-only state. Start it in the read-write state.
But! Wait for upstream lag * 2 before start processing of tasks (to don't hit a conflict due to parallel processing of remaining transactions from the old master and from the new master).
But! Ignore box.info.ro for instance local and temporary spaces (see the table here).
Keep force option just in case (for some very unusual cases).

See tarantool/queue#120 (comment). We recently solved quite similar problem in the queue module.

oleg-jukovec · 2022-07-13T07:22:09Z

Notes:

The implementation of upstream lag * 2 in queue:
https://github.com/tarantool/queue/blob/ac1a9af96dae9bc1d73a0de52875c45ff361a8b6/queue/abstract/queue_state.lua#L72
box.info.ro since 1.7.4, box.info.server.ro before:
tarantool/tarantool@56462bc
box.info.replication[i].upstream.lag since 1.7.4:
tarantool/tarantool@7ed4848
box.info.replication[i].lag since 1.7.1:
tarantool/tarantool@57ce8b9

A minimum possible version is 1.7.1 (we need to calculate a maximum lag for upstreams), but it will require additional code. There is not much difference between 1.7.1 and 1.7.4 from my point of view. So I think the issue is a good reason to bump a required Tarantool version from 1.6.8 to 1.7.4.

box.info.ro [1] and box.info.replication[n].lag [2] were introduced in Tarantool 1.7.4. 1. tarantool/tarantool@56462bc 2. tarantool/tarantool@7ed4848 Part of #42

expirationd does not process task by default if a replication is configured for a node. It does not matter the node is master or replica. The patch improves the behavior. The behavior becomes different for RW and RO nodes [1]: * A task processes for all types of spaces for RW nodes. * A task processes for temporary and local spaces for RO nodes. 1. https://www.tarantool.io/en/doc/latest/reference/configuration/#confval-read_only Closes #42

expirationd does not process task by default if a replication is configured for a node. It does not matter the node is master or replica. The patch improves the behavior. The behavior becomes different for RW and RO nodes [1] by default: * A task processes for all types of spaces for RW nodes. * A task processes for temporary and local spaces for RO nodes. An user can still force the processing using `force` option. 1. https://www.tarantool.io/en/doc/latest/reference/configuration/#confval-read_only Closes #42

expirationd does not process task by default if a replication is configured for a node. It does not matter the node is master or replica. The patch improves the behavior. The behavior becomes different for RW and RO nodes [1] by default: * A task processes on all types of spaces for RW nodes. * A task processes on temporary and local spaces for RO nodes. An user can still force the processing using `force` option. 1. https://www.tarantool.io/en/doc/latest/reference/configuration/#confval-read_only Closes #42

expirationd does not process a task by default if a replication is configured for a node. It does not matter the node is a master or a replica. The patch improves the behavior. The behavior becomes different for RW and RO nodes [1] by default: * A task processes on all types of spaces for RW nodes. * A task processes on temporary and local spaces for RO nodes. An user can still force the processing using the `force` option. 1. 8000 https://www.tarantool.io/en/doc/latest/reference/configuration/#confval-read_only Closes #42

expirationd does not process a task by default if a replication is configured for a node. It does not matter the node is a master or a replica. The patch improves the behavior. The behavior becomes different for RW and RO nodes [1] by default: * A task processes on all types of spaces for RW nodes. * A task processes on temporary and local spaces for RO nodes. An user can still force the processing using the `force` option. 1. https://www.tarantool.io/en/doc/latest/reference/configuration/#confval-read_only Closes #42

expirationd does not process a task by default if a replication is configured for a node. It does not matter the node is a master or a replica. The patch improves the behavior. The behavior becomes different for RW and RO nodes [1] by default: * A task processes on all types of spaces for a RW node. * A task processes on temporary and local spaces for a RO node. An user can still force the processing using the `force` option. 1. https://www.tarantool.io/en/doc/latest/reference/configuration/#confval-read_only Closes #42

expirationd does not process a task by default if a replication is configured for a node. It does not matter the node is a master or a replica. The patch improves the behavior. The behavior beco 10000 mes different for RW and RO nodes [1] by default: * A task processes on all types of spaces for a RW node. * A task processes on temporary and local spaces for a RO node. An user can still force the processing using the `force` option. 1. https://www.tarantool.io/en/doc/latest/reference/configuration/#confval-read_only Closes #42

expirationd does not process a task by default if a replication is configured for a node. It does not matter the node is a master or a replica. The patch improves the behavior. The behavior becomes different for RW and RO nodes [1] by default: * A task processes on all types of spaces for a RW node. * A task processes on temporary and local spaces for a RO node. An user can still force the processing using the `force` option. 1. https://www.tarantool.io/en/doc/latest/reference/configuration/#confval-read_only Closes #42

box.info.ro [1] and box.info.replication[n].lag [2] were introduced in Tarantool 1.7.4. 1. tarantool/tarantool@56462bc 2. tarantool/tarantool@7ed4848 Part of #42

expirationd does not process a task by default if a replication is configured for a node. It does not matter the node is a master or a replica. The patch improves the behavior. The behavior becomes different for RW and RO nodes [1] by default: * A task processes on all types of spaces for a RW node. * A task processes on temporary and local spaces for a RO node. An user can still force the processing using the `force` option. 1. https://www.tarantool.io/en/doc/latest/reference/configuration/#confval-read_only Closes #42

box.info.ro [1] and box.info.replication[n].lag [2] were introduced in Tarantool 1.7.4. 1. tarantool/tarantool@56462bc 2. tarantool/tarantool@7ed4848 Part of #42

expirationd does not process a task by default if a replication is configured for a node. It does not matter the node is a master or a replica. The patch improves the behavior. The behavior becomes different for RW and RO nodes [1] by default: * A task processes on all types of spaces for a RW node. * A task processes on temporary and local spaces for a RO node. An user can still force the processing using the `force` option. 1. https://www.tarantool.io/en/doc/latest/reference/configuration/#confval-read_only Closes #42

box.info.ro [1] and box.info.replication[n].lag [2] were introduced in Tarantool 1.7.4. 1. tarantool/tarantool@56462bc 2. tarantool/tarantool@7ed4848 Part of #42

expirationd does not process a task by default if a replication is configured for a node. It does not matter the node is a master or a replica. The patch improves the behavior. The behavior becomes different for RW and RO nodes [1] by default: * A task processes on all types of spaces for a RW node. * A task processes on temporary and local spaces for a RO node. An user can still force the processing using the `force` option. 1. https://www.tarantool.io/en/doc/latest/reference/configuration/#confval-read_only Closes #42

box.info.ro [1] and box.info.replication[n].lag [2] were introduced in Tarantool 1.7.4. 1. tarantool/tarantool@56462bc 2. tarantool/tarantool@7ed4848 Part of #42

expirationd does not process a task by default if a replication is configured for a node. It does not matter the node is a master or a replica. The patch improves the behavior. The behavior becomes different for RW and RO nodes [1] by default: * A task processes on all types of spaces for a RW node. * A task processes on temporary and local spaces for a RO node. An user can still force the processing using the `force` option. 1. https://www.tarantool.io/en/doc/latest/reference/configuration/#confval-read_only Closes #42

Overview This release adds a Tarantool Cartridge role for expirationd package and improves the default behavior. Breaking changes None. New features Continue a task from a last tuple (#54). Decrease tarantool-checks dependency from 3.1 to 2.1 (#124). Process a task on a writable space by default (#42). Wait until a space or an index is created (#68, #116). Tarantool Cartridge role (#107). Bugfixes Fix build and installation of rpm/deb packages (#124). Do not restart work fiber if index does not exist (#64). Update changelogs for rpm/deb packages. Testing Shuffle tests (#118). Fix test_mvcc_vinyl_tx_conflict (#104, #105). Fix flaky 'simple expires test' (#90). Other Add GitHub Actions workflow with debug Tarantool build (#102). Add GitHub Actions workflow for deploying module packages to S3 based repositories (#43).

Overview This release adds a Tarantool Cartridge role for expirationd package and improves the default behavior. Breaking changes None. New features Continue a task from a last tuple (#54). Decrease tarantool-checks dependency from 3.1 to 2.1 (#124). Process a task on a writable space by default (#42). Wait until a space or an index is created (#68, #116). Tarantool Cartridge role (#107). Bugfixes Fix build and installation of rpm/deb packages (#124). Do not restart a work fiber if an index does not exist (#64). Update changelogs for rpm/deb packages. Testing Shuffle tests (#118). Fix test_mvcc_vinyl_tx_conflict (#104, #105). Fix flaky 'simple expires test' (#90). Other Add GitHub Actions workflow with debug Tarantool build (#102). Add GitHub Actions workflow for deploying module packages to S3 based repositories (#43).

Overview This release adds a Tarantool Cartridge role for expirationd package and improves the default behavior. Breaking changes None. Deprecated Obsolete functions: task_stats, kill_task, get_task, get_tasks, run_task, show_task_list. New features Continue a task from a last tuple (#54). Decrease tarantool-checks dependency from 3.1 to 2.1 (#124). Process a task on a writable space by default (#42). Wait until a space or an index is created (#68, #116). Tarantool Cartridge role (#107). Bugfixes Fix build and installation of rpm/deb packages (#124). Do not restart a work fiber if an index does not exist (#64). Update changelogs for rpm/deb packages. Testing 67ED Shuffle tests (#118). Fix test_mvcc_vinyl_tx_conflict (#104, #105). Fix flaky 'simple expires test' (#90). Other Add GitHub Actions workflow with debug Tarantool build (#102). Add GitHub Actions workflow for deploying module packages to S3 based repositories (#43).

Overview This release adds a Tarantool Cartridge role for expirationd package and improves the default behavior. Breaking changes None. Deprecated Obsolete functions: task_stats, kill_task, get_task, get_tasks, run_task, show_task_list. New features Continue a task from a last tuple (#54). Decrease tarantool-checks dependency from 3.1 to 2.1 (#124). Process a task on a writable space by default (#42). Wait until a space or an index is created (#68, #116). Tarantool Cartridge role (#107). Bugfixes Fix build and installation of rpm/deb packages (#124). Do not restart a work fiber if an index does not exist (#64). Update changelogs for rpm/deb packages. Testing Shuffle tests (#118). Fix test_mvcc_vinyl_tx_conflict (#104, #105). Fix flaky 'simple expires test' (#90). Other expirationd.start() parameter `space_id` has been renamed to `space` (#112). Add GitHub Actions workflow with debug Tarantool build (#102). Add GitHub Actions workflow for deploying module packages to S3 based repositories (#43).

Totktonada added the enhancement label Oct 23, 2020

Totktonada mentioned this issue Oct 23, 2020

Add case when tarantool is in read_only mode #37

Closed

Totktonada added backlog prio5 labels Dec 21, 2020

akudiyar mentioned this issue Apr 23, 2021

RFC: Adding Iterations over Secondary Indexes #50

Closed

Totktonada mentioned this issue Oct 7, 2021

Support LDoc #72

Merged

Totktonada removed the backlog label Oct 7, 2021

Totktonada removed the prio5 label May 31, 2022

Totktonada mentioned this issue May 31, 2022

Fix a description and behavior of the force option for expirationd.start #109

Merged

Totktonada added the 3sp label Jun 18, 2022

oleg-jukovec mentioned this issue Jul 13, 2022

api: process a task on a writable space by default #122

Merged

oleg-jukovec self-assigned this Jul 14, 2022

oleg-jukovec added a commit that referenced this issue Aug 3, 2022

packages: bump a required Tarantool version to 1.7.4

016782c

box.info.ro [1] and box.info.replication[n].lag [2] were introduced in Tarantool 1.7.4. 1. tarantool/tarantool@56462bc 2. tarantool/tarantool@7ed4848 Part of #42

oleg-jukovec added a commit that referenced this issue Aug 3, 2022

packages: bump a required Tarantool version to 1.7.4

6e262be

box.info.ro [1] and box.info.replication[n].lag [2] were introduced in Tarantool 1.7.4. 1. tarantool/tarantool@56462bc 2. tarantool/tarantool@7ed4848 Part of #42

oleg-jukovec closed this as completed in #122 Aug 3, 2022

oleg-jukovec mentioned this issue Aug 4, 2022

Release 1.3.0 #128

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Refine master-master mode support #42

Refine master-master mode support #42

Uh oh!

Uh oh!

Uh oh!

Refine master-master mode support #42

Refine master-master mode support #42

Comments

Uh oh!

Uh oh!

Uh oh!

Uh oh!