8000 Maxwell crashes mostly on DDL statements. Can I get stop processing the DDL events ? · Issue #1457 · zendesk/maxwell · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Maxwell crashes mostly on DDL statements. Can I get stop processing the DDL events ? #1457

New issue

8000 Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
nithinreddy-mt opened this issue Mar 30, 2020 · 2 comments
Open

Maxwell crashes mostly on DDL statements. Can I get stop processing the DDL events ? #1457

nithinreddy-mt opened this issue Mar 30, 2020 · 2 comments

Comments

@nithinreddy-mt
Copy link

Can i stop processing DDL events and re-build the schemaStore object(composed inside BinlogConnectorReplicator) from scratch when ever i see a DDL event ?

I see that processing DDL events makes sense for pushing the schema change events to downstream consumer. I our case we do not output DDL events.

Please help me if this can cause some other issues.
@nithinreddy-mt
Copy link
Author

Sharing more context,
We noticed that maxwell crashes mostly due to the new DDL features which are given in mysql 8. Also we noticed it crashes on JSON columns. Which is again a Mysql 8 feature.

JSON issue -> #1290
DDL issue -> #1438

I see that above two issues are already fixed.

Another DDL issue we have seen,

ALTER DATABASE CHARACTER SET = utf8mb4 COLLATE = utf8mb4_unicode_ci;

maxwell crashes here by throwing error -> Couldn't find database ''

com.zendesk.maxwell.schema.ddl.InvalidSchemaError: Couldn't find database ''
	at com.zendesk.maxwell.schema.Schema.findDatabaseOrThrow(Schema.java:52)
	at com.zendesk.maxwell.schema.ddl.ResolvedDatabaseAlter.apply(ResolvedDatabaseAlter.java:21)
	at com.zendesk.maxwell.schema.AbstractSchemaStore.resolveSQL(AbstractSchemaStore.java:60)
	at com.zendesk.maxwell.schema.MysqlSchemaStore.processSQL(MysqlSchemaStore.java:104)
	at com.zendesk.maxwell.replication.BinlogConnectorReplicator.processQueryEvent(BinlogConnectorReplicator.java:243)
	at com.zendesk.maxwell.replication.BinlogConnectorReplicator.processQueryEvent(BinlogConnectorReplicator.java:261)
	at com.zendesk.maxwell.replication.BinlogConnectorReplicator.getRow(BinlogConnectorReplicator.java:496)
	at com.zendesk.maxwell.replication.BinlogConnectorReplicator.work(BinlogConnectorReplicator.java:150)
	at com.zendesk.maxwell.util.RunLoopProcess.runLoop(RunLoopProcess.java:27)
	at com.zendesk.maxwell.Maxwell.startInner(Maxwell.java:224)
	at com.zendesk.maxwell.Maxwell.start(Maxwell.java:156)
	at com.zendesk.maxwell.Maxwell.main(Maxwell.java:245)

Where as it works fine when we run

ALTER DATABASE some_db_name CHARACTER SET = utf8mb4 COLLATE = utf8mb4_unicode_ci;

@osheroff
Copy link
Collaborator

I've always resisted the urge to recapture the schema on error. it's prone to race conditions and also means that generally people won't report bugs in the DDL parser, and the DDL parser is used for much much more than outputting DDL statements (maxwell needs to maintain a copy of the database schema in order to output rows).

I don't know. I always waffle on this one.

I'll address the parser bug in a separate PR.

osheroff added a commit that referenced this issue Mar 31, 2020
@osheroff
handle implicit ALTER DATABASE name, fixes issue #1457
c8e7f17
osheroff added a commit that referenced this issue Apr 4, 2020
@osheroff
Merge pull request #1458 from zendesk/ben/alter_database_implicit_name
42e0aec 
handle implicit ALTER DATABASE name, fixes issue #1457
osheroff added a commit that referenced this issue Apr 22, 2020
@osheroff
v1.25.1, "nowhere to put it"
d703111 
- issue #1457, ALTER DATABASE with implicit database name
- maxwell now runs on JDK 11 in docker
- exit with status 2 when we can't find binlog files
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@osheroff @nithinreddy-mt
0