Use proper API call in bigtable.py's make_dataset #2618
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@ulzha Could you take a look here - as I understand Spotify uses BigQuery? The motivation and reasoning makes sense to me. |
|
I can confirm that it works, and I just did exactly the same fix. |
varjoranta
approved these changes
Jan 10, 2019
|
Thanks! I don't have write access -- could someone merge it for me? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
There used to be two ways to make a dataset in the Google BigQuery API: the correct new way and the obsolete old way. Of course our code used the obsolete old way. Google seems to have taken the obsolete old way down. This PR is to do it the correct new way instead.
Motivation and Context
The BigQuery make_dataset API has two fields, "datasetId" and "id", which could be used to specify a dataset name. The documentation for "id" says to use "datasetId" when creating a dataset. Our make_dataset method of course uses "id". I conjecture that Google used to use "id" if "datasetId" were missing, but changed their impl to match the docco recently.
The make_dataset method in bigquery.py of course uses the "id" approach rather than the "datasetId", so the conjectured change would have broken it.
https://jira.spotify.net/browse/INCIDENT-8998
Have you tested this? If so, how?
I confirmed this by cloning the relevant code and trying to create a dataset with it. The old way fails with the same characteristic message that our users have been seeing. The new way works.