8000 EMR: fix EBS configuration by cbamelis · Pull Request #497 · cloudtools/troposphere · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

EMR: fix EBS configuration #497

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
May 28, 2016
Merged

EMR: fix EBS configuration #497

merged 2 commits into from
May 28, 2016

Conversation

cbamelis
Copy link
Contributor

When I tried to create an EMR cluster with EBS volumes, my CloudFormation template yesterday failed. After creating an AWS support ticket, it appeared the AWS documentation was wrong (EbsConfiguration cannot be specified on Cluster level, but needs to be specified in InstanceGroupConfigProperty). The documentation has meanwhile been fixed and can be found on http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-emr-cluster-jobflowinstancesconfig-instancegroupconfig.html

The code below reflects this update.

rcuza
rcuza reviewed May 20, 2016
View reviewed changes
examples/EMR_Cluster.py
@@ -90,7 +92,7 @@
BootstrapActions=[emr.BootstrapActionConfig(
Name='Dummy bootstrap action',
ScriptBootstrapAction=emr.ScriptBootstrapActionConfig(
Path='/bin/sh',
Path='file:/bin/sh',
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How do you know the file: change is required? It makes sense but I don't see it in the AWS documentation on http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-emr-cluster-bootstrapactionconfig-scriptbootstrapactionconfig.html. When you stood up this template, did you verify that this worked?

@rcuza
Copy link
Contributor
rcuza commented May 20, 2016

I like the changes to the example script, especially how you imported the troposphere.constants.

The changes in this PR move the EbcConfiguration into JobFlowInstancesConfig as per the AWS documentation. The PR updates the troposphere code, the example code, and the README.

I was not able to build an EMR cluster using the example template in my account in US-WEST-2.

09:50:40 UTC-0400   ROLLBACK_IN_PROGRESS    AWS::CloudFormation::Stack  foo-emr-test    The following resource(s) failed to create: [EMRSampleCluster]. . Rollback requested by user.
09:50:39 UTC-0400   CREATE_FAILED   AWS::EMR::Cluster   EMRSampleCluster    Encountered unsupported property EbsConfiguration

The example scripts should produce templates that work.

@ChristopheBamelis-TomTom
Fix EMR_Cluster example (cloudformation stack now succeeds, but EMR s…
9481626 
…tep fails because of obviously unexisting S3 bucket in my account)
@cbamelis
Copy link
Contributor Author

I pushed a new commit with slightly cleaned up EMR_Cluster example (none of the parameters were actually used in the script). I changed it to use the KeyPair and SubNet parameter and removed the unused security group parameters. The stack is started successfully on my account in eu-west-1. The hadoop step which was already present in the example fails (because it refers to an unexisting jar on S3, but I guess that's fine since the cloudformation stack is created successfully).

The CI build however failed on a curl download of python 3.5 (sounds unrelated to my changes). Do I need to push another (artificial) change in order to trigger a new CI build? Or is there another way to retry the CI build?

@rcuza
Copy link
Contributor
rcuza commented May 24, 2016

The example script stands up a successfully. It creates the following resources:

AWS::IAM::InstanceProfile       
WS::IAM::Role       
AWS::EMR::Cluster       
AWS::IAM::Role      
AWS::EMR::Step

LGTM 👍

@rcuza
Copy link
Contributor
rcuza commented May 25, 2016

@markpeek - What is the best way to trigger the tests to run again? Should @cbamelis check in a minor commit or is there some other way to run them? Thank you.

@markpeek markpeek merged commit 3bc4bc7 into cloudtools:master May 28, 2016
@markpeek
Copy link
Member

Triggered build, successful run, and now merged. Thanks for the change.

@rcuza thanks for the code review and input.

amosshapira pushed a commit to amosshapira/troposphere that referenced this pull request Oct 24, 2016
@cbamelis @covataamos
EMR: fix EBS configuration (cloudtools#497)
6979822 
* EMR: change EBS configuration (moved from Cluster to InstanceGroupConfigProperty)
* Fix EMR_Cluster example (cloudformation stack now succeeds, but EMR step fails because of obviously unexisting S3 bucket in my account)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@cbamelis @rcuza @markpeek @ChristopheBamelis-TomTom
0