Adding support for EMR resources #421
Conversation
|
|
||
| class Application(AWSProperty): | ||
| props = { | ||
| 'AdditionalInfo': (dict, False), |
There was a problem hiding this comment.
This doesn't appear to be a dict in the Cloudformation docs - it might call for it's own validator.
There was a problem hiding this comment.
AdditionalInfo seems to be documented as a JSON Object, that's why I chose the dict to represent it. Is there a better way?
There was a problem hiding this comment.
That's in the Cluster resource, not the Application Property: http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-emr-cluster-application.html#cfn-emr-cluster-application-additionalinfo
There was a problem hiding this comment.
oh yes! sorry, it's using another colon-separated key-value pair list... ¬¬
|
Right, so I added more strict validations and enriched the test template with more details plus the sample step from AWS docs. There seems to be some incongruences in the AWS docs anyway. One of them is regarding that self-recursive Another incongruence can be seen in the configuration of the step. The Will try to test the last version of the example template once I get a chance in our cluster to see what is the correct encoding. |
|
Cool - thanks for diving into this @alonsodomin - I'll try to get a chance to look at it today! |
|
Sorry for the delay on this, getting a slot to experiment cluster configuration in the cluster usually doesn't sit at the top of the priority list ;) Anyway, it seems that the correct encoding for the cluster & instance group configuration, using Same for the |
|
btw, also took the chance to rebase my fork on top of master. |
| class Application(AWSProperty): | ||
| props = { | ||
| 'AdditionalInfo': (additional_info_validator, False), | ||
| 'Args': (list, False), |
There was a problem hiding this comment.
I think this should be [basestring]
# Conflicts: # examples/EMR_Cluster.py # tests/examples_output/EMR_Cluster.template # troposphere/emr.py
|
Long time since last update. I managed to test the incongruence in the AWS docs regarding the Let me know your thoughts about the other comments (i.e.: the validator for the self-recursive |
|
Great - thanks @alonsodomin - I went ahead and used the new 'squash & merge' feature of github to merge this. |
Created new module to hold all the new EMR resources and types plus a basic example that creates an EMR cluster will all the applications (Hadoop, Spark, etc.) installed in it.