GitHub

Some basic PoC about the ML infra pipelines and related settings

PoC1

1. The pipelines contains AWS glue, AWS lambda, AWS S3 and they are integrated by AWS step functions.

2. Please note the suitable IAM roles/users need to be setup for various groups or purposes.

3. For the scalable and expandable services, components configuration need to be set correctly like S3 folder names to separate different trigger/policies.

4. State machine work flow is defined in step function file.

PoC2

1. The only differnce between this one and PoC1 is the feature store loading step.

2. Please note that the feature store loading step will require two layers as shown in the code, otherwise the workflow will fail.

PoC3

1. This is the high level design to handle the streaming data and store the features for both stream data features and batch data aggregation features.

2. This is only for PoC, some details like event notifications, IAM, error handling and related configurations are not included in the code.

PoC4

1. This is a simplified process for the infra change and the model/api changes workflow.

2. Please note in the production there are much more elements need to be considered like different environments (DEVX, UATX, PRDX), also various tags, dependencies, input parameters, KMS...

3. Docker image build process can be integrated to the Jenkins or Circle CI instead of the AWS codecommit in this example.

4. Model size and performance related metrics need to be properly considered.

Name		Name	Last commit message	Last commit date
Latest commit History 19 Commits
PoC1		PoC1
PoC2		PoC2
PoC3		PoC3
PoC4		PoC4
Transformer_Encoder_From_Draft		Transformer_Encoder_From_Draft
README.md		README.md

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Repository files navigation

Some basic PoC about the ML infra pipelines and related settings

PoC1

1. The pipelines contains AWS glue, AWS lambda, AWS S3 and they are integrated by AWS step functions.

2. Please note the suitable IAM roles/users need to be setup for various groups or purposes.

3. For the scalable and expandable services, components configuration need to be set correctly like S3 folder names to separate different trigger/policies.

4. State machine work flow is defined in step function file.

PoC2

1. The only differnce between this one and PoC1 is the feature store loading step.

2. Please note that the feature store loading step will require two layers as shown in the code, otherwise the workflow will fail.

PoC3

1. This is the high level design to handle the streaming data and store the features for both stream data features and batch data aggregation features.

2. This is only for PoC, some details like event notifications, IAM, error handling and related configurations are not included in the code.

PoC4

1. This is a simplified process for the infra change and the model/api changes workflow.

2. Please note in the production there are much more elements need to be considered like different environments (DEVX, UATX, PRDX), also various tags, dependencies, input parameters, KMS...

3. Docker image build process can be integrated to the Jenkins or Circle CI instead of the AWS codecommit in this example.

4. Model size and performance related metrics need to be properly considered.

About

Uh oh!

Releases

Packages

Uh oh!

Languages

TianqGuo/ML_Infra_POC

Folders and files

Latest commit

History

Repository files navigation

Some basic PoC about the ML infra pipelines and related settings

PoC1

1. The pipelines contains AWS glue, AWS lambda, AWS S3 and they are integrated by AWS step functions.

2. Please note the suitable IAM roles/users need to be setup for various groups or purposes.

3. For the scalable and expandable services, components configuration need to be set correctly like S3 folder names to separate different trigger/policies.

4. State machine work flow is defined in step function file.

PoC2

1. The only differnce between this one and PoC1 is the feature store loading step.

2. Please note that the feature store loading step will require two layers as shown in the code, otherwise the workflow will fail.

PoC3

1. This is the high level design to handle the streaming data and store the features for both stream data features and batch data aggregation features.

2. This is only for PoC, some details like event notifications, IAM, error handling and related configurations are not included in the code.

PoC4

1. This is a simplified process for the infra change and the model/api changes workflow.

2. Please note in the production there are much more elements need to be considered like different environments (DEVX, UATX, PRDX), also various tags, dependencies, input parameters, KMS...

3. Docker image build process can be integrated to the Jenkins or Circle CI instead of the AWS codecommit in this example.

4. Model size and performance related metrics need to be properly considered.

About

Resources

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Languages

Packages