Add ability to install packages + xgboost model deployer + example #431
Conversation
…r deploy_xgboost_model
|
Can one of the admins verify this patch? |
There was a problem hiding this comment.
- Revert the deletion of the
yapfsubmodule.
This mostly looks good. Can you remove the ipython notebook example and turn it into a tutorial on the Clipper website? Once you've rewritten it in pure markdown, file a PR against the Clipper website repo.
| @@ -368,7 +385,7 @@ def build_model(self, | |||
| context_tar.add(model_data_path) | |||
| # From https://stackoverflow.com/a/740854/814642 | |||
| df_contents = six.StringIO( | |||
| "FROM {container_name}\nCOPY {data_path} /model/\n".format( | |||
| ("FROM {container_name}\nCOPY {data_path} /model/\n"+run_cmd).format( | |||
There was a problem hiding this comment.
Use string formatting instead of appending:
"FROM {container_name}\nCOPY {data_path} /model/\n{run_cmd}".format(container_name=base_image, data_path=model_data_path, run_cmd=run_cmd)
| @@ -237,7 +237,8 @@ def build_and_deploy_model(self, | |||
| labels=None, | |||
| container_registry=None, | |||
| num_replicas=1, | |||
| batch_size=-1): | |||
| batch_size=-1, | |||
| pkgs_to_install=None): | |||
There was a problem hiding this comment.
Add this pkgs_to_install command arg to the all of the model deployers in clipper_admin/deployers/.
|
|
||
| base_image = 'clipper/python-closure-container:develop' | ||
| clipper_conn.build_and_deploy_model(model_name, version, "integers", | ||
| serialization_dir, base_image, pkgs_to_install=['xgboost']) |
There was a problem hiding this comment.
You shouldn't save the function and build the model directly. Use the Python closure deployer to do this for you. That's exactly what it is meant for.
| @@ -360,6 +368,15 @@ def build_model(self, | |||
|
|
|||
| _validate_versioned_model_name(name, version) | |||
|
|
|||
| run_cmd = '' | |||
| if pkgs_to_install: | |||
| run_cmd = 'RUN apt-get -y install build-essential && pip install ' | |||
There was a problem hiding this comment.
I would argue that the following is cleaner:
base_cmd = 'RUN apt-get -y install build-essential && pip install'.split(' ')
run_cmd = ' '.join(base_cmd + pkgs_to_install)
You can then add a newline to the end of the run_cmd argument in the df_contents string definition. If run_cmd is undefined, then this just safely adds an additional newline to the end of the dockerfile.
| clipper_conn = create_docker_connection( | ||
| cleanup=True, start_clipper=True) | ||
|
|
||
| train_path = os.path.join(cur_dir, "data/agaricus.txt.train") |
There was a problem hiding this comment.
Include these training / test datasets in your PR within the data directory. These are currently undefined.
There was a problem hiding this comment.
This is actually old code, and I forgot to delete this line. We don't actually use this to train/test.
There was a problem hiding this comment.
This looks like it's almost ready to go. However, I couldn't run the integration test for XGBoost. The XGBoost container crashes with the following stacktrace. Please let me know if you have trouble reproducing this behavior.
Starting Python Closure container
Connecting to Clipper with default port: 7000
Initializing Python function container
Traceback (most recent call last):
File "/container/python_closure_container.py", line 90, in <module>
model = PythonContainer(model_path, input_type)
File "/container/python_closure_container.py", line 24, in __init__
self.predict_func = load_predict_func(predict_path)
File "/container/python_closure_container.py", line 13, in load_predict_func
return cloudpickle.load(serialized_func_file)
File "/opt/conda/lib/python2.7/pickle.py", line 1384, in load
return Unpickler(file).load()
File "/opt/conda/lib/python2.7/pickle.py", line 864, in load
dispatch[key](self)
File "/opt/conda/lib/python2.7/pickle.py", line 1139, in load_reduce
value = func(*args)
File "/opt/conda/lib/python2.7/site-packages/cloudpickle/cloudpickle.py", line 1048, in _make_skel_func
if cell_count >= 0 else
TypeError: range() integer end argument expected, got list.
Encountered error not related to missing packages. Please refer to the container log to diagnose.```
| "FROM {container_name}\nCOPY {data_path} /model/\n".format( | ||
| container_name=base_image, data_path=model_data_path)) | ||
| "FROM {container_name}\nCOPY {data_path} /model/\n{run_command}".format( | ||
| container_name=base_image, data_path=model_data_path, run_command=(run_cmd + '\n'))) |
There was a problem hiding this comment.
Nit: Can you move the newline to the end of the expression being formatted on line 383?
|
@Corey-Zumar 18-03-16:15:04:17 INFO [test_utils.py:64] Creating DockerContainerManager
18-03-16:15:04:18 INFO [clipper_admin.py:1201] Stopped all Clipper cluster and all model containers
18-03-16:15:04:18 INFO [test_utils.py:80] Starting Clipper
18-03-16:15:04:18 INFO [docker_container_manager.py:106] Starting managed Redis instance in Docker
18-03-16:15:04:21 INFO [clipper_admin.py:111] Clipper is running
18-03-16:15:04:22 INFO [clipper_admin.py:186] Application xgboost-test was successfully registered
[0] train-error:0
[1] train-error:0
18-03-16:15:04:23 INFO [deployer_utils.py:49] Saving function to /tmp/clipper/tmpFDSGb5
18-03-16:15:04:28 INFO [deployer_utils.py:58] Anaconda environment found. Verifying packages.
18-03-16:15:04:35 INFO [deployer_utils.py:158] The following packages in your conda environment are not available in the linux-64 conda channel the container will use:
ca-certificates==2018.1.18=0, clangdev==5.0.0=default_0, llvmdev==5.0.0=default_0, openmp==5.0.0=0, openssl== ...
We will skip their installation when deploying your function. If your function uses these packages, the container will experience a runtime error when queried.
18-03-16:15:04:35 INFO [deployer_utils.py:159] Using Anaconda API: https://api.anaconda.org
18-03-16:15:04:35 INFO [deployer_utils.py:67] Supplied environment details
18-03-16:15:04:43 INFO [deployer_utils.py:79] Supplied local modules
18-03-16:15:04:43 INFO [deployer_utils.py:85] Serialized and supplied predict function
18-03-16:15:04:43 INFO [python.py:192] Python closure saved
18-03-16:15:04:44 INFO [clipper_admin.py:400] Building model Docker image with model data from /tmp/clipper/tmpFDSGb5
18-03-16:15:05:57 INFO [clipper_admin.py:404] Pushing model Docker image to xgboost-model:1
18-03-16:15:05:58 INFO [docker_container_manager.py:243] Found 0 replicas for xgboost-model:1. Adding 1
18-03-16:15:05:59 INFO [clipper_admin.py:578] Successfully registered model xgboost-model:1
18-03-16:15:05:59 INFO [clipper_admin.py:496] Done deploying model xgboost-model:1.
18-03-16:15:06:04 INFO [clipper_admin.py:229] Model xgboost-model is now linked to application xgboost-test
18-03-16:15:06:35 INFO [test_utils.py:64] Creating DockerContainerManager
18-03-16:15:07:08 INFO [clipper_admin.py:1201] Stopped all Clipper cluster and all model containers
18-03-16:15:07:08 INFO [clipper_admin.py:123] Successfully connected to Clipper cluster at localhost:38681 |
| import time | ||
| import logging | ||
| import xgboost as xgb | ||
| import pickle |
|
LGTM. Nice work! |
Corey Reviewed and said we should get this merged.
|
jenkins test this please |
|
Test PASSed. |
I've added the ability to deploy containers and install additional packages using pip, as well as an integration test for the XGBoost model, and an example walkthrough (using an iPython Notebook) of building and deploying Clipper, and then using it with XGBoost.
closes #341