Open
Description
Configuration
- Version of DeepDetect:
- Locally compiled on:
- Ubuntu 18.04 LTS
- Other:
- Docker CPU
- Docker GPU tensorrt
- Docker tensorrt
- Amazon AMI
- Locally compiled on:
- Commit (shown by the server when starting):
I've tested the last two images and it fails for boths
74cf54c
4de3807
Your question / the problem you're facing:
I tried to make a prediction using a refinedet model with tensorrt backend. However I get an error and if I try to make a prediction two times it generated a core dump.
Error message (if any) / steps to reproduce the problem:
- First, let us get one refinedet model (faces)
BASE_PATH="PATH_WANTED"
PATH_MODEL="${BASE_PATH}/face"
mkdir $PATH_MODEL && cd $PATH_MODEL
wget https://deepdetect.com/models/init/desktop/images/detection/faces_512.tar.gz
wget -O test_image.jpeg https://miro.medium.com/max/6528/1*DYUaxku5bfbZaDLW-4SyWg.jpeg
tar -xvf faces_512.tar.gz
- Pull the last version (v0.15.0) of dd with tensorrt
docker pull jolibrain/deepdetect_gpu_tensorrt:v0.15.0
- Launch DD's server
docker run --gpus 1 -d -u $(id -u ${USER}):$(id -g ${USER}) -p 8080:8080 -v $BASE_PATH:/opt/models/ jolibrain/deepdetect_gpu_tensorrt:v0.15.0
- Launch the service
curl -X PUT "http://localhost:8080/services/imageserv" -d '{
"mllib":"tensorrt",
"description":"image classification service",
"type":"supervised",
"parameters":{
"input":{
"connector":"image",
"width":512,
"height":512,
"mean": [104.146, 110.808, 119.856]
},
"mllib": {
"datatype": "fp32",
"nclasses": 1,
"maxBatchSize": 1,
"maxWorkspaceSize": 8000,
"gpuid":0
}
},
"model":{
"repository":"/opt/models/face"
}
}'
Output
[2021-04-13 07:28:54.342] [imageserv] [info] Registered plugin creator - ::SpecialSlice_TRT version 1
[2021-04-13 07:28:54.342] [imageserv] [info] setting max workspace size to 8388608000
[2021-04-13 07:28:54.342] [imageserv] [info] setting max batch size to 1
[2021-04-13 07:28:54.724] [imageserv] [info] trying to determine the input size...
[2021-04-13 07:28:54.739] [imageserv] [info] found 512x512 as input size
[2021-04-13 07:28:54.739] [api] [info] 172.17.0.1 "PUT /services/imageserv" 201 399ms
- Launch prediction
curl -X POST "http://localhost:8080/predict" -d '{
"service":"imageserv",
"parameters":{
"input":{
"width":512,
"height":512
},
"output":{
"confidence_threshold": 0.5,
"bbox": true
}
},
"data": ["https://icour.fr/ELeveSeconde/ajout/yann_lecum_vidal/images/yann_LeCun.jpg"]
}'
Output using v 0.14.0
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(PluginV2): conv5_3_norm_mbox_priorbox, Tactic: 0, conv5_3_norm[Float(512,32,32)], data[Float(3,512,512)] -> conv5_3_norm_mbox_priorbox[Float(2,12288,1)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(FusedConvActDirect): fc7_mbox_loc_ftune || fc7_mbox_conf_ftune, Tactic: 162, fc7[Float(1024,16,16)] -> fc7_mbox_loc_ftune || fc7_mbox_conf_ftune[Float(18,16,16)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(Shuffle): fc7_mbox_loc_perm_ftune + fc7_mbox_loc_flat_ftune, Tactic: 0, fc7_mbox_loc_ftune || fc7_mbox_conf_ftune[Float(12,16,16)] -> arm_loc_ftune[Float(3072,1,1)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(Shuffle): fc7_mbox_conf_perm_ftune + fc7_mbox_conf_flat_ftune, Tactic: 0, fc7_mbox_loc_ftune || fc7_mbox_conf_ftune[Float(6,16,16)] -> arm_conf_ftune[Float(1536,1,1)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(PluginV2): fc7_mbox_priorbox, Tactic: 0, fc7[Float(1024,16,16)], data[Float(3,512,512)] -> fc7_mbox_priorbox[Float(2,3072,1)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(FusedConvActDirect): conv6_2_mbox_loc_ftune || conv6_2_mbox_conf_ftune, Tactic: 39, conv6_2[Float(512,8,8)] -> conv6_2_mbox_loc_ftune || conv6_2_mbox_conf_ftune[Float(18,8,8)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(Shuffle): conv6_2_mbox_loc_perm_ftune + conv6_2_mbox_loc_flat_ftune, Tactic: 0, conv6_2_mbox_loc_ftune || conv6_2_mbox_conf_ftune[Float(12,8,8)] -> arm_loc_ftune[Float(768,1,1)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(Shuffle): conv6_2_mbox_conf_perm_ftune + conv6_2_mbox_conf_flat_ftune, Tactic: 0, conv6_2_mbox_loc_ftune || conv6_2_mbox_conf_ftune[Float(6,8,8)] -> arm_conf_ftune[Float(384,1,1)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(PluginV2): conv6_2_mbox_priorbox, Tactic: 0, conv6_2[Float(512,8,8)], data[Float(3,512,512)] -> conv6_2_mbox_priorbox[Float(2,768,1)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(Reformat): conv4_3_norm_mbox_priorbox copy, Tactic: 0, conv4_3_norm_mbox_priorbox[Float(2,49152,1)] -> arm_priorbox[Float(2,49152,1)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(Reformat): conv5_3_norm_mbox_priorbox copy, Tactic: 0, conv5_3_norm_mbox_priorbox[Float(2,12288,1)] -> arm_priorbox[Float(2,12288,1)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(Reformat): fc7_mbox_priorbox copy, Tactic: 0, fc7_mbox_priorbox[Float(2,3072,1)] -> arm_priorbox[Float(2,3072,1)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(Reformat): conv6_2_mbox_priorbox copy, Tactic: 0, conv6_2_mbox_priorbox[Float(2,768,1)] -> arm_priorbox[Float(2,768,1)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(scudnn_winograd): P3_mbox_loc_ftune || P3_mbox_conf_ftune, Tactic: 3827454225649558724, P3[Float(256,64,64)] -> P3_mbox_loc_ftune || P3_mbox_conf_ftune[Float(18,64,64)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(Shuffle): P3_mbox_loc_perm_ftune + P3_mbox_loc_flat_ftune, Tactic: 0, P3_mbox_loc_ftune || P3_mbox_conf_ftune[Float(12,64,64)] -> P3_mbox_loc_flat_ftune[Float(49152,1,1)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(Shuffle): P3_mbox_conf_perm_ftune + P3_mbox_conf_flat_ftune, Tactic: 0, P3_mbox_loc_ftune || P3_mbox_conf_ftune[Float(6,64,64)] -> P3_mbox_conf_flat_ftune[Float(24576,1,1)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(scudnn_winograd): P4_mbox_loc_ftune || P4_mbox_conf_ftune, Tactic: 3827454225649558724, P4[Float(256,32,32)] -> P4_mbox_loc_ftune || P4_mbox_conf_ftune[Float(18,32,32)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(Shuffle): P4_mbox_loc_perm_ftune + P4_mbox_loc_flat_ftune, Tactic: 0, P4_mbox_loc_ftune || P4_mbox_conf_ftune[Float(12,32,32)] -> P4_mbox_loc_flat_ftune[Float(12288,1,1)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(Shuffle): P4_mbox_conf_perm_ftune + P4_mbox_conf_flat_ftune, Tactic: 0, P4_mbox_loc_ftune || P4_mbox_conf_ftune[Float(6,32,32)] -> P4_mbox_conf_flat_ftune[Float(6144,1,1)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(FusedConvActDirect): P5_mbox_loc_ftune || P5_mbox_conf_ftune, Tactic: 162, P5[Float(256,16,16)] -> P5_mbox_loc_ftune || P5_mbox_conf_ftune[Float(18,16,16)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(Shuffle): P5_mbox_loc_perm_ftune + P5_mbox_loc_flat_ftune, Tactic: 0, P5_mbox_loc_ftune || P5_mbox_conf_ftune[Float(12,16,16)] -> P5_mbox_loc_flat_ftune[Float(3072,1,1)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(Shuffle): P5_mbox_conf_perm_ftune + P5_mbox_conf_flat_ftune, Tactic: 0, P5_mbox_loc_ftune || P5_mbox_conf_ftune[Float(6,16,16)] -> P5_mbox_conf_flat_ftune[Float(1536,1,1)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(FusedConvActDirect): P6_mbox_loc_ftune || P6_mbox_conf_ftune_ftune, Tactic: 39, P6[Float(256,8,8)] -> P6_mbox_loc_ftune || P6_mbox_conf_ftune_ftune[Float(18,8,8)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(Shuffle): P6_mbox_loc_perm_ftune + P6_mbox_loc_flat_ftune, Tactic: 0, P6_mbox_loc_ftune || P6_mbox_conf_ftune_ftune[Float(12,8,8)] -> P6_mbox_loc_flat_ftune[Float(768,1,1)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(Shuffle): P6_mbox_conf_perm_ftune + P6_mbox_conf_flat_ftune, Tactic: 0, P6_mbox_loc_ftune || P6_mbox_conf_ftune_ftune[Float(6,8,8)] -> P6_mbox_conf_flat_ftune[Float(384,1,1)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(Reformat): P3_mbox_loc_flat_ftune copy, Tactic: 0, P3_mbox_loc_flat_ftune[Float(49152,1,1)] -> odm_loc_ftune[Float(49152,1,1)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(Reformat): P4_mbox_loc_flat_ftune copy, Tactic: 0, P4_mbox_loc_flat_ftune[Float(12288,1,1)] -> odm_loc_ftune[Float(12288,1,1)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(Reformat): P5_mbox_loc_flat_ftune copy, Tactic: 0, P5_mbox_loc_flat_ftune[Float(3072,1,1)] -> odm_loc_ftune[Float(3072,1,1)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(Reformat): P6_mbox_loc_flat_ftune copy, Tactic: 0, P6_mbox_loc_flat_ftune[Float(768,1,1)] -> odm_loc_ftune[Float(768,1,1)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(Reformat): P3_mbox_conf_flat_ftune copy, Tactic: 0, P3_mbox_conf_flat_ftune[Float(24576,1,1)] -> odm_conf_ftune[Float(24576,1,1)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(Reformat): P4_mbox_conf_flat_ftune copy, Tactic: 0, P4_mbox_conf_flat_ftune[Float(6144,1,1)] -> odm_conf_ftune[Float(6144,1,1)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(Reformat): P5_mbox_conf_flat_ftune copy, Tactic: 0, P5_mbox_conf_flat_ftune[Float(1536,1,1)] -> odm_conf_ftune[Float(1536,1,1)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(Reformat): P6_mbox_conf_flat_ftune copy, Tactic: 0, P6_mbox_conf_flat_ftune[Float(384,1,1)] -> odm_conf_ftune[Float(384,1,1)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(ExtSoftMax): arm_conf_softmax, Tactic: 0, arm_
B6CB
conf_reshape_ftune[Float(16320,2)] -> arm_conf_softmax_ftune[Float(16320,2)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(ExtSoftMax): odm_conf_softmax, Tactic: 0, odm_conf_reshape_ftune[Float(16320,2)] -> odm_conf_softmax_ftune[Float(16320,2)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(PluginV2): detection_out, Tactic: 0, odm_loc_ftune[Float(65280,1,1)], odm_conf_flatten_ftune[Float(32640,1,1)], arm_priorbox[Float(2,65280,1)], arm_conf_flatten_ftune[Float(32640,1,1)], arm_loc_ftune[Float(65280,1,1)] -> detection_out[Float(1,500,7)], keep_count[Float(1,1,1)]
[2021-04-13 07:40:28.936] [imageserv] [warning] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles
[2021-04-13 07:40:29.008] [imageserv] [error] ../rtSafe/cuda/caskConvolutionRunner.cpp (370) - Cuda Error in execute: 98 (invalid device function)
[2021-04-13 07:40:29.008] [imageserv] [error] FAILED_EXECUTION: std::exception
[2021-04-13 07:40:29.008] [imageserv] [error] Error while processing forward TRT pass, not enough memory ? Failed TRT enqueue call
[2021-04-13 07:40:29.008] [imageserv] [error] mllib internal error: Failed TRT enqueue call
[2021-04-13 07:40:29.008] [api] [error] 172.17.0.1 "POST /predict" 500 150564ms
Output using v 0.14.0, If the prediction is launched a second time we get the information
[2021-04-13 07:40:29.008] [api] [error] 172.17.0.1 "POST /predict" 500 150564ms
#assertion/opt/deepdetect/build/tensorrt-oss/src/tensorrt-oss/plugin/normalizePlugin/normalizePlugin.cpp,112
0# dd::HttpJsonAPI::abort(int) at /opt/deepdetect/src/httpjsonapi.cc:626
1# 0x00007F6259BB8F20 in /lib/x86_64-linux-gnu/libc.so.6
2# 0x00007F6259BB8E97 in /lib/x86_64-linux-gnu/libc.so.6
3# 0x00007F6259BBA801 in /lib/x86_64-linux-gnu/libc.so.6
4# 0x00007F625E0A7348 in /opt/deepdetect/build/lib/libnvinfer_plugin.so.7
5# 0x00007F625F9AD555 in /usr/lib/x86_64-linux-gnu/libnvinfer.so.7
6# 0x00007F625F726327 in /usr/lib/x86_64-linux-gnu/libnvinfer.so.7
7# 0x00007F625F7293E0 in /usr/lib/x86_64-linux-gnu/libnvinfer.so.7
8# dd::TensorRTLib<dd::ImgTensorRTInputFileConn, dd::SupervisedOutput, dd::TensorRTModel>::predict(dd::APIData const&, dd::APIData&) at /opt/deepdetect/src/backends/tensorrt/tensorrtlib.cc:678
9# mapbox::util::detail::dispatcher<dd::visitor_mllib::v_predict_job&, mapbox::util::variant<dd::MLService<dd::TensorRTLib, dd::ImgTensorRTInputFileConn, dd::SupervisedOutput, dd::TensorRTModel> >, int, dd::MLService<dd::TensorRTLib, dd::ImgTensorRTInputFileConn, dd::SupervisedOutput, dd::TensorRTModel> >::apply(mapbox::util::variant<dd::MLService<dd::TensorRTLib, dd::ImgTensorRTInputFileConn, dd::SupervisedOutput, dd::TensorRTModel> >&, dd::visitor_mllib::v_predict_job&) at /usr/include/mapbox/variant.hpp:344
10# dd::Services::predict(dd::APIData const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, dd::APIData&, bool const&) at /opt/deepdetect/src/services.h:623
11# dd::JsonAPI::service_predict(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) at /opt/deepdetect/src/jsonapi.cc:1161
12# APIHandler::operator()(boost::network::http::basic_request<boost::network::http::tags::http_server> const&, boost::network::http::basic_response<boost::network::http::tags::http_server>&) at /opt/deepdetect/src/httpjsonapi.cc:465
13# boost::network::http::sync_connection<boost::network::http::tags::http_server, APIHandler>::handle_read_headers(boost::system::error_code const&, unsigned long) at /opt/deepdetect/build/cpp-netlib/src/cpp-netlib/boost/network/protocol/http/server/sync_connection.hpp:139
14# void boost::asio::detail::strand_service::dispatch<boost::asio::detail::binder2<boost::_bi::bind_t<void, boost::_mfi::mf2<void, boost::network::http::sync_connection<boost::network::http::tags::http_server, APIHandler>, boost::system::error_code const&, unsigned long>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<boost::network::http::sync_connection<boost::network::http::tags::http_server, APIHandler> > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::system::error_code, unsigned long> >(boost::asio::detail::strand_service::strand_impl*&, boost::asio::detail::binder2<boost::_bi::bind_t<void, boost::_mfi::mf2<void, boost::network::http::sync_connection<boost::network::http::tags::http_server, APIHandler>, boost::system::error_code const&, unsigned long>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<boost::network::http::sync_connection<boost::network::http::tags::http_server, APIHandler> > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::system::error_code, unsigned long>&) at /usr/include/boost/asio/detail/impl/strand_service.hpp:91
15# boost::asio::detail::completion_handler<boost::asio::detail::rewrapped_handler<boost::asio::detail::binder2<boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, boost::_bi::bind_t<void, boost::_mfi::mf2<void, boost::network::http::sync_connection<boost::network::http::tags::http_server, APIHandler>, boost::system::error_code const&, unsigned long>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<boost::network::http::sync_connection<boost::network::http::tags::http_server, APIHandler> > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::asio::detail::is_continuation_if_running>, boost::system::error_code, unsigned long>, boost::_bi::bind_t<void, boost::_mfi::mf2<void, boost::network::http::sync_connection<boost::network::http::tags::http_server, APIHandler>, boost::system::error_code const&, unsigned long>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<boost::network::http::sync_connection<boost::network::http::tags::http_server, APIHandler> > >, boost::arg<1> (*)(), boost::arg<2> (*)()> > > >::do_complete(boost::asio::detail::task_io_service*, boost::asio::detail::task_io_service_operation*, boost::system::error_code const&, unsigned long) at /usr/include/boost/asio/detail/completion_handler.hpp:68
16# void boost::asio::detail::strand_service::dispatch<boost::asio::detail::rewrapped_handler<boost::asio::detail::binder2<boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, boost::_bi::bind_t<void, boost::_mfi::mf2<void, boost::network::http::sync_connection<boost::network::http::tags::http_server, APIHandler>, boost::system::error_code const&, unsigned long>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<boost::network::http::sync_connection<boost::network::http::tags::http_server, APIHandler> > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::asio::detail::is_continuation_if_running>, boost::system::error_code, unsigned long>, boost::_bi::bind_t<void, boost::_mfi::mf2<void, boost::network::http::sync_connection<boost::network::http::tags::http_server, APIHandler>, boost::system::error_code const&, unsigned long>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<boost::network::http::sync_connection<boost::network::http::tags::http_server, APIHandler> > >, boost::arg<1> (*)(), boost::arg<2> (*)()> > > >(boost::asio::detail::strand_service::strand_impl*&, boost::asio::detail::rewrapped_handler<boost::asio::detail::binder2<boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, boost::_bi::bind_t<void, boost::_mfi::mf2<void, boost::network::http::sync_connection<boost::network::http::tags::http_server, APIHandler>, boost::system::error_code const&, unsigned long>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<boost::network::http::sync_connection<boost::network::http::tags::http_server, APIHandler> > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::asio::detail::is_continuation_if_running>, boost::system::error_code, unsigned long>, boost::_bi::bind_t<void, boost::_mfi::mf2<void, boost::network::http::sync_connection<boost::network::http::tags::http_server, APIHandler>, boost::system::error_code const&, unsigned long>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<boost::network::http::sync_connection<boost::network::http::tags::http_server, APIHandler> > >, boost::arg<1> (*)(), boost::arg<2> (*)()> > >&) at /usr/include/boost/asio/detail/impl/strand_service.hpp:85
17# boost::asio::detail::reactive_socket_recv_op<boost::asio::mutable_buffers_1, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, boost::_bi::bind_t<void, boost::_mfi::mf2<void, boost::network::http::sync_connection<boost::network::http::tags::http_server, APIHandler>, boost::system::error_code const&, unsigned long>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<boost::network::http::sync_connection<boost::network::http::tags::http_server, APIHandler> > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::asio::detail::is_continuation_if_running> >::do_complete(boost::asio::detail::task_io_service*, boost::asio::detail::task_io_service_operation*, boost::system::error_code
const&, unsigned long) at /usr/include/boost/asio/detail/reactive_socket_recv_op.hpp:110
18# boost::asio::detail::epoll_reactor::descriptor_state::do_complete(boost::asio::detail::task_io_service*, boost::asio::detail::task_io_service_operation*, boost::system::error_code const&, unsigned long) at /usr/include/boost/asio/detail/impl/epoll_reactor.ipp:707
19# boost::network::http::sync_server_base<boost::network::http::tags::http_server, APIHandler>::run() at /opt/deepdetect/build/cpp-netlib/src/cpp-netlib/boost/network/protocol/http/server/sync_server.hpp:49
20# 0x00007F625A80D66F in /usr/lib/x86_64-linux-gnu/libstdc++.so.6
21# 0x00007F625D9EA6DB in /lib/x86_64-linux-gnu/libpthread.so.0
22# 0x00007F6259C9B88F in /lib/x86_64-linux-gnu/libc.so.6
Aborted (core dumped)
It shows message saying "not enough memory" however I have a 1080Ti and enough gpu memory.
Output v0.15.0
etection_out reformatted input 4[Float(65280,1,1)] -> detection_out[Float(1,500,7)], keep_count[Float(1,1,1)]
[2021-04-13 07:37:01.040] [imageserv] [info] Allocated persistent device memory of size 289784320
[2021-04-13 07:37:01.040] [imageserv] [info] Allocated activation device memory of size 146426880
[2021-04-13 07:37:01.040] [imageserv] [info] Assigning persistent memory blocks for various profiles
#assertion/opt/deepdetect/build/tensorrt-oss/src/tensorrt-oss/plugin/normalizePlugin/normalizePlugin.cpp,112
0# dd::OatppJsonAPI::abort(int) at /opt/deepdetect/src/oatppjsonapi.cc:253
1# 0x00007F2C7EC30210 in /usr/lib/x86_64-linux-gnu/libc.so.6
2# raise in /usr/lib/x86_64-linux-gnu/libc.so.6
3# abort in /usr/lib/x86_64-linux-gnu/libc.so.6
4# 0x00007F2C80685C3D in /opt/deepdetect/build/lib/libnvinfer_plugin.so.7
5# nvinfer1::rt::cuda::PluginV2Runner::execute(nvinfer1::rt::CommonContext const&, nvinfer1::rt::ExecutionParameters const&) const in /usr/lib/x86_64-linux-gnu/libnvinfer.so.7
6# nvinfer1::rt::ExecutionContext::enqueueInternal(CUevent_st**) in /usr/lib/x86_64-linux-gnu/libnvinfer.so.7
7# nvinfer1::rt::ExecutionContext::enqueue(int, void**, CUstream_st*, CUevent_st**) in /usr/lib/x86_64-linux-gnu/libnvinfer.so.7
8# dd::TensorRTLib<dd::ImgTensorRTInputFileConn, dd::SupervisedOutput, dd::TensorRTModel>::predict(dd::APIData const&, dd::APIData&) at /opt/deepdetect/src/backends/tensorrt/tensorrtlib.cc:678
9# mapbox::util::detail::dispatcher<dd::visitor_mllib::v_predict_job&, mapbox::util::variant<dd::MLService<dd::TensorRTLib, dd::ImgTensorRTInputFileConn, dd::SupervisedOutput, dd::TensorRTModel> >, int, dd::MLService<dd::TensorRTLib, dd::ImgTensorRTInputFileConn, dd::SupervisedOutput, dd::TensorRTModel> >::apply(mapbox::util::variant<dd::MLService<dd::TensorRTLib, dd::ImgTensorRTInputFileConn, dd::SupervisedOutput, dd::TensorRTModel> >&, dd::visitor_mllib::v_predict_job&) at /usr/include/mapbox/variant.hpp:348
10# dd::Services::predict(dd::APIData const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, dd::APIData&, bool const&) at /opt/deepdetect/src/services.h:623
11# dd::JsonAPI::service_predict(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) at /opt/deepdetect/src/jsonapi.cc:1161
12# DedeController::Z__PROXY_METHOD_predict(std::shared_ptr<oatpp::web::protocol::http::incoming::Request> const&) at /opt/deepdetect/src/http/controller.hpp:150
13# oatpp::web::server::api::ApiController::Handler<DedeController>::handle(std::shared_ptr<oatpp::web::protocol::http::incoming::Request> const&) at /opt/deepdetect/build/oatpp/src/oatpp/src/oatpp/web/server/api/ApiController.hpp:300
14# oatpp::web::server::HttpProcessor::processNextRequest(oatpp::web::server::HttpProcessor::ProcessingResources&, std::shared_ptr<oatpp::web::protocol::http::incoming::Request> const&, oatpp::web::protocol::http::utils::CommunicationUtils::ConnectionState&) in /opt/deepdetect/build/main/dede
15# oatpp::web::server::HttpProcessor::processNextRequest(oatpp::web::server::HttpProcessor::ProcessingResources&) in /opt/deepdetect/build/main/dede
16# oatpp::web::server::HttpProcessor::Task::run() in /opt/deepdetect/build/main/dede
17# 0x00007F2C7EF0FD84 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6
18# start_thread in /usr/lib/x86_64-linux-gnu/libpthread.so.0
19# __clone in /usr/lib/x86_64-linux-gnu/libc.so.6
We already had a few issues with refinedet and you modified the CI but it does not seem to work well here.
Thanks in advance for your help
Metadata
Metadata
Assignees
Labels
No labels