8000 CUDNN_STATUS_BAD_PARAM tensorflow/stream_executor/cuda/cuda_dnn.cc:423 · Issue #4111 · tensorflow/tensorflow · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

CUDNN_STATUS_BAD_PARAM tensorflow/stream_executor/cuda/cuda_dnn.cc:423 #4111

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
chez-tozhang opened this issue Aug 31, 2016 · 0 comments
Closed

CUDNN_STATUS_BAD_PARAM tensorflow/stream_executor/cuda/cuda_dnn.cc:423 #4111

chez-tozhang opened this issue Aug 31, 2016 · 0 comments

Comments

@chez-tozhang
Copy link

NOTE: Only file GitHub issues for bugs and feature requests. All other topics will be closed.

For general support from the community, see StackOverflow.
To make bugs and feature requests more easy to find and organize, we close issues that are deemed
out of scope for GitHub Issues and point people to StackOverflow.

For bugs or installation issues, please provide the following information.
The more information you provide, the more easily we will be able to offer
help and advice.

What related GitHub issues or StackOverflow threads have you found by searching the web for your problem?

CUDNN_STATUS_BAD_PARAM

Environment info

Operating System:OS: Ubuntu 14.04

Installed version of CUDA and cuDNN:
(please attach the output of ls -l /path/to/cuda/lib/libcud*):
-rw-r--r-- 1 root root 322936 Aug 16 2015 /usr/local/cuda/lib64/libcudadevrt.a
lrwxrwxrwx 1 root root 16 Aug 16 2015 /usr/local/cuda/lib64/libcudart.so -> libcudart.so.7.5
lrwxrwxrwx 1 root root 19 Aug 16 2015 /usr/local/cuda/lib64/libcudart.so.7.5 -> libcudart.so.7.5.18
-rwxr-xr-x 1 root root 383336 Aug 16 2015 /usr/local/cuda/lib64/libcudart.so.7.5.18
-rw-r--r-- 1 root root 720192 Aug 16 2015 /usr/local/cuda/lib64/libcudart_static.a
lrwxrwxrwx 1 root root 13 Aug 30 22:56 /usr/local/cuda/lib64/libcudnn.so -> libcudnn.so.4
lrwxrwxrwx 1 root root 17 Aug 30 22:56 /usr/local/cuda/lib64/libcudnn.so.4 -> libcudnn.so.4.0.7
-rwxr-xr-x 1 root root 61272736 Jan 12 2016 /usr/local/cuda/lib64/libcudnn.so.4.0.4
-rwxr-xr-x 1 root root 61453024 Aug 30 22:55 /usr/local/cuda/lib64/libcudnn.so.4.0.7
-rw-r--r-- 1 root root 62025862 Aug 30 22:55 /usr/local/cuda/lib64/libcudnn_static.a

If installed from binary pip package, provide:

  1. A link to the pip package you installed:
    https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow-0.7.1-cp27-none-linux_x86_64.whl
  2. The output from python -c "import tensorflow; print(tensorflow.__version__)".
    I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcublas.so locally
    I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcudnn.so locally
    I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcufft.so locally
    I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcuda.so.1 locally
    I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcurand.so locally
    0.10.0rc0
    If installed from source, provide
  3. The commit hash (git rev-parse HEAD)
  4. The output of bazel version

If possible, provide a minimal reproducible example (We usually don't have time to read hundreds of lines of your code)

I also used data from hd5f file, before the tf.Session I extract the data first.
It works for the 200000 iterations, but when it comes more than 200000 the program will be terminated.

What other attempted solutions have you tried?

I tried to re-install cudnn 4.0

Logs or other output that would be helpful

(If logs are large, please upload as attachment or provide link).
Iteration: 240640
F tensorflow/stream_executor/cuda/cuda_dnn.cc:423] could not set cudnn tensor descriptor: CUDNN_STATUS_BAD_PARAM
Aborted (core dumped)
Strange thing happens after the iteration:240640 every time
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@chez-tozhang
0