8000 Crash while importing photos · Issue #202 · photoprism/photoprism · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Crash while importing photos #202

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
lastzero opened this issue Jan 16, 2020 · 35 comments
Closed

Crash while importing photos #202

lastzero opened this issue Jan 16, 2020 · 35 comments
Assignees
@lastzero
Labels
bug Something isn't working priority Supported by early sponsors or popular demand released Available in the stable release
Milestone
MVP

Comments

@lastzero
Copy link
Member

Reported by Cristian: Running import from the web works for a few seconds and then crashes. If I go into the container and manually run photoprism import it works for a while and then:

...
INFO[2020-01-16T17:28:01Z] exif: taken at 2014-10-20 10:09:16 +0000 UTC 
INFO[2020-01-16T17:28:01Z] import: moving main jpg file "google-takeout/Google Photos/2014-10-20/IMG_20141020_100916895.jpg" to "/assets/photos/originals/2014/10/20141020_100916_34CD65076EBE.jpg" 
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x1d577e6]

goroutine 16 [running]:
github.com/photoprism/photoprism/internal/photoprism.(*MediaFile).Filename(...)
	/go/src/github.com/photoprism/photoprism/internal/photoprism/mediafile.go:307
github.com/photoprism/photoprism/internal/photoprism.(*MediaFile).Basename(0x0, 0xc000130280, 0x40687c0)
	/go/src/github.com/photoprism/photoprism/internal/photoprism/mediafile.go:367 +0x26
github.com/photoprism/photoprism/internal/photoprism.(*Index).MediaFile(0xc004cf29a0, 0x0, 0x101010101010101, 0x101, 0x0, 0x0)
	/go/src/github.com/photoprism/photoprism/internal/photoprism/index_mediafile.go:37 +0x117
github.com/photoprism/photoprism/internal/photoprism.importWorker(0xc00009e960)
	/go/src/github.com/photoprism/photoprism/internal/photoprism/import_worker.go:91 +0xffc
github.com/photoprism/photoprism/internal/photoprism.(*Import).Start.func1(0xc00009e960, 0xc0063fe870)
	/go/src/github.com/photoprism/photoprism/internal/photoprism/import.go:73 +0x2b
created by github.com/photoprism/photoprism/internal/photoprism.(*Import).Start
	/go/src/github.com/photoprism/photoprism/internal/photoprism/import.go:72 +0x209
@lastzero lastzero added the bug Something isn't working label Jan 16, 2020
@lastzero lastzero added this to the MVP milestone Jan 16, 2020
@lastzero
Copy link
Member Author

Our tests are green. Unclear if it's a known issue with darktable-cli (doesn't like to be executed from goroutines like during import) or a new issue. Needs further investigation. I'm off until Monday, wouldn't be sad if somebody can have a look - otherwise I'll do it next week.

@lastzero lastzero added important help wanted Help with this would be much appreciated! labels Jan 16, 2020
@lastzero
Copy link
Member Author

Might be due to a fix I added to import_worker.go recently: Files other than JPEG were not indexed properly after importing them, you had to run index manually

@lastzero lastzero mentioned this issue Jan 16, 2020
Closed
5 tasks
@lastzero
Copy link
Member Author
lastzero commented Jan 16, 2020 •
edited
Loading

It would be possible that another goroutine now imports a newly created JPEG if your import directory contains non-JPEG files that need to be converted first. However, I'm 90% sure filepath.Walk() does not iterate over files added later.

No idea why m should be nil in photoprism/mediafile.go:307:

func (m *MediaFile) Filename() string {
	return m.filename
}

@lastzero lastzero mentioned this issue Jan 16, 2020
Closed
@lastzero
Copy link
Member Author

@casidiablo Tested with 30 files and worked for me, except the expected segmentation faults from darktable-cli:

Screenshot 2020-01-16 at 23 14 14

@lastzero
Copy link
Member Author

Even importing 500 files doesn't cause a crash... need more information! Any specific file or file type that causes this? Special filesystem?

lastzero added a commit that referenced this issue Jan 17, 2020
@lastzero
Backend: Upgrade darktable and don't run more than one command at once …
64a16a2 
…#202

Signed-off-by: Michael Mayer <michael@liquidbytes.net>
@lastzero
Copy link
Member Author

@casidiablo Hope your issue is fixed with this change, if not let us know! Just merged to master, updated Docker image should be available soon.

@lastzero lastzero added the please-test Ready for acceptance test label Jan 17, 2020
@casidiablo
Copy link

I will try once the image is available.

@lastzero
Copy link
Member Author

You can pull and try, only takes ~30 minutes to build and upload.

@lastzero
Copy link
Member Author

See https://travis-ci.org/photoprism/photoprism/builds

@lastzero
Copy link
Member Author

@casidiablo Is this fixed too? See #196

8000
@lastzero
Copy link
Member Author

Downgraded darktable-cli to 3.0.0 to avoid errors when converting RAW to JPEG: darktable-org/darktable#4150

@lastzero lastzero removed the help wanted Help with this would be much appreciated! label Jan 20, 2020
@lastzero
Copy link
Member Author

Closing this since we received no more feedback.

@lastzero lastzero added released Available in the stable release and removed please-test Ready for acceptance test labels Jan 20, 2020
@casidiablo
Copy link

Sorry, was busy. Still happening on 20200124:

INFO[2020-01-24T20:31:46Z] exif: taken at 2017-05-11 15:01:13 +0000 UTC 
INFO[2020-01-24T20:31:46Z] import: moving main jpg file "google-takeout/Google Photos/2017-05-11/IMG_20170511_080113.jpg" to "/assets/photos/originals/2017/05/20170511_150113_0C1F709DE7A0.jpg" 
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x1d5b7c6]

goroutine 43 [running]:
github.com/photoprism/photoprism/internal/photoprism.(*MediaFile).Filename(...)
	/go/src/github.com/photoprism/photoprism/internal/photoprism/mediafile.go:307
github.com/photoprism/photoprism/internal/photoprism.(*MediaFile).Basename(0x0, 0xc000478280, 0x40708a0)
	/go/src/github.com/photoprism/photoprism/internal/photoprism/mediafile.go:367 +0x26
github.com/photoprism/photoprism/internal/photoprism.(*Index).MediaFile(0xc004cd8440, 0x0, 0x101010101010101, 0x101, 0x0, 0x0)
	/go/src/github.com/photoprism/photoprism/internal/photoprism/index_mediafile.go:37 +0x117
github.com/photoprism/photoprism/internal/photoprism.importWorker(0xc00004b6e0)
	/go/src/github.com/photoprism/photoprism/internal/photoprism/import_worker.go:91 +0x1005
github.com/photoprism/photoprism/internal/photoprism.(*Import).Start.func1(0xc00004b6e0, 0xc0063f7c90)
	/go/src/github.com/photoprism/photoprism/internal/photoprism/import.go:73 +0x2b
created by github.com/photoprism/photoprism/internal/photoprism.(*Import).Start
	/go/src/github.com/photoprism/photoprism/internal/photoprism/import.go:72 +0x209

@lastzero
Copy link
Member Author

Seems to be a JPEG file, so not related to converting RAWs? Can you send us a sample to hello@photoprism.org?

@lastzero lastzero added in-progress Somebody is working on this and removed released Available in the stable release labels Jan 24, 2020
@casidiablo
Copy link

Done. It took me a while to reproduce this with a photo that I could actually share.

Also, at first I thought this could be because I was running this on a very small machine, so I moved this to a node with 16GB of ram and 4 cores and it crashed too.

@lastzero
Copy link
Member Author

The error is Data too long for column 'photo_exposure' at row 1

Fixed that a while ago, but you need to delete your db (will be recreated automatically) or increase the size of this column manually. Check our GitHub issues, you'll find other related tickets.

@lastzero
Copy link
Member Author

See also #227

@casidiablo
Copy link

Fixed that a while ago, but you need to delete your db (will be recreated automatically) or increase the size of this column manually. Check our GitHub issues, you'll find other related tickets.

So delete the dbs and run photoprism import or photoprism index?

@lastzero
Copy link
Member Author

Index

@casidiablo
Copy link

Deleted the db, and it did run indexing for longer without crashing until:

INFO[2020-01-26T16:27:30Z] index: adding labels [{Name:building Source:image Uncertainty:80 Priority:0 Categories:[]} {Name:building Source:image Uncertainty:80 Priority:0 Categories:[]}]
WARN[2020-01-26T16:27:32Z] file: no latitude and longitude in metadata
INFO[2020-01-26T16:27:32Z] exif: taken at 2016-12-08 02:48:46 +0000 UTC
INFO[2020-01-26T16:27:32Z] index: changed empty photo title to "Portrait / 2016"
INFO[2020-01-26T16:27:32Z] index: adding labels [{Name:portrait Source:image Uncertainty:70 Priority:0 Categories:[people]} {Name:portrait Source:image Uncertainty:77 Priority:0 Categories:[people]} {Name:portrait Source:image Uncertainty:70 Priority:0 Categories:[people]} {Name:portrait Source:image Uncertainty:77 Priority:0 Categories:[people]}]
INFO[2020-01-26T16:27:32Z] index: added main jpg file "2016/12/20161208_024846_2CB982068C03.jpg"
INFO[2020-01-26T16:27:46Z] index: added main jpg file "2016/12/20161208_024753_A937B89A615A.jpg"
WARN[2020-01-26T16:27:51Z] file: no latitude and longitude in metadata
INFO[2020-01-26T16:27:51Z] index: changed empty photo title to "Unknown / 2016"
INFO[2020-01-26T16:27:56Z] index: added main jpg file "2016/12/20161208_024922_317951135E30.jpg"
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x1d5b7c6]

goroutine 57 [running]:
github.com/photoprism/photoprism/internal/photoprism.(*MediaFile).Filename(...)
        /go/src/github.com/photoprism/photoprism/internal/photoprism/mediafile.go:307
github.com/photoprism/photoprism/internal/photoprism.(*MediaFile).Basename(0x0, 0xc00011e140, 0x40708a0)
        /go/src/github.com/photoprism/photoprism/internal/photoprism/mediafile.go:367 +0x26
github.com/photoprism/photoprism/internal/photoprism.(*Index).MediaFile(0xc004d09500, 0x0, 0x0, 0x0, 0xc00070ff90, 0x3)
        /go/src/github.com/photoprism/photoprism/internal/photoprism/index_mediafile.go:37 +0x117
github.com/photoprism/photoprism/internal/photoprism.indexWorker(0xc00003f1a0)
        /go/src/github.com/photoprism/photoprism/internal/photoprism/index_worker.go:16 +0x4bc
github.com/photoprism/photoprism/internal/photoprism.(*Index).Start.func1(0xc00003f1a0, 0xc0063ef1a0)
        /go/src/github.com/photoprism/photoprism/internal/photoprism/index.go:77 +0x2b
created by github.com/photoprism/photoprism/internal/photoprism.(*Index).Start
        /go/src/github.com/photoprism/photoprism/internal/photoprism/index.go:76 +0x190
command terminated with exit code 2

@lastzero
Copy link
Member Author

Perfect that looks like the original error this ticket was for. Does it always happen with the same file or randomly? If not randomly: can you provide a sample for fixing this?

@lastzero lastzero reopened this Jan 27, 2020
@casidiablo
Copy link

Yeah, I will try. Every time it fails it does so with an image that for privacy reasons I can't share. But I will keep trying. Thanks

@lastzero
Copy link
Member Author

Maybe you can clear the sensitive image contents with Photoshop? Probably has to do with format, sidecar files or metadata, not with the image itself.

@casidiablo
Copy link

Finally found something I can share. Sent to hello@

@lastzero
Copy link
Member Author

Cool, thank you :)

@lastzero
Copy link
Member Author

It's a known issue with the JPEG implementation of Go:

golang/go#10447

We can improve the error message however. Does it affect many important images for you?
Of course we could implement a fallback to another tool in case of read errors, but I'm not sure if this must be part of our first release...

lastzero added a commit that referenced this issue Jan 29, 2020
@lastzero
Backend: Add FileError col to files table #202
183ebd3 
Signed-off-by: Michael Mayer <michael@liquidbytes.net>
@lastzero
Copy link
Member Author

We're logging this now and show a special icon to indicate the file can not be decoded:

Screenshot 2020-01-29 at 17 38 53

Screenshot 2020-01-29 at 17 36 11

@lastzero
Copy link
Member Author

You can search for broken files by using the "error:true" search filter, however there is no "view error details" or "resolve errors" page yet - might do this later!

@lastzero
Copy link
Member Author

We also show the error in the files tab now:

Screenshot 2020-01-29 at 19 08 06
Screenshot 2020-01-29 at 19 07 47
Screenshot 2020-01-29 at 19 07 12

@casidiablo
Copy link

Sure... it's fine if some images can't be indexed, as long as there is good logging AND the indexing/importing process does not crash. Otherwise you need to be running and re-running photoprism import

@lastzero
Copy link
Member Author
lastzero commented Jan 30, 2020 •
edited
Loading

I'll check out the crashing, this should not happen... probably because we use a pointer somewhere to access a thumbnail that isn't there. Hope you c F438 an test again later as I didn't have issues with nil pointers for a long time over here.

@lastzero
Copy link
Member Author

@casidiablo Only happens during photoprism import or also with photoprism index?

lastzero added a commit that referenced this issue Jan 30, 2020
@lastzero
Import: Check if main file is nil #202
ab6680e 
Signed-off-by: Michael Mayer <michael@liquidbytes.net>
@lastzero
Copy link
Member Author

@casidiablo Ok, think I found it... might also happen when there is no main file for some reason... for example because conversion failed or it is not readable. Added nil checks and some more logging, hope this works for you!

We should think about what should happen when there are only sidecar files or videos. Maybe some of your errors are related to video files? We don't have video support yet, but found a sponsor for #17.

@lastzero
Copy link
Member Author

You can test when the master build is green (soonish):

https://travis-ci.org/photoprism/photoprism/builds/643669831

@lastzero lastzero added please-test Ready for acceptance test and removed in-progress Somebody is working on this labels Jan 30, 2020
@casidiablo
Copy link

OK, I will test it when I can. Thanks for the quick response

@lastzero lastzero added released Available in the stable release and removed please-test Ready for acceptance test labels May 3, 2020
@lastzero lastzero self-assigned this May 3, 2020
@lastzero lastzero closed this as completed May 3, 2020
@lastzero lastzero added the priority Supported by early sponsors or popular demand label Dec 14, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@lastzero lastzero

Labels
bug Something isn't working priority Supported by early sponsors or popular demand released Available in the stable release
Projects
None yet
Milestone
MVP
Development

No branches or pull requests
2 participants
@casidiablo @lastzero
0