Once you have more than a few paths assigned to a user, it becomes
quite hard to oversee the list.
Personally, I would add my albums to Photoview one by one, so I can
manage the processing better. For example, I can review and fix the face
recognition for one new album at a time, and I don't need to deal with
too much data.
Since I started to add the albums in a random order, it's relatively
hard to find which album is added and which is not.
An easy fix is to order the photo paths by their name.
- Extended the query of the rootAlbums to order them by path
* Validate incoming GPS data and throw an error if it is incorrect, storing Null values
* Extracted GPS data processing to function in external parser; optimized IF in internal parser; removed unnecessary comments; set exact values for positive test
* Add the migration for removing existing invalid GPS data and its test; added better errors to asserts in the GPS validation test
* Install FFmpeg and ExifTool on the API unit-test environment
* Fix 'stripped.jpg', 'IncorrectGPS.jpg', and 'CorrectGPS.jpg' tests for the external parser
* Optimized data validation in the external parser, returned error by the internal parser for invalid data, updated test to expect errors and handle them
* Switched from error to log entry in case of incorrect GPS data, as error handling is not so transparent in the internal parser
---------
Co-authored-by: Konstantin Koval
* Bump go version to 1.22. Bump all go module to the newest version, except pinning libheif@v1.15.1 to match the c lib version in Debian bookworm.
* Update go_wrapper.sh to fit golang image.
* Update the way to set go envs.
* Test against go 1.22
* Set default shell for all building stage in Dockerfile.
Add a "no_face_detection" build tag to disable face detection
when building. This is useful when installing the face detection
dependencies is undesirable and cuts down build times (e.g. on a
Raspberry Pi).
* Don't stop scanning album on media fail
* Update api/scanner/scanner_album.go
accepting suggestion from Jordan
Co-authored-by: Jordan Hellier <13520761+jordy2254@users.noreply.github.com>
---------
Co-authored-by: Konstantin Koval <kkb@ukr.net>
Co-authored-by: Jordan Hellier <13520761+jordy2254@users.noreply.github.com>
I encountered with the following error:
> 2023/02/05 07:33:00 /app/scanner/face_detection/face_detector.go:92 sql: transaction has already been committed or rolled back
> [0.042ms] [rows:0] SELECT * FROM `media` WHERE `media`.`id` = 823 ORDER BY `media`.`id` LIMIT 1
> 2023/02/05 07:33:00 ERROR: Error detecting faces in image (/photos/Borzsony2017/DSC_0028.NEF): sql: transaction has already been committed or rolled back
It turned out it comes from the api/routes/photos.go
I found a very similar code in album_scanner.go.
The difference I saw was that while in the single photo request the
transaction passed to the `scanner_tasks.Tasks.BeforeProcessMedia` call,
in the album_scann.go the transaction created after this call and
created from the context which returned by `BeforeProcessMedia`.
Another difference was that in the `ProcessSingleMedia` call the
`AfterProcessMedia` call was called with the same - db transaction -
context, in the album_scanner it was called outside of the transaction.
I changed the logic by merging the two behavior:
Create the transaction from the context of `BeforeProcessMedia` and also
use the transaction context in the `AfterProcessMedia`.
After the change the error disappeared.
So to have it in a common place I extracted that logic into a function
and use for both the single photo request and in the album scanner.
I did not go more deeper to find out what's going on with the context
under the hood.
2.3.12 -> 2.3.13 restructured ScanAlbum so that errors were returned rather than raising a ScannerError directly - this ensures that the errors are logged correctly.
The subquery returns all potential media id's. Typically, we have a couple of faces and thousands of media files.
A join is much faster.
With about 50k images and a face that was present in 2'000 images the query was unusable slow, it took about 60s.
This is a very important performance fix, as I think many users will run into it.
Maybe there are other areas where the same improvement is possible, I didn't check.
Googol Lee