API: Add support for search queries with incremental updates #995
Conversation
|
I haven't really understood things well enough to know whether this separate API endpoint is a good idea or not. But regarding the feedback on #723 - I think a better way to do paging in that PR might have been to do it by timestamp instead of count. New pictures auto-uploaded from phone will have a recent timestamp. Paging by count that may shift the results, but paging by timestamp should avoid that issue |
The new API returns all database entries sorted such that the oldest comes first and the newest comes last. Like this, changes will always be added to the bottom of the list. If I use the |
|
@lastzero did you already have time to look into this PR, do you have feedback for me? |
|
Sorry, no :/ drowning in work... didn't forget about this and all the other PRs! |
|
Any update on this? Have been holding off updating the mobile app but auto uploading is pretty broken on the old version. |
|
Unfortunately not, we won't find time to review pull requests before April. |
Ok, no problem. Not urgent for me until google photos ends free storage in june, after that I think that the mobile app will become a lot more important to more people! |
|
Sorry for the wait, I finally have time to look through open pull requests (once again). Probably makes sense to add a third search endpoint in addition to the regular (file based) API and the Geo API for Places. Question: If this endpoint only returns updated entries, is sorting then happening in the client code? If true, we may skip sorting on the server and return all matching results at once. |
Yes sorting and filtering happens on the client side. We could skip sorting on the server side but then I think we should add an offset parameter to get the remaining results if there are more than |
|
I'm not sure offset alone would work as well. What if a new photo is added in the meantime? |
Without sorting, an example workflow for the API could be:
this assumes (both, current implementation and without sorting):
If you can still think of a case where adding a new photo would pose a problem, can you please explain it with an example? |
In your example you get 100 results in the first query and 40 results in the second query. Let’s say result A is part of the first 100 results and result B is not. Whithout sorting, which guarantees exist to make sure result A is not getting returned again as part of the 40 results, and result B again not being returned? If you want all results retuned, you would be relying on an implicit ordering as far as I can see. |
Yes, we would be relying on an implicit ordering, but that is much better from a performance perspective compared to sorting the entries explicitly. |
|
@lastzero I removed the sorting from endpoint. Anything other remarks or could we merge it like this? |
b332dd0 to
0bd0d18
Compare
|
Any updates on this ? |
|
We did not forget about this. But at the moment Facial Recognition is consuming all our resources. |
|
Since our search API is part of the application core and merging it is a commitment to maintain it for years to come, I'm currently the only one who can review it, merge it, and refactor it if needed. However, you're welcome to take a look at any issues labeled "help wanted", as that will give me time to work on pull requests instead of taking care of small bugs and improvements that could be addressed by contributors. |
|
I'd like to see this merged as much as everyone else, but until then, I've found this mobile app which runs nicely without any extra APIs: https://radiokot.com.ua/p/photoprism-android-gallery |
Cool, thanks! It lacks the one thing I miss, which is the automatic sync support. I have not yet found any free apps that does this well with Photoprism. |
Yeh, similar for me. I'm fairly sure the flutter based app referenced in the OP does sync and supports iOS. I was originally waiting for that app to be usable (so by extension, waiting for this issue to be completed). That and multi-user support via OIDC are the two things I needed before implementing a PhotoPrisim instance. Getting concerned both are going to be behind the outrageously expensive paywall though so might have to start looking for an alternative or consider forking this and stripping the paywalls. |
Foldersync can do auto sync for free but with adds |
How do you think we can release OIDC support and merge this if we can't pay our bills? 🤔 |
I'll take that as admission OIDC is going to be a subscription only feature. I have no issues with you needing to pay your bills, but your pricing is extreme considering it doesn't include storage and bandwidth. It also seems like development choices are being made that are more focused on profit over quality of product. High detailed maps being paid because it costs you money for example... hmm, seems like an excuse when many users would be ecstatic with OSM based maps. Time for a community fork potentially. PhotoPrisium anyone? |
Trust me, I'd rather not go down that road unless absolutely needed. I am happy to engage any and every conversation that could lead away from that road. I'm creating an opinionated stack for people to be able to self-host and stop needing Google/Apple/Facebook/etc. Much more interested on the orchestration and integration side of hosting multiple services than needing to put effort onto maintaining individual services. Work historically fills my desire for product driven development activities 😅 Have already read this before my earlier comments. Is there anything in particular you want me to take from it though? As I've said, I have no issue with paying. I pay for several FOSS based services, even when using the self-hosted (usually completely free and unrestricted) version because I believe as you do that projects need funding to be sustainable. But your pricing is very difficult to justify (this comment is directed primarily at the self-hosted options; managed options that need to cover bandwidth, storage, admin and support are a different kettle of fish entirely). Have also read previously to commenting. Is the reason for linking to it implying I am violating it or encouraging me to contribute to the below issues instead of forking? I am open to contributing but not with the current situation of PRs sitting open for years. A focus on "zero bugs" at the cost of progress is typically a reason to suspect an inexperienced development team and unsustainable underlying business model. I'd be more likely to contribute to Immich where development moves faster in the current state of the two projects as that will ultimately be more likely to lead to a sustainable project/product. Have you considered seeking funding from venture capitalists as an option or crowdsourcing funding? The specifically linked comment is the standard boilerplate explanation why things are taking time. Not sure what specifically to take from it. There is no answer anywhere (I have seen at least) on whether OIDC support will be a premium feature or not. Currently users can use the CLI to add users according comments I've read elsewhere so that's good. Convenience features being premium tend to work out fine if you can get similar functionality another way. Typically the users who can use a CLI are also the ones who are more likely to be able to contribute in others ways (i.e. PRs). I'm willing to assume most users who are looking to self host are also using tools like Authelia. I'd be very curious to hear the justification of why OIDC should be a premium feature beyond "because we can and because we need funding". Yes, this is slightly more interesting in that there is the semblance of a technical driven argument over political. But again, an easy technical solution was proposed and pushed aside by yourself for political reasons. You're essentially asking your community in the linked issue above to duplicate the work already done by the OSM community. That mentality doesn't tend to sit well with OSS communities. We want to build on what has already been done, not recreate the wheel (yes, I know there are some developers that love to recreate technically superior wheels and we love their work, but there is a time and place for that and this is not it). Excuse my appearance of ranting. Please take it as evidence of someone who wants your team to succeed and be profitable rather than someone who is entitled. I understand the work that goes into these types of projects. |
I just do that with Syncthing. Very reliable. |
|
Have you considered other funding options like venture capital? Yes, we have considered other options many times. Based on feedback from our community, we decided to fund development with paid memberships that provide additional features and services. Keep in mind that by accepting venture capital, we would lose our independence, which could ultimately ruin our mission. |
Yeh, I've also seen a few suggestions of using icloudpd and the google play equivalent options as well. Definitely options although a tad hacky and not the easiest to setup and maintain in household situations (not many want to grab a cookie to get past Apple's 2FA). |
I do that with Nextcloud, no problems at all, either. |
@binaryben It is a membership fee that allows us to maintain the project for everyone, including users who don't pay anything. If we provide all features for free to self-hosters, we'll have to change our funding model to focus e.g. on a SaaS solution, which goes against our mission to protect your privacy. That said, we are open to discussing our pricing - just not here.
With venture capital, we lose control and make sure that everything we offer is commercialized. If you have experience in the industry, you should know that. Our model is basically crowdfunding or sponsorware. That means we plan to release member features to everyone at a later time, which is mentioned in the FAQs and has already happened for some features and config options.
You seem to think that we are making a big profit and OIDC will be developed anyway, which is not the case. Once we have it, we can think about how to make it available.
Giving advice is one thing, but doing so in GitHub Issues (PR comments) in a disrespectful way is something we cannot tolerate. This is because it affects our well-being and keeps us from doing the work you seem to think will get done anyway. |
I'll respectfully disagree. There are plenty of well known and established privacy focused providers of various services. Hosted solutions does not equate to being anti privacy. Take Protonmail for example. There are numerous ways to implement this. You could take a zero knowledge approach for example and encrypt the photos at rest.
Fair. Please direct me to a better place. I'd prefer somewhere that is still publicly visible for transparency.
Again, a flawed assumption. Yes there is a risk of that happening. I've even recently been concerned exactly that was happening in another open source project currently navigating the same issue of how to be profitable because I have dealt with it. It's not a given though (see same project and linked discussion for more info).
It's an attempt at bootstrapping to be fair. In itself, that's a very noble thing. It's just going to be an uphill battle to convince users to pay $80 in perpetuity to host their own data. If you want it to be seen as sponsorware or crowdfunded, put bounties on feature requests and let people vote with their wallets. I certainly would for some key features.
Neat. That could certainly be communicated more widely and there could be a more defined process around this. It's similar to Directus taking on a BSL which I support.
Quite the opposite, I think you're struggling and it's sad to see.
I can only infer what will be developed by what is shown in your roadmap which clearly shows development of OIDC support is in-progress #782.
I have no idea what to make of this... all that comes to mind is failure to have a plan is a plan to fail. I'd really like this project to succeed and become commercially viable. I am very concerned about the monopoly the likes of Apple and Google are having on everyones most personal and sentimental data.
I'm not being disrespectful, just blunt and to the point. If you need me to soften my communication, I will. Please direct me to where you would like discussion about pricing to take place. |
@binaryben It may be that you think this kind of communication is respectful, but it is not and your comments have distracted us from other work. As a result, I have blocked you from commenting until further notice. The reason why we don't explain our approach more often and in more detail (e.g. in a blog) is simply that our team is too small. However, we know it's important and something we should improve. If we offer everything for free, that won't change though. Of course, we have also considered many other options. Your comments about pricing suggest that everyone has to sign up for the most expensive tier, which is simply not the case. Those who do sign up give us overwhelmingly positive feedback. If you don't agree with that, you are free to use other software. Finally, the prices on our website were preliminary. On GitHub, you can currently sign up for $64 (one time). The amount we now quote for PhotoPrism+ when you pay via Stripe is the same, just in Euros. We've also lowered the price for the Essentials tier from EUR 24.90 to EUR 23.00, and are offering a discount to students, teachers, and non-profit organizations. |
|
It would have been appropriate for binaryben to fully disclose his motivation for creating a fork. On LinkedIn, he describes himself as an entrepreneur and the only obvious reason he can't use our software without OpenID Connect or other member features is that he wants to monetize it as part of his own product: |
|
@thielepaul My apologies for taking such an unusually long time to provide final feedback! We really appreciate the work that has been put into the mobile app, so we didn't want to just close the PR saying that it can't be implemented this way. In order to come to a final conclusion, it was important for us to have a meaningful and secure multi-user support ready so we know what is required and could maintain our ability to release incremental updates. It is now clear that we cannot allow clients to perform direct queries that bypass existing filters and security mechanisms. I am happy to make suggestions for alternatives if there is a chance that the mobile app will continue to be maintained. This has not always been the case in the past, even in the early days (as explained above). That's why we moved on to other issues before this PR was submitted. Obviously, the back and forth was confusing for our team as well, but sometimes that's just the way it is. In any case, this needs to be reworked (e.g. integrated with our existing search API) as security is a top priority for us and many of our users. It therefore seems best to close this PR with an offer to work on an alternative solution if that is an option and there is time and motivation to continue working on this particular mobile app. To all users who are still following, I would like to say that this was never in the way of developing a mobile app in general, as there are many other apps that work flawlessly with PhotoPrism and where the developers never had any problems with the integration, for example Gallery for PhotoPrism and the apps we feature on our partners page. In other words, it was completely unnecessary to get angry, and in some cases, extremely disrespectful. That's one more reason to close this PR now and make a fresh start. Thank you very much! |
Following the discussions in thielepaul/photoprism-mobile#78, I created a new backend API for the community-maintained flutter app.
It is a new endpoint so there is no influence on web frontend.
The concept behind this API is to use the "UpdatedAt" and "DeletedAt" fields in the SQL database provided by Gorm that allow us to fetch all changes since the last request.
A few data updates in the photoprism backend use the UpdateColumn method instead of the Update method and thus, change the data in the database without updating the timestamp. As this would mean that the new API could not get these updates, I replaced all usages of the UpdateColumn method with the Update method.
see thielepaul/photoprism-mobile#96 for the implementation of the new API in the app