Fetch more articles for current feed or category #532

Alkarex · 2024-11-20T10:31:17Z

Alkarex
Nov 20, 2024

In the GReader API, it is possible to fetch articles for a given feed or for a given category.
This is especially useful when there are many articles on the server (more than what the app is fetching) and the user wants to fetch older articles for a given feed or category.

So in the CapyReader app, it would be nice if the "pull to refresh" feature, when done from within a feed or a category, would fetch articles from that feed or category.

I can give more details if this is not clear enough, and do not hesitate to ask for help on back-end side (FreshRSS).

jocmp · 2024-12-03T23:49:21Z

jocmp
Dec 3, 2024
Maintainer

@Alkarex I've added functionality for this in the latest dev release, https://github.com/jocmp/capyreader/releases/tag/2024.12.1080-dev. Let me know if this is working as expected!

5 replies

Alkarex Dec 4, 2024
Author

Hello,
I am glad to see that you are progressing :-)
Are you sure it is the "Fetch more articles for current feed or category" feature, you have added?
Maybe confused with:

Fetch read articles #564

Quick test on a feed that has plenty of entries on the server, but not so many recently, in view all articles modes.
I initially have two articles on that view, scroll to refresh, many (big) requests are made (could be improved), but no request that I can see is filtered on that feed. No new articles appear, unfortunately. Here is a log below, and let me know if you like more feedback.

2024-12-04T07:24:55.800064523Z 130.XXX.XXX.XXX - - [04/Dec/2024:08:24:55 +0100] "GET /api/greader.php/reader/api/0/subscription/list?output=json HTTP/1.1" 200 9987 "-" "CapyReader (RSS Reader https://capyreader.com/) okhttp/4.12.0"
2024-12-04T07:24:56.173118421Z 130.XXX.XXX.XXX - - [04/Dec/2024:08:24:55 +0100] "GET /api/greader.php/reader/api/0/stream/items/ids?s=user%2F-%2Fstate%2Fcom.google%2Fstarred&n=10000&output=json HTTP/1.1" 200 14372 "-" "CapyReader (RSS Reader https://capyreader.com/) okhttp/4.12.0"
2024-12-04T07:24:56.678484261Z 130.XXX.XXX.XXX - - [04/Dec/2024:08:24:56 +0100] "GET /api/greader.php/reader/api/0/stream/items/ids?s=user%2F-%2Fstate%2Fcom.google%2Freading-list&n=10000&xt=user%2F-%2Fstate%2Fcom.google%2Fread&output=json HTTP/1.1" 200 59249 "-" "CapyReader (RSS Reader https://capyreader.com/) okhttp/4.12.0"
2024-12-04T07:24:57.090059148Z 130.XXX.XXX.XXX - - [04/Dec/2024:08:24:56 +0100] "GET /api/greader.php/reader/api/0/stream/items/ids?s=user%2F-%2Fstate%2Fcom.google%2Freading-list&ot=1733297062&n=100&output=json HTTP/1.1" 200 16 "-" "CapyReader (RSS Reader https://capyreader.com/) okhttp/4.12.0"
2024-12-04T07:24:57.651534860Z 130.XXX.XXX.XXX - - [04/Dec/2024:08:24:57 +0100] "POST /api/greader.php/reader/api/0/stream/items/contents?output=json HTTP/1.1" 200 100107 "-" "CapyReader (RSS Reader https://capyreader.com/) okhttp/4.12.0"
2024-12-04T07:24:57.659281396Z 130.XXX.XXX.XXX - - [04/Dec/2024:08:24:57 +0100] "POST /api/greader.php/reader/api/0/stream/items/contents?output=json HTTP/1.1" 200 114071 "-" "CapyReader (RSS Reader https://capyreader.com/) okhttp/4.12.0"
2024-12-04T07:24:57.719098348Z 130.XXX.XXX.XXX - - [04/Dec/2024:08:24:57 +0100] "POST /api/greader.php/reader/api/0/stream/items/contents?output=json HTTP/1.1" 200 108591 "-" "CapyReader (RSS Reader https://capyreader.com/) okhttp/4.12.0"
2024-12-04T07:24:57.758594976Z 130.XXX.XXX.XXX - - [04/Dec/2024:08:24:57 +0100] "POST /api/greader.php/reader/api/0/stream/items/contents?output=json HTTP/1.1" 200 56884 "-" "CapyReader (RSS Reader https://capyreader.com/) okhttp/4.12.0"
2024-12-04T07:24:57.800327384Z 130.XXX.XXX.XXX - - [04/Dec/2024:08:24:57 +0100] "POST /api/greader.php/reader/api/0/stream/items/contents?output=json HTTP/1.1" 200 140060 "-" "CapyReader (RSS Reader https://capyreader.com/) okhttp/4.12.0"
2024-12-04T07:24:58.212670999Z 130.XXX.XXX.XXX - - [04/Dec/2024:08:24:57 +0100] "POST /api/greader.php/reader/api/0/stream/items/contents?output=json HTTP/1.1" 200 62139 "-" "CapyReader (RSS Reader https://capyreader.com/) okhttp/4.12.0"
2024-12-04T07:24:58.228826293Z 130.XXX.XXX.XXX - - [04/Dec/2024:08:24:57 +0100] "POST /api/greader.php/reader/api/0/stream/items/contents?output=json HTTP/1.1" 200 90874 "-" "CapyReader (RSS Reader https://capyreader.com/) okhttp/4.12.0"
2024-12-04T07:24:58.387913569Z 130.XXX.XXX.XXX - - [04/Dec/2024:08:24:57 +0100] "POST /api/greader.php/reader/api/0/stream/items/contents?output=json HTTP/1.1" 200 108199 "-" "CapyReader (RSS Reader https://capyreader.com/) okhttp/4.12.0"
2024-12-04T07:24:58.392909937Z 130.XXX.XXX.XXX - - [04/Dec/2024:08:24:57 +0100] "POST /api/greader.php/reader/api/0/stream/items/contents?output=json HTTP/1.1" 200 97157 "-" "CapyReader (RSS Reader https://capyreader.com/) okhttp/4.12.0"
2024-12-04T07:24:58.407226480Z 130.XXX.XXX.XXX - - [04/Dec/2024:08:24:57 +0100] "POST /api/greader.php/reader/api/0/stream/items/contents?output=json HTTP/1.1" 200 86380 "-" "CapyReader (RSS Reader https://capyreader.com/) okhttp/4.12.0"
2024-12-04T07:24:58.640276581Z 130.XXX.XXX.XXX - - [04/Dec/2024:08:24:58 +0100] "POST /api/greader.php/reader/api/0/stream/items/contents?output=json HTTP/1.1" 200 62427 "-" "CapyReader (RSS Reader https://capyreader.com/) okhttp/4.12.0"
2024-12-04T07:24:58.878302391Z 130.XXX.XXX.XXX - - [04/Dec/2024:08:24:58 +0100] "POST /api/greader.php/reader/api/0/stream/items/contents?output=json HTTP/1.1" 200 96584 "-" "CapyReader (RSS Reader https://capyreader.com/) okhttp/4.12.0"
2024-12-04T07:24:58.889764214Z 130.XXX.XXX.XXX - - [04/Dec/2024:08:24:58 +0100] "POST /api/greader.php/reader/api/0/stream/items/contents?output=json HTTP/1.1" 200 77906 "-" "CapyReader (RSS Reader https://capyreader.com/) okhttp/4.12.0"
2024-12-04T07:24:58.987641530Z 130.XXX.XXX.XXX - - [04/Dec/2024:08:24:58 +0100] "POST /api/greader.php/reader/api/0/stream/items/contents?output=json HTTP/1.1" 200 78372 "-" "CapyReader (RSS Reader https://capyreader.com/) okhttp/4.12.0"
2024-12-04T07:24:59.026547174Z 130.XXX.XXX.XXX - - [04/Dec/2024:08:24:58 +0100] "POST /api/greader.php/reader/api/0/stream/items/contents?output=json HTTP/1.1" 200 120494 "-" "CapyReader (RSS Reader https://capyreader.com/) okhttp/4.12.0"
2024-12-04T07:24:59.047526236Z 130.XXX.XXX.XXX - - [04/Dec/2024:08:24:58 +0100] "POST /api/greader.php/reader/api/0/stream/items/contents?output=json HTTP/1.1" 200 42030 "-" "CapyReader (RSS Reader https://capyreader.com/) okhttp/4.12.0"
2024-12-04T07:24:59.300856026Z 130.XXX.XXX.XXX - - [04/Dec/2024:08:24:58 +0100] "POST /api/greader.php/reader/api/0/stream/items/contents?output=json HTTP/1.1" 200 62758 "-" "CapyReader (RSS Reader https://capyreader.com/) okhttp/4.12.0"
2024-12-04T07:24:59.318638320Z 130.XXX.XXX.XXX - - [04/Dec/2024:08:24:59 +0100] "POST /api/greader.php/reader/api/0/stream/items/contents?output=json HTTP/1.1" 200 81365 "-" "CapyReader (RSS Reader https://capyreader.com/) okhttp/4.12.0"
2024-12-04T07:24:59.336580003Z 130.XXX.XXX.XXX - - [04/Dec/2024:08:24:58 +0100] "POST /api/greader.php/reader/api/0/stream/items/contents?output=json HTTP/1.1" 200 71952 "-" "CapyReader (RSS Reader https://capyreader.com/) okhttp/4.12.0"

jocmp Dec 4, 2024
Maintainer

hey @Alkarex - I posted to the wrong thread, thanks for following up in the other one. I haven't started work on this "Fetch more" but I'll track it via #574.

In both this case and the "read" case the "Pull to Refresh" currently refreshes all feeds as you've noted. Initially I did this to avoid the complexity of passing conditional logic to the refresh task, but as you note that comes with a performance hit. I'll track that via #575.

issteve Jan 5, 2025

It looks like there is no pull to refresh in the feeds overview, right? I have it on "refresh only manual" and it would be great to be able to refresh all feeds with one swipe from above in the feeds overview...

jocmp Jan 7, 2025
Maintainer

@issteve yep there is - it's available on the top-level feeds, folders, and individual feeds as well.

Screen_recording_20250106_185645.mp4

issteve Jan 7, 2025

@issteve yep there is - it's available on the top-level feeds, folders, and individual feeds as well.
Screen_recording_20250106_185645.mp4

Thank you for your response. I tried it again in the feeds overview (the one under the three lines having the "Feeds" headline and showing all the feeds) and it didn't show the behaviour in your video.
Why I need it there?
I have the unread filter always on and the "refresh only manual" option. So the feed I want to refresh is not listed and it doesn't get refreshed and I can't refresh it manually because I can't go into it. So I would have to switch the unread filter always back and forth.
I helped myself by keeping always one item in one feed to do the manual refresh there and hope it will update all the other feeds as well... But it's still somehow one annoying step to much if this feed doesn't have something new...

jocmp · 2025-01-10T02:25:59Z

jocmp
Jan 10, 2025
Maintainer

@Alkarex this is included in the 2025.01.1093-dev release. Here's how it works

On a top level feed, e.g. "All" or "Unread," only the latest read and unread articles are fetched based on the "max arrived at" timestamp. This is the timestamp of the last article that was received. This is unchanged from previous versions.
On a category feed, e.g. a folder or an individual feed, the entire feed is refetched. This has some overhead but should pick up on any gaps including older articles.

2 replies

Alkarex Jan 11, 2025
Author

Nice progress 👍🏻

the entire feed is refetched

I suggest adding an option to define the max number of articles to retrieve (including high values such as 10k). It is very possible to have a category (or even a feed) with 100k+ articles, and fetching everything is a no-go

jocmp Jan 11, 2025
Maintainer

Good point! I'll add the 10k max to the next update

[FreshRSS] Only fetch 10k articles at most from categories #713

Uh oh!

Fetch more articles for current feed or category #532

Uh oh!

Alkarex Nov 20, 2024

Replies: 2 comments · 7 replies

Uh oh!

jocmp Dec 3, 2024 Maintainer

Uh oh!

Alkarex Dec 4, 2024 Author

Uh oh!

jocmp Dec 4, 2024 Maintainer

Uh oh!

Uh oh!

issteve Jan 5, 2025

Uh oh!

jocmp Jan 7, 2025 Maintainer

Uh oh!

issteve Jan 7, 2025

Uh oh!

jocmp Jan 10, 2025 Maintainer

Uh oh!

Uh oh!

Alkarex Jan 11, 2025 Author

Uh oh!

jocmp Jan 11, 2025 Maintainer

Alkarex
Nov 20, 2024

Replies: 2 comments 7 replies

jocmp
Dec 3, 2024
Maintainer

Alkarex Dec 4, 2024
Author

jocmp Dec 4, 2024
Maintainer

jocmp Jan 7, 2025
Maintainer

jocmp
Jan 10, 2025
Maintainer

Alkarex Jan 11, 2025
Author

jocmp Jan 11, 2025
Maintainer