10000 feat(archive): archive manage by KirCute · Pull Request #7817 · AlistGo/alist · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

feat(archive): archive manage #7817

New issue

Have a question about this project? Sign up for a free GitHub account 8000 to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 10 commits into from
Jan 18, 2025
Merged

feat(archive): archive manage #7817

merged 10 commits into from
Jan 18, 2025

Conversation

KirCute
Copy link
Contributor
@KirCute KirCute commented Jan 13, 2025

对压缩文件的在线预览/在线解压支持,在线预览需要驱动支持 range 请求。
尚未解决的问题:

  1. mholt/archives实现了对加密rar7z的支持,但internal/archive/archives/utils.go#L28这种用法似乎是错误的。
  2. zip 压缩文件的文件名存在编码问题,现使用saintfish/chardet推测文件名的编码,但对于文件名较短的情况效果不尽如人意,可能删除一些不常用的编码效果会更好。

内容:

  1. 新增以下九个 API:
    1. /api/public/archive_extensions:获取后端支持的压缩文件后缀。
    2. /api/fs/archive/meta:获取压缩文件的元信息,包括是否被加密、注释内容、和可选的完整目录结构。
    3. /api/fs/archive/list:当/meta没有返回完整目录结构时,用于获取压缩文件内部的目录结构。
    4. /api/fs/archive/decompress:解压。
    5. /ad/*path?inner=<内部路径>pass=<解压密码>:通过驱动提取压缩文件内部的一个文件。
    6. /ap/*path?inner=<内部路径>pass=<解压密码>:通过驱动本地代理上述文件。
    7. /ae/*path?inner=<内部路径>pass=<解压密码>:通过内部工具提取压缩文件内部的一个文件。
    8. /api/task/decompress/*:管理下载和解压压缩文件任务。
    9. /api/task/decompress_upload/*:管理上传解压后文件的任务。
  2. 新增decompressdecompress_upload两种任务,分别用于下载待解压的压缩文件和上传解压后的内容,支持使用 HttpRange 访问压缩文件的部分内容以实现不下载压缩文件的完整内容就解压其一部分。
  3. 新增以下驱动接口,用于管理压缩文件:
    1. GetArchiveMeta(ctx context.Context, obj model.Obj, args model.ArchiveArgs) (model.ArchiveMeta, error):获取压缩文件的元信息。
    2. ListArchive(ctx context.Context, obj model.Obj, args model.ArchiveInnerArgs) ([]model.Obj, error):列举压缩文件内部的目录结构。
    3. Extract(ctx context.Context, obj model.Obj, args model.ArchiveInnerArgs) (*model.Link, error):提取压缩文件内的文件。
    4. ArchiveGet(ctx context.Context, obj model.Obj, args model.ArchiveInnerArgs) (model.Obj, error):直接获取压缩文件内的文件信息。
    5. ArchiveDecompress(ctx context.Context, srcObj, dstDir model.Obj, args model.ArchiveDecompressArgs) ([]model.Obj, error):解压。
  4. 新增压缩文件管理模块internal/archive,用于在驱动没有实现压缩文件相关接口,或进行跨驱动解压时,使用内部工具操作压缩文件,管理工具接口:
    1. AcceptedExtensions() []string:返回工具支持的压缩文件后缀。
    2. GetMeta(ss *stream.SeekableStream, args model.ArchiveArgs) (model.ArchiveMeta, error):获取元信息。
    3. List(ss *stream.SeekableStream, args model.ArchiveInnerArgs) ([]model.Obj, error):列举目录结构。
    4. Extract(ss *stream.SeekableStream, args model.ArchiveInnerArgs) (io.ReadCloser, int64, error):提取。
    5. Decompress(ss *stream.SeekableStream, outputPath string, args model.ArchiveInnerArgs, up model.UpdateProgress) error:解压(到临时目录)。
  5. 基于以下库支持了以下类型的压缩文件:
    1. mholt/archives.br, .bz2, .gz, .lz4, .lz, .sz, .s2, .xz, .zz, .zst, .tar, .rar, .7z
    2. yeka/zip.zipmholt/archives不支持加密的.zip,所以用了这个)
    3. kdomanski/iso9660.iso
  6. 解压和预览功能的前端部分。
  7. 新增两个用户权限:读取压缩文件、解压。
  8. 一些与该功能关系不大的小修改:
    1. 新增stream.SStreamReadAtSeeker用于将stream.SeekableStream封装为io.ReaderAtio.Seeker使用。
    2. 新增stream.ReaderUpdatingProgress用于在读取一个流的同时基于driver.UpdateProgress(现移至model包)提交进度。
    3. model.FileStreamer新增CacheFullInTempFileAndUpdateProgress(up UpdateProgress)用于在将流缓存到本地的过程中提交进度。
    4. 新增命令kill,与原stop指令相同,stop指令终止服务端进程的方式从发送SIGKILL信号改为发送SIGTERM信号。这一改动是因为decompress_upload为了实现重试,会在失败的时候不删除本地的临时文件,直到任务本身被删除时才同时删除临时文件,为了避免服务端退出时还有剩余失败的decompress_upload任务,需要在退出时将它们删除,这就必须保证stop指令的效果是在cmd/server.go#L160处让程序继续执行,而不是直接杀死进程。
  9. 阿里云盘Open又拍云存储本地存储SFTP上测试了不缓存模式的解压,在又拍云存储本地存储上通过了预览相关各项功能的测试,在百度网盘阿里云盘Open上没有通过预览相关功能的测试

Closes #7573
Closes #5114

@KirCute
Copy link
Contributor Author
KirCute commented Jan 13, 2025

Front-end part AlistGo/alist-web#234

@KirCute KirCute marked this pull request as ready for review January 16, 2025 19:36
@xhofe xhofe merged commit bb40e2e into AlistGo:main Jan 18, 2025
6 checks passed
@Clouddark75
Copy link
Clouddark75 commented Jan 18, 2025

Cool. Just testing the latest beta, and I don't see the archive support anywhere (it will make my backup process slot better). Offline downloading only shows simple HTTP; it does not matter if it's a supported host or not. I don't see any new options to configure or change things. What am I missing? 🤔

@KirCute
Copy link
Contributor Author
KirCute commented Jan 19, 2025

Cool. Just testing the latest beta, and I don't see the archive support anywhere (it will make my backup process slot better). Offline downloading only shows simple HTTP; it does not matter if it's a supported host or not. I don't see any new options to configure or change things. What am I missing? 🤔

Check whether there are two additional permissions in the user management. If so, you need to enable them to use this new feature. If there are no new permissions, you may have encountered the familiar issue of inconsistent front-end and back-end versions.

@KirCute
Copy link
Contributor Author
KirCute commented Jan 19, 2025

Cool. Just testing the latest beta, and I don't see the archive support anywhere (it will make my backup process slot better). Offline downloading only shows simple HTTP; it does not matter if it's a supported host or not. I don't see any new options to configure or change things. What am I missing? 🤔

After enabling the two permissions, you can see the new "Decompress" option by right-clicking the compressed file. Clicking on the compressed file will reveal the new preview method.

@Clouddark75
Copy link
Clouddark75 commented Jan 19, 2025 via email

@KirCute
Copy link
Contributor Author
KirCute commented Jan 19, 2025

Same problem, then. Hahaha. But this time those new options aren't there, at all. Can you tell me the code for the db to have them all enabled? El sáb., 18 de ene. de 2025 11:02 PM, KirCute_ECT @.> escribió:

Cool. Just testing the latest beta, and I don't see the archive support anywhere (it will make my backup process slot better). Offline downloading only shows simple HTTP; it does not matter if it's a supported host or not. I don't see any new options to configure or change things. What am I missing? 🤔 After enabling the two permissions, you can see the new "Decompress" option by right-clicking the compressed file. Clicking on the compressed file will reveal the new preview method. — Reply to this email directly, view it on GitHub <#7817 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AUE6MR7AEGLSGEKJPHCONMD2LMIVRAVCNFSM6AAAAABVDN47BSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMMBQGUZTEMRZGY . You are receiving this because you commented.Message ID: @.
>

You can change the permission to 16383. But simply enabling permissions in the database may not solve the problem as it did last time, because this new feature also requires support from new front-end code.

@KirCute KirCute deleted the feat/archive-manage branch January 19, 2025 12:31
@Clouddark75
Copy link

You're right, nothing shows up. I'm using the last Docker beta composer; I can't see any options under the user panel. So I tried the Windows version and set up a local drive on one of my SSD with some Zip, RAR, 7z, GZ, TAR, and Z01, part.RAR, 7z.001... part files.

I can preview the files for ZIP, TAR, 7z, and RAR, but when I try to uncompress RAR files, it fails with an error: "rardecode: decoded file too short."

If I try to download the RAR from the preview, the download just keeps restarting when it's finished redownloading the file again and again until you cancel it or stop it: "failed to close file streamer, closing closer: invalid argument, Error #1: rardecode: decoded file too short."

GZ files cannot be opened or decompressed for some reason.

Part archive files (z0, 7z001, part.rar) RAR part give an error: "archive met: extract: rardecode: filename required for multi-volume archive: extract: rardecode: filename required for multi-volume archive." The others can't even be opened or decompressed.

@KirCute
Copy link
Contributor Author
KirCute commented Jan 20, 2025

You're right, nothing shows up. I'm using the last Docker beta composer; I can't see any options under the user panel. So I tried the Windows version and set up a local drive on one of my SSD with some Zip, RAR, 7z, GZ, TAR, and Z01, part.RAR, 7z.001... part files.

I can preview the files for ZIP, TAR, 7z, and RAR, but when I try to uncompress RAR files, it fails with an error: "rardecode: decoded file too short."

If I try to download the RAR from the preview, the download just keeps restarting when it's finished redownloading the file again and again until you cancel it or stop it: "failed to close file streamer, closing closer: invalid argument, Error #1: rardecode: decoded file too short."

GZ files cannot be opened or decompressed for some reason.

Part archive files (z0, 7z001, part.rar) RAR part give an error: "archive met: extract: rardecode: filename required for multi-volume archive: extract: rardecode: filename required for multi-volume archive." The others can't even be opened or decompressed.

There is no support for part archive files currently. As for the issues with RAR and GZ, I will look into them later.

@Clouddark75
Copy link

You're right, nothing shows up. I'm using the last Docker beta composer; I can't see any options under the user panel. So I tried the Windows version and set up a local drive on one of my SSD with some Zip, RAR, 7z, GZ, TAR, and Z01, part.RAR, 7z.001... part files.
I can preview the files for ZIP, TAR, 7z, and RAR, but when I try to uncompress RAR files, it fails with an error: "rardecode: decoded file too short."
If I try to download the RAR from the preview, the download just keeps restarting when it's finished redownloading the file again and again until you cancel it or stop it: "failed to close file streamer, closing closer: invalid argument, Error #1: rardecode: decoded file too short."
GZ files cannot be opened or decompressed for some reason.
Part archive files (z0, 7z001, part.rar) RAR part give an error: "archive met: extract: rardecode: filename required for multi-volume archive: extract: rardecode: filename required for multi-volume archive." The others can't even be opened or decompressed.

There is no support for part archive files currently. As for the issues with RAR and GZ, I will look into them later.

Now that Docker beta got fixed for the back-end, I'm testing again with it.

Hmm, rardecode should support multi-part compressed files, at least that's what I saw on the code:
https://github.com/nwaples/rardecode

@KirCute
Copy link
Contributor Author
KirCute commented Jan 25, 2025

Hmm, rardecode should support multi-part compressed files, at least that's what I saw on the code: https://github.com/nwaples/rardecode

I know rardecode supports multi-part compressed files. Just I haven't implemented it yet.

@TomyJan
Copy link
TomyJan commented Jan 29, 2025

Would it be possible to add a setting to control the preview of zip files, like the preview of other file formats? I don't want to allow previewing of zip files

@KirCute
Copy link
Contributor Author
KirCute commented Jan 29, 2025

Would it be possible to add a setting to control the preview of zip files, like the preview of other file formats? I don't want to allow previewing of zip files

If you wish to disable the preview of any type of archive files, you can achieve this by revoking the user's "read archives" permission. If you want to disable only the preview of zip files, could I know the reason for this requirement before discussing the solution? Allowing the preview of archive files seems to have no effect on downloads and custom file preview methods.

@TomyJan
Copy link
TomyJan commented Jan 29, 2025

Would it be possible to add a setting to control the preview of zip files, like the preview of other file formats? I don't want to allow previewing of zip files

If you wish to disable the preview of any type of archive files, you can achieve this by revoking the user's "read archives" permission. If you want to disable only the preview of zip files, could I know the reason for this requirement before discussing the solution? Allowing the preview of archive files seems to have no effect on downloads and custom file preview methods.

I didn't realize I could control this in the user management, but now I can. It's a bit counter-intuitive, so I think it's better to have it in the preview settings.

long2005a1 added a commit to long2005a1/Long-Cloud that referenced this pull request Jan 30, 2025
@xyz7z
Copy link
xyz7z commented Feb 27, 2025

有谁知道怎么将离线下载的文件,先添加到解压任务吗?
下载的原始文件是压缩包,只想保留压缩包内的其中的部分文件,然后再上传到目标存储桶。这是个很常见的需求

@KirCute
Copy link
Contributor Author
KirCute commented Feb 27, 2025

载的原始文件是压缩包,只想保留压缩包内的其中的部分文件,然后再

你用压缩包预览功能应该就能直接把压缩包里的文件复制到另一个位置,除非驱动对range请求的支持有限

@xyz7z
Copy link
xyz7z commented Feb 27, 2025

载的原始文件是压缩包,只想保留压缩包内的其中的部分文件,然后再

你用压缩包预览功能应该就能直接把压缩包里的文件复制到另一个位置,除非驱动对range请求的支持有限

云服务器部署的alist,从下载源离线下载到alist的网盘存储。只能在alist后台的任务/离线下载看到下载的文件,似乎没有预览的选项。

@KirCute
Copy link
Contributor Author
KirCute commented Feb 27, 2025

云服务器部署的alist,从下载源离线下载到alist的网盘存储。只能在alist后台的任务/离线下载看到下载的文件,似乎没有预览的选项。

你在目录里直接点击压缩文件,预览方式选择Archive Preview(没有就是用户没开读取压缩文件权限),对里面的文件右键,提取到

@xyz7z
Copy link
xyz7z commented Feb 27, 2025

云服务器部署的alist,从下载源离线下载到alist的网盘存储。只能在alist后台的任务/离线下载看到下载的文件,似乎没有预览的选项。

你在目录里直接点击压缩文件,预览方式选择Archive Preview(没有就是用户没开读取压缩文件权限),对里面的文件右键,提取到

直接离线下载到添加的网盘,在上传前手动停止的,文件在云服务器不在本地,/opt/alist/data/temp/aria2/
似乎应该先给alist新建一个在云服务器上的本地存储,然后离线下载到该目录解压处理后再上传,Thanks。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

希望可以增加在线解压的功能 能否支持zip、tar等压缩包的在线内容预览
5 participants
0