8000 "make dist" ReaR tarball misses usr/share/rear/skel/default/root/.vimrc · Issue #3368 · rear/rear · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

"make dist" ReaR tarball misses usr/share/rear/skel/default/root/.vimrc #3368

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
jsmeix opened this issue Dec 18, 2024 · 37 comments · May be fixed by #3371
Open

"make dist" ReaR tarball misses usr/share/rear/skel/default/root/.vimrc #3368

jsmeix opened this issue Dec 18, 2024 · 37 comments · May be fixed by #3371
Labels
minor bug An alternative or workaround exists
Milestone
ReaR 3.0

Comments

@jsmeix
Copy link
Member
jsmeix commented Dec 18, 2024

See
#3238 (comment)
excerpt

# make dist OFFICIAL=1 && echo OK || echo FAILED
...
OK

# git ls-files . --ignored --exclude-standard --others
dist/rear-2.7.tar.gz

usr/share/rear/skel/default/root/.vimrc
is not in dist/rear-2.7.tar.gz
but this file is needed because
#3151
cf.
#3238 (comment)
@jsmeix jsmeix added the bug The code does not do what it is meant to do label Dec 18, 2024
@jsmeix jsmeix added this to the ReaR 2.8 milestone Dec 18, 2024
@jsmeix jsmeix added minor bug An alternative or workaround exists and removed bug The code does not do what it is meant to do labels Dec 18, 2024
@jsmeix
Copy link
Member Author
jsmeix commented Dec 18, 2024 •
edited
Loading

@rear/contributors
could you please have a look here?

I think this should be fixed for ReaR 2.8
because otherwise the minor issue
#3151
would not be fixed in ReaR 2.8 packages
that are built from our dist/rear-2.8.tar.gz

@schlomo
Copy link
Member
schlomo commented Dec 18, 2024

Let's just add an empty .vimrc to skel/default with a comment line explaining why it is there. Or we can add it to /etc/vimrc same place? Wouldn't that have the same effect and it would be more obvious?

Users who want to carry along their own VIM config can still use COPY_AS_IS to do so and it will override our empty default

@jsmeix
Copy link
Member Author
jsmeix commented Dec 18, 2024 •
edited
Loading

I have the same .vimrc

" An empty vimrc file to avoid that 'vi' inside the ReaR recovery system shows
" Failed to source defaults.vim - Press ENTER or type command to continue
" see https://github.com/rear/rear/issues/3151

as copies now in
usr/share/rear/skel/default/etc/.vimrc
usr/share/rear/skel/default/.vimrc
usr/share/rear/skel/default/root/.vimrc
and did

make dist OFFICIAL=1

and all three are missing in dist/rear-2.7.tar.gz

BUT
when I make another copy as
usr/share/rear/skel/default/root/vimrc
(i.e. without leading dot)
then it gets included in dist/rear-2.7.tar.gz
so it seems dot-files are ignored.

@pcahyna
Copy link
Member
pcahyna commented Dec 18, 2024

usr/share/rear/skel/default/root/.vimrc
is not in dist/rear-2.7.tar.gz

Do you know why?

@jsmeix
Copy link
Member Author
jsmeix commented Dec 18, 2024 •
edited
Loading

If I knew why I would "just fix" it, cf.
#3238 (comment)

@pcahyna
Copy link
Member
pcahyna commented Dec 18, 2024

I guess it is because of --exclude=".??*" passed to tar, which originates in 56959b5 by @schlomo .

@pcahyna
Copy link
Member
pcahyna commented Dec 18, 2024

Unfortunately, if we remove this, I suspect we will get unwanted stuff in the build, like .github, .fmf and .shellcheckrc.

@schlomo
Copy link
Member
schlomo commented Dec 18, 2024 •
edited
Loading

@jsmeix the root cause is that we exlcude dot files when building the tar in

rear/Makefile

8000 Line 90 in 3d17f43

tarparams = $(shell test -f .gitignore && echo --exclude-from=.gitignore --exclude=.gitignore) --exclude=".??*" $(DIST_CONTENT)

Therefore we have the ability to use an archive as skel, see https://github.com/rear/rear/blob/master/usr/share/rear/skel/Debian/default.tar.gz

But for this specific problem I'd prefer to solve the vim problem via shipping an /etc/vimrc in the hope that such a file would make vim happy without a /root/.vimrc as dot files are often not noticed by users. If possible.

Or we use code to create the dotfile in the rescue system.

@jsmeix
Copy link
Member Author
jsmeix commented Dec 18, 2024 •
edited
Loading

Ah!
/etc/vimrc is not a dot-file - I missed that.
I will try to fix it this way for ReaR 2.8...

@pcahyna
Copy link
Member
pcahyna commented Dec 18, 2024

If I remove --exclude=".??*" from the Makefile, I get this difference in the resulting tar :

--- unpatched.list      2024-12-18 15:26:29.207157564 +0100
+++ patched.list        2024-12-18 15:25:45.518053248 +0100
@@ -462,6 +462,7 @@
 rear-2.7/usr/share/rear/lib/output-functions.sh
 rear-2.7/usr/share/rear/lib/config-functions.sh
 rear-2.7/usr/share/rear/lib/mkbackup-workflow.sh
+rear-2.7/usr/share/rear/lib/.shellcheckrc
 rear-2.7/usr/share/rear/lib/udev-workflow.sh
 rear-2.7/usr/share/rear/lib/mail-functions.sh
 rear-2.7/usr/share/rear/lib/layoutonly-workflow.sh
@@ -579,6 +580,7 @@
 rear-2.7/usr/share/rear/skel/default/mnt/cdrom/
 rear-2.7/usr/share/rear/skel/default/mnt/local/
 rear-2.7/usr/share/rear/skel/default/root/
+rear-2.7/usr/share/rear/skel/default/root/.vimrc
 rear-2.7/usr/share/rear/skel/default/etc/
 rear-2.7/usr/share/rear/skel/default/etc/lilo.conf
 rear-2.7/usr/share/rear/skel/default/etc/syslog-ng.conf
@@ -1422,6 +1424,7 @@
 rear-2.7/usr/share/rear/restore/NSR/default/400_restore_with_nsr.sh
 rear-2.7/usr/share/rear/restore/SUSE_LINUX/
 rear-2.7/usr/share/rear/restore/SUSE_LINUX/910_create_missing_directories.sh
+rear-2.7/usr/share/rear/.shellcheckrc
 rear-2.7/usr/share/rear/rescue/
 rear-2.7/usr/share/rear/rescue/NETFS/
 rear-2.7/usr/share/rear/rescue/NETFS/default/

looks good. @schlomo what do you mean by "Therefore we have the ability to use an archive as skel"? How is it related to excludign dot files from the dist archive?

@schlomo
Copy link
Member
schlomo commented Dec 18, 2024

@pcahyna

rear/usr/share/rear/rescue/default/010_merge_skeletons.sh

Lines 30 to 34 in 3d17f43

8000
# If $skel_dir.tar.gz exists (e.g. usr/share/rear/skel/Debian/default.tar.gz) extract it:
if test -s "$skel_dir.tar.gz" ; then
Log "Extracting '$skel_path/$skel_dir.tar.gz' into $ROOTFS_DIR"
tar -C $ROOTFS_DIR -xzf "$skel_dir.tar.gz" || Error "Failed to extract '$skel_path/$skel_dir.tar.gz' into $ROOTFS_DIR"
fi

If the .vimrc dotfile would be in such a skel archive then it wouldn't be filtered out in the make dist process. I had added this feature a long time ago as a way to add arbitrary stuff via the skel process that would cause problems with git. For example, a block or character device probably wouldn't work either.

But, I'd rather use a non-dotfile to solve a problem as this is much easier to see and understand than having to look into an archive hidden somewhere deep in our source tree. That is why I was asking if /etc/vimrc would also solve the problem.

@schlomo
Copy link
Member
schlomo commented Dec 18, 2024

@pcahyna and I'd rather exclude the development relevant files like .shellcheckrc from our dist archive.

In any case we have slightly different behavior of the make dist process depending on if it runs from a git checkout or not, as it reacts to the presence of .gitignore files to also exclude everything mentioned there, which covers for example a var directory that is created if you happen to run ./usr/sbin/rear from the git checkout prior to running make dist (this is the original reason for me to include that feature of handling .gitignore properly for tar).

HTH for context

@jsmeix
Copy link
Member Author
jsmeix commented Dec 18, 2024

Unfortunately it neither works with /etc/vimrc
nor with /etc/.vimrc in the ReaR recovery system.

@schlomo
Copy link
Member
schlomo commented Dec 18, 2024

@jsmeix vim --version suggests to use /usr/share/vim/vimrc and maybe we should simply add this folder to our default COPY_AS_IS to simply include the original system's vim configuration. If vim is not installed, then nothing happens. And if it is installed, then it will work with the defaults.

image

So it seems like /etc/vim/vimrc is not standard but enabled by Debian via a symlink. Maybe adding both paths (/etc/vim and /usr/share/vim) to COPY_AS_IS would be a "correct" way to resolve this?

@pcahyna
Copy link
Member
pcahyna commented Dec 18, 2024

If the .vimrc dotfile would be in such a skel archive then it wouldn't be filtered out in the make dist process.

Not sure if I get it. What if there is a non-dotfile in the archive? It would not get filtered either. What makes the .??* pattern special?

@schlomo
Copy link
Member
schlomo commented Dec 18, 2024

the skel archive is a means to transport anything whatsoever into the rescue system. this of course also works for non-dotfiles. It is only required for special or weird stuff.

The .??* pattern is not special, it simply ensures to not match on dotfiles with less than 2 characters after the dot which is a common way to not match on . and ... Maybe a shorter pattern like .* would have sufficed, I don't remember if I tried that or not.

@pcahyna
Copy link
Member
pcahyna commented Dec 18, 2024

But what's the problem with dotfiles in the first place? Is it related to the skel archive or not (if it also works for non-dotfiles)?

@jsmeix
Copy link
Member Author
jsmeix commented Dec 18, 2024

Unfortunately it neither works with /etc/vim/vimrc
nor with /etc/vim/.vimrc in the ReaR recovery system.

@schlomo
Copy link
Member
schlomo commented Dec 18, 2024

I think the only "problem" is, that we filter out dotfiles in the make dist process.

BTW, I just noticed in

rear/usr/share/rear/build/GNU/Linux/130_create_dotfiles.sh

Line 32 in 486b76b

if get_path vi 1>&2 ; then

we check for vi not vim but talk about including the config for vim. Maybe vi (the not-improved VI) has a different config or works without?

I now understand that @jsmeix tried to fix the missing VIM rc file via 700727b#diff-86dae0514a9be4af4b9a40cd2670d62f8d815757a620cdebd29038458af0c2d6 and that we now realize that it didn't work for package building.

@jsmeix to quickly resolve this I'd suggest to also create the .vimrc (with the dot) in

rear/usr/share/rear/build/GNU/Linux/130_create_dotfiles.sh

Line 32 in 486b76b

if get_path vi 1>&2 ; then
or, to try creating a /usr/share/vim/vimrc file instead. AFAIK /etc/vim/vimrc is actually not the path that vim looks for. vim --version will show you the actual paths it checks.

@jsmeix
Copy link
Member Author
jsmeix commented Dec 18, 2024

In my ReaR recovery sytem made on SLES 15 SP6:

RESCUE localhost:~ # vim --version
-bash: vim: command not found

RESCUE localhost:~ # vi --version
VIM - Vi IMproved 9.1 (2024 Jan 02)
...
   system vimrc file: "/etc/vimrc"
     user vimrc file: "$HOME/.vimrc"
 2nd user vimrc file: "~/.vim/vimrc"
      user exrc file: "$HOME/.exrc"
       defaults file: "$VIMRUNTIME/defaults.vim"
  fall-back for $VIM: "/usr/share/vim"

No, it just lies because /etc/vimrc does not work
at least not in my ReaR recovery system :-(

@pcahyna
Copy link
Member
pcahyna commented Dec 18, 2024

I think the only "problem" is, that we filter out dotfiles in the make dist process.

Let's stop filtering them, then?

diff --git a/Makefile b/Makefile
index 731c05af7..54f5d4ee1 100644
--- a/Makefile
+++ b/Makefile
@@ -87,7 +87,7 @@ RUNASUSER :=
 endif
 
 # .gitignore is optional, avoid tar errors if it does not exist, e.g. in a dist archive
-tarparams = $(shell test -f .gitignore && echo --exclude-from=.gitignore --exclude=.gitignore) --exclude=".??*" $(DIST_CONTENT)
+tarparams = $(shell test -f .gitignore && echo --exclude-from=.gitignore --exclude=.gitignore) $(DIST_CONTENT)
 
 DIST_FILES := $(shell tar -cv -f /dev/null $(tarparams))

@jsmeix
Copy link
Member Author
jsmeix commented Dec 18, 2024 •
edited
Loading

@pcahyna
my personal feeling is that currently we are too near
the ReaR 2.8 release so that I feel rather uncomfortable
with changing what "make dist OFFICIAL=1" does
ONLY
because of this minor issue here, in particular because
#3151
is only an annoying message so 'vi' works inside
the ReaR recovery system - and how often is 'vi'
used inside ReaR recovery system in practice?

@pcahyna
Copy link
Member
pcahyna commented Dec 18, 2024

Therefore we have the ability to use an archive as skel, see https://github.com/rear/rear/blob/master/usr/share/rear/skel/Debian/default.tar.gz

By the way (not related to the original problem anymore), having a gzipped archive in our source tree is at best an antipattern and at worst a way to sneakily inject malware (who knows what is inside? see the xz attack), so I hope we will get rid of it.

@jsmeix
Copy link
Member Author
jsmeix commented Dec 18, 2024 •
edited
Loading

@pcahyna
I agree that having a gzipped archive in our source tree
is bad but I don't agree to get rid of that functionality
where a user could add a gzipped archive as neded in his
skel tree to get special things into his ReaR recovery system
as he needs it - think about a complex piece of software
where COPY_AS_IS, REQUIRED_PROGS, and LIBS could be
needlessly complicaded for the usre compared to
"simply pack what is needed into an archive".

But I never used that functionality myself
so I cannot tell if it is actually useful in practice.

@pcahyna
Copy link
Member
pcahyna commented Dec 18, 2024

I meant, get rid of the archive, not of the functionality.

@schlomo
Copy link
Member
schlomo commented Dec 18, 2024

While I understand and agree with your thoughts about having an archive in the source tree I'd like to keep it there to keep testing the skel archive feature.

I'm fine with removing the dotfile exclude from the tar creation process because in

rear/Makefile

Line 12 in cc1e9e4

DIST_CONTENT = COPYING doc etc MAINTAINERS Makefile packaging README.md tests tools usr
we have an explicit include of the files relevant for ReaR. That already excludes top-level dotfiles and development stuff in order to create a "clean" dist archive. For cleaning up we should then also remove all the other development-related dotfiles (shellcheckrc, gitignore) from the source tree and have those only on the top level.

For actually solving the vimrc problem, the SLES example maybe confirms my suggestion that we should add common vimrc locations to COPY_AS_IS.

Or we don't deal with it for 2.8 - also totally fine for me as I'm using nano for small editing needs 😁

@jsmeix
Copy link
Member Author
jsmeix commented Dec 18, 2024 •
edited
Loading

Regarding usr/share/rear/skel/Debian/default.tar.gz

# tar -tzvf usr/share/rear/skel/Debian/default.tar.gz
-rw-r--r-- root/root 1059 2007-10-05 16:06 lib/linux-sound-base/noOSS.modprobe.conf

It doesn't look as if this single file
could not be a regular file in skel?

# git log -p --follow usr/share/rear/skel/Debian/default.tar.gz

tells that it originated since "Switch to trunk model"
which means it is there "since the beginning of time"
i.e. beyond what we can see in retrospect in git

@schlomo
Copy link
Member
schlomo commented Dec 18, 2024

Yes, I just put some BS into that archive to see that the mechanism works.

@jsmeix
Copy link
Member Author
jsmeix commented Dec 18, 2024 •
edited
Loading

From my current (Wednesday evening) point of view
I would prefer to "don't deal with it for 2.8"
and deal with it properly for ReaR 3.0.

I assume an appropriate

COPY_AS_IS+=( /path/to/some/vimrc )

is a sufficient workaround in ReaR 2.8
for users who want to avoid the annoying message.

Alternatively @schlomo @pcahyna if you work longer today
and find a solution that you tested that it works for you
the please make a pull request (we may still postpone
a pull request to ReaR 3.0 if we think it is better/safer
to "don't deal with it for 2.8").

@jsmeix
Copy link
Member Author
jsmeix commented Dec 18, 2024

@schlomo
does your
#3368 (comment)
mean the current usr/share/rear/skel/Debian/default.tar.gz
content lib/linux-sound-base/noOSS.modprobe.conf
is only a test and that noOSS.modprobe.conf is not
actually needed in the ReaR recovery system for Debian?

@schlomo
Copy link
Member
schlomo commented Dec 18, 2024

@schlomo does your #3368 (comment) mean the current usr/share/rear/skel/Debian/default.tar.gz content lib/linux-sound-base/noOSS.modprobe.conf is only a test and that noOSS.modprobe.conf is not actually needed in the ReaR recovery system for Debian?

The file contains the following:

blacklist ac97
blacklist ac97_codec
blacklist ac97_plugin_ad1980
blacklist ad1848
blacklist ad1889
blacklist adlib_card
blacklist aedsp16
blacklist ali5455
blacklist btaudio
blacklist cmpci
blacklist cs4232
blacklist cs4281
blacklist cs461x
blacklist cs46xx
blacklist emu10k1
blacklist es1370
blacklist es1371
blacklist esssolo1
blacklist forte
blacklist gus
blacklist i810_audio
blacklist kahlua
blacklist mad16
blacklist maestro
blacklist maestro3
blacklist maui
blacklist mpu401
blacklist nm256_audio
blacklist opl3
blacklist opl3sa
blacklist opl3sa2
blacklist pas2
blacklist pss
blacklist rme96xx
blacklist sb
blacklist sb_lib
blacklist sgalaxy
blacklist sonicvibes
blacklist sound
blacklist sscape
blacklist trident
blacklist trix
blacklist uart401
blacklist uart6850
blacklist via82cxxx_audio
blacklist v_midi
blacklist wavefront
blacklist ymfpci
blacklist ac97_plugin_wm97xx
blacklist ad1816
blacklist audio
blacklist awe_wave
blacklist dmasound_core
blacklist dmasound_pmac
blacklist 
F438
harmony
blacklist sequencer
blacklist soundcard
blacklist usb-midi

I would indeed assume that this is long obsolete, at least https://en.wikipedia.org/wiki/Open_Sound_System suggests that this doesn't apply any more.

I'm also totally fine with just removing this archive and to fix problems if somebody complains about a problem here.

@pcahyna
Copy link
Member
pcahyna commented Dec 18, 2024

my personal feeling is that currently we are too near
the ReaR 2.8 release so that I feel rather uncomfortable
with changing what "make dist OFFICIAL=1" does
ONLY
because of this minor issue here

@jsmeix this is an entirely reasonable approach. OTOH, when changing the Makefile, it is easy to check what is being introduced by the change:

make OFFICIAL=1 dist
tar tzf dist/rear-2.7.tar.gz > unpatched.list
# here change the Makefile...

make OFFICIAL=1 dist
tar tzf dist/rear-2.7.tar.gz > patched.list
diff -u unpatched.list patched.list

so I will leave it up to you.

@schlomo
Copy link
Member
schlomo commented Dec 18, 2024

Yes, I also prefer to get a release now, TBH. So please what you deem right @jsmeix

@pcahyna
Copy link
Member
pcahyna commented Dec 18, 2024

if you work longer today
and find a solution that you tested that it works for you
the please make a pull request (we may still postpone
a pull request to ReaR 3.0 if we think it is better/safer
to "don't deal with it for 2.8").

I will try to make a PR out of the proposed Makefile change.

@pcahyna
Copy link
Member
pcahyna commented Dec 18, 2024 •
edited
Loading

While I understand and agree with your thoughts about having an archive in the source tree I'd like to keep it there to keep testing the skel archive feature.

Testing is useful, but I would very much prefer to move the test to https://github.com/rear/rear-integration-tests/ where it is clearly separate from ReaR sources.

@jsmeix jsmeix modified the milestones: ReaR 2.8, ReaR 3.0 Dec 19, 2024
@jsmeix
Copy link
Member Author
jsmeix commented Dec 19, 2024

I postpone this issue to ReaR 3.0 because
I like to focus on the actual ReaR 2.8 release now
because I have to leave today already at 14:00

pcahyna added a commit to pcahyna/rear that referenced this issue Dec 19, 2024
@pcahyna
Do not remove dotfiles from the release archive
35acee9 
Avoids dropping .vimrc from /root of the rescue ramdisk.

This adds .shellcheckrc to the release tarballs, so package builds using
static analyzers (OpenScanHub) will know what ShellCheck warnings to
ignore.

Closes rear#3368.
@pcahyna pcahyna linked a pull request Dec 19, 2024 that will close this issue
Open
@gdha
Copy link
Member
gdha commented Feb 24, 2025

Just be aware we could always use script build/GNU/Linux/130_create_dotfiles.sh to add nasty dot-files ;-)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
minor bug An alternative or workaround exists
Projects
None yet
Milestone
ReaR 3.0
Development

Successfully merging a pull request may close this issue.

Do not remove dotfiles from the release archive pcahyna/rear
4 participants
@schlomo @gdha @jsmeix @pcahyna
0