8000 Investigating the reason behind "Permission denied when reading from process 2132183" · Issue #323 · rbspy/rbspy · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Investigating the reason behind "Permission denied when reading from process 2132183" #323

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
jarthod opened this issue May 12, 2021 · 6 comments

Comments

@jarthod
Copy link
jarthod commented May 12, 2021

When I try to simply record a small ruby program started by rbspy it fails with the following error message

> ./target/debug/rbspy record -- ruby ci/ruby-programs/recurse-sleep.rb 
Press Ctrl+C to stop
Wrote raw data to /home/bigbourin/.cache/rbspy/records/rbspy-2021-05-10-0JKM71znjH.raw.gz
Writing formatted output to /home/bigbourin/.cache/rbspy/records/rbspy-2021-05-10-1Wa9J4CugF.flamegraph.svg
Warning: no profile samples were collected
Error. Causes: 
- Couldn't get ruby version: Couldn't determine Ruby version

Caused by:
    Permission denied when reading from process 2132183. If you're not running as root, try again with sudo. If you're using Docker, try passing `--cap-add=SYS_PTRACE` to `docker run`
Couldn't get ruby version: Couldn't determine Ruby version

Caused by:
    Permission denied when reading from process 2132183. If you're not running as root, try again with sudo. If you're using Docker, try passing `--cap-add=SYS_PTRACE` to `docker run`

We started investigating this in #268 and moved to a new issue now that the error message is more precise.

OS:

> sudo lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 20.10
Release:	20.10
Codename:	groovy
> uname -a
Linux desktop 5.8.0-49-generic #55-Ubuntu SMP Wed Mar 24 14:45:45 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

To already answer the last comment from @acj:

I get very similar output for those commands in my VM. Given the error message, I think it's having trouble opening (or reading) /proc//maps for the ruby process. Not sure why. If you run ruby recurse-sleep.rb from your shell and find its PID, can you successfully cat its maps file?

Indeed I cannot:

> ruby ci/ruby-programs/recurse-sleep.rb &
[1] 46374
> cat /proc/46374/maps
cat: /proc/46374/maps: Permission denied
> ls -l /proc/46374/maps
-r--r--r-- 1 root root 0 mai   12 19:30 /proc/46374/maps
 > ls -ld /proc/46374
dr-xr-xr-x 9 bigbourin bigbourin 0 mai   12 19:30 /proc/46374

I'm not sure why as the permission set is 444, the parent folder (pid folder) is 555 (and owned by the user)
But I suppose the meta files in /proc do not behave like the rest ^^
People here https://groups.google.com/g/comp.os.linux.development.system/c/RAPRvAocEtU seems to say you need some "ptrace" capability. Here it says ptrace is only possible on child process in Ubuntu 10.10+: https://en.wikipedia.org/wiki/Ptrace but this is the case when I start the ruby process through rbspy I suppose? I understand I can't attach to any process as a normal user but in this case it should be possible.

@acj
Copy link
Member
acj commented May 15, 2021

Thanks. It's helpful to know that the maps file isn't accessible. I think investigating capabilities (like cap_sys_ptrace) might be the right path here.

In my shell, cap_ptrace is present (in the middle of the "Bounding set" list):

$ capsh --print
Current: =
Bounding set =cap_chown,cap_dac_override,cap_dac_read_search,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_linux_immutable,cap_net_bind_service,cap_net_broadcast,cap_net_admin,cap_net_raw,cap_ipc_lock,cap_ipc_owner,cap_sys_module,cap_sys_rawio,cap_sys_chroot,cap_sys_ptrace,cap_sys_pacct,cap_sys_admin,cap_sys_boot,cap_sys_nice,cap_sys_resource,cap_sys_time,cap_sys_tty_config,cap_mknod,cap_lease,cap_audit_write,cap_audit_control,cap_setfcap,cap_mac_override,cap_mac_admin,cap_syslog,cap_wake_alarm,cap_block_suspend,cap_audit_read,cap_perfmon,cap_bpf
Ambient set =
Current IAB:
Securebits: 00/0x0/1'b0
 secure-noroot: no (unlocked)
 secure-no-suid-fixup: no (unlocked)
 secure-keep-caps: no (unlocked)
 secure-no-ambient-raise: no (unlocked)
uid=1000(vagrant) euid=1000(vagrant)
gid=1000(vagrant)
groups=4(adm),24(cdrom),27(sudo),30(dip),46(plugdev),116(lxd),1000(vagrant)
Guessed mode: UNCERTAIN (0)

And if I explicitly drop the capability cap_sys_ptrace when I run rbspy, I get the same error as you:

$ sudo capsh --drop=cap_sys_ptrace -- -c 'target/debug/rbspy record -s -- ruby ci/ruby-programs/recurse-sleep.rb'
Dropping permissions: running Ruby command as user vagrant
Press Ctrl+C to stop
Wrote raw data to /root/.cache/rbspy/records/rbspy-2021-05-15-X4L6JVzgUq.raw.gz
Writing formatted output to /root/.cache/rbspy/records/rbspy-2021-05-15-e1NOp7yxES.flamegraph.svg
Warning: no profile samples were collected
Error. Causes:
- Couldn't get ruby version: Couldn't determine Ruby version

Caused by:
    Permission denied when reading from process 3139. If you're not running as root, try again with sudo. If you're using Docker, try passing `--cap-add=SYS_PTRACE` to `docker run`
Couldn't get ruby version: Couldn't determine Ruby version

Caused by:
    Permission denied when reading from process 3139. If you're not running as root, try again with sudo. If you're using Docker, try passing `--cap-add=SYS_PTRACE` to `docker run`

(Note to self -- we should improve that error message so that it covers this scenario.)

So, maybe your system is configured to drop cap_sys_ptrace by default. What do you have in /etc/security/capability.conf, if you don't mind sharing?

For comparison, here's mine (default package-provided file from Ubuntu 20.10):

$ cat /etc/security/capability.conf
#
# /etc/security/capability.conf
#
# this is a sample capability file (to be used in conjunction with
# the pam_cap.so module)
#
# In order to use this module, it must have been linked with libcap
# and thus you'll know about Linux's capability support.
# [If you don't know about libcap, the sources for it are here:
#
#   http://www.kernel.org/pub/linux/libs/security/linux-privs/
#
# .]
#
# Here are some sample lines (remove the preceding '#' if you want to
# use them

## user 'morgan' gets the CAP_SETFCAP inheritable capability (commented out!)
#cap_setfcap		morgan

## user 'luser' inherits the CAP_DAC_OVERRIDE capability (commented out!)
#cap_dac_override	luser

## 'everyone else' gets no inheritable capabilities (restrictive config)
none  *

## if there is no '*' entry, all users not explicitly mentioned will
## get all available capabilities. This is a permissive default, and
## possibly not what you want... On first reading, you might think this
## is a security problem waiting to happen, but it defaults to not being
## so in this sample file! Further, by 'get', we mean 'get in their inheritable
## set'. That is, if you look at a random process, even one run by root,
## you will see it has no inheritable capabilities (by default):
##
##   $ /sbin/capsh --decode=$(grep CapInh /proc/1/status|awk '{print $2}')
##   0000000000000000=
##
## The pam_cap module simply alters the value of this capability
## set. Including the 'none *' forces use of this module with an
## unspecified user to have their inheritable set forced to zero.
##
## Omitting the line will cause the inheritable set to be unmodified
## from what the parent process had (which is generally 0 unless the
## invoking user was bestowed with some inheritable capabilities by a
## previous invocation).

@jarthod
Copy link
Author
jarthod commented May 17, 2021
8000

Interesting, if I do capsh --print I also see the cap_sys_ptrace capability in the Bounding set:

> capsh --print
Current: =
Bounding set =cap_chown,cap_dac_override,cap_dac_read_search,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_linux_immutable,cap_net_bind_service,cap_net_broadcast,cap_net_admin,cap_net_raw,cap_ipc_lock,cap_ipc_owner,cap_sys_module,cap_sys_rawio,cap_sys_chroot,cap_sys_ptrace,cap_sys_pacct,cap_sys_admin,cap_sys_boot,cap_sys_nice,cap_sys_resource,cap_sys_time,cap_sys_tty_config,cap_mknod,cap_lease,cap_audit_write,cap_audit_control,cap_setfcap,cap_mac_override,cap_mac_admin,cap_syslog,cap_wake_alarm,cap_block_suspend,cap_audit_read,cap_perfmon,cap_bpf
Ambient set =
Current IAB: 
Securebits: 00/0x0/1'b0
 secure-noroot: no (unlocked)
 secure-no-suid-fixup: no (unlocked)
 secure-keep-caps: no (unlocked)
 secure-no-ambient-raise: no (unlocked)
uid=1000(bigbourin) euid=1000(bigbourin)
gid=1000(bigbourin)
groups=4(adm),24(cdrom),27(sudo),29(audio),30(dip),44(video),46(plugdev),105(fuse),108(lpadmin),124(sambashare),996(nordvpn),1000(bigbourin)
Guessed mode: UNCERTAIN (0)

My capabilities.conf file seems similar to yours:

> cat /etc/security/capability.conf
#
# /etc/security/capability.conf
#
# this is a sample capability file (to be used in conjunction with
# the pam_cap.so module)
#
# In order to use this module, it must have been linked with libcap
# and thus you'll know about Linux's capability support.
# [If you don't know about libcap, the sources for it are here:
#
#   http://www.kernel.org/pub/linux/libs/security/linux-privs/
#
# .]
#
# Here are some sample lines (remove the preceding '#' if you want to
# use them

## user 'morgan' gets the CAP_SETFCAP inheritable capability (commented out!)
#cap_setfcap		morgan

## user 'luser' inherits the CAP_DAC_OVERRIDE capability (commented out!)
#cap_dac_override	luser

## 'everyone else' gets no inheritable capabilities (restrictive config)
none  *

## if there is no '*' entry, all users not explicitly mentioned will
## get all available capabilities. This is a permissive default, and
## possibly not what you want... On first reading, you might think this
## is a security problem waiting to happen, but it defaults to not being
## so in this sample file! Further, by 'get', we mean 'get in their inheritable
## set'. That is, if you look at a random process, even one run by root,
## you will see it has no inheritable capabilities (by default):
##
##   $ /sbin/capsh --decode=$(grep CapInh /proc/1/status|awk '{print $2}')
##   0000000000000000=
##
## The pam_cap module simply alters the value of this capability
## set. Including the 'none *' forces use of this module with an
## unspecified user to have their inheritable set forced to zero.
##
## Omitting the line will cause the inheritable set to be unmodified
## from what the parent process had (which is generally 0 unless the
## invoking user was bestowed with some inheritable capabilities by a
## previous invocation).

I read more about capabilities (in https://blog.container-solutions.com/linux-capabilities-in-practice notably)
If I check in more details the capabilities in my bash shell I get:

> cat /proc/$BASHPID/status
Name:	bash
Umask:	0002
State:	S (sleeping)
Tgid:	427697
Ngid:	0
Pid:	427697
PPid:	10817
TracerPid:	0
Uid:	1000	1000	1000	1000
Gid:	1000	1000	1000	1000
FDSize:	256
Groups:	4 24 27 29 30 44 46 105 108 124 996 1000 
NStgid:	427697
NSpid:	427697
NSpgid:	427697
NSsid:	427697
VmPeak:	   17716 kB
VmSize:	   17716 kB
VmLck:	       0 kB
VmPin:	       0 kB
VmHWM:	    8148 kB
VmRSS:	    8148 kB
RssAnon:	    3708 kB
RssFile:	    4440 kB
RssShmem:	       0 kB
VmData:	    3584 kB
VmStk:	     132 kB
VmExe:	     708 kB
VmLib:	    2788 kB
VmPTE:	      76 kB
VmSwap:	       0 kB
HugetlbPages:	       0 kB
CoreDumping:	0
THP_enabled:	1
Threads:	1
SigQ:	3/127905
SigPnd:	0000000000000000
ShdPnd:	0000000000000000
SigBlk:	0000000000010000
SigIgn:	0000000000380004
SigCgt:	00000001cb817efb
CapInh:	0000000000000000
CapPrm:	0000000000000000
CapEff:	0000000000000000
CapBnd:	000000ffffffffff
CapAmb:	0000000000000000
NoNewPrivs:	0
Seccomp:	0
Speculation_Store_Bypass:	thread vulnerable
Cpus_allowed:	3f
Cpus_allowed_list:	0-5
Mems_allowed:	00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000001
Mems_allowed_list:	0
voluntary_ctxt_switches:	359
nonvoluntary_ctxt_switches:	28

As the article explains, the Permitted and Effective set are empty here (for user) whereas it's usually full for root. This means (if I understand correctly) that the process won't get any capabilities by default unless explicitly set for the binary. Which is obviously not the case by default right after compiling:

> getcap ./target/debug/rbspy
>

If I give the capability to the binary (with the "effective" flag because it seems rbspy does not set the capability as effective when it's just permitted), it works:

> sudo setcap 'cap_sys_ptrace+ep' ./target/debug/rbspy
> getcap ./target/debug/rbspy
./target/debug/rbspy cap_sys_ptrace=ep
> ./target/debug/rbspy record -- ruby ci/ruby-programs/recurse-sleep.rb
# works normally

(This might be a good workaround for people even if we can't figure out the deeper issue)

So in your case, do you have all capabilities in effective set already from bash? I tried to dig a bit about what other global configuration could yield to a different result from you VM but couldn't find it for the moment. I tried on a couple others Ubuntu (e.g. 18.04 server) and I also see:

CapInh:	0000000000000000
CapPrm:	0000000000000000
CapEff:	0000000000000000
CapBnd:	0000003fffffffff
CapAmb:	0000000000000000

BUT in this one I can use rbspy record without any problem...

I'm simplified my testing line to: ~/rbspy record -- ruby -e "sleep 5"
And one my server where it's working, I checked the process capabilities at runtime with:

> cat /proc/10207/status 
Name:	rbspy
Umask:	0002
State:	S (sleeping)
...
CapInh:	0000000000000000
CapPrm:	0000000000000000
CapEff:	0000000000000000
CapBnd:	0000003fffffffff
CapAmb:	0000000000000000
...

And as we can see here there's actually no capabilities in "Effective" or "Permitted", yet it still manages to run and snapshot the ruby process. I'm confused 🤔

@acj
Copy link
Member
acj commented May 20, 2021

So in your case, do you have all capabilities in effective set already from bash?

No, they're pretty similar to yours:

CapInh:	0000000000000000
CapPrm:	0000000000000000
CapEff:	0000000000000000
CapBnd:	000000ffffffffff # includes cap_sys_ptrace
CapAmb:	0000000000000000

Another possibly-interesting difference is that I can't get capabilities for files because my kernel doesn't support extended attributes for ext4:

$ getcap ./target/debug/rbspy
Failed to get capabilities of file './target/debug/rbspy' (Operation not supported)
$ cat /boot/config-$(uname -r) | grep FS_XATTR
CONFIG_REISERFS_FS_XATTR=y
CONFIG_F2FS_FS_XATTR=y
CONFIG_TMPFS_XATTR=y
CONFIG_JFFS2_FS_XATTR=y
CONFIG_UBIFS_FS_XATTR=y
CONFIG_SQUASHFS_XATTR=y
CONFIG_EROFS_FS_XATTR=y
CONFIG_AUFS_XATTR=y
CONFIG_CIFS_XATTR=y

Is that also true on your server?

(This might be a good workaround for people even if we can't figure out the deeper issue)

Nice. Yeah, I'm glad that we have a fairly simple workaround.

@jarthod
Copy link
Author
jarthod commented May 20, 2021

Mmm interesting but no it's not the case on my server (I'm also on ext4):

> getcap rbspy 
> cat /boot/config-$(uname -r) | grep FS_XATTR
CONFIG_REISERFS_FS_XATTR=y
CONFIG_F2FS_FS_XATTR=y
CONFIG_TMPFS_XATTR=y
CONFIG_JFFS2_FS_XATTR=y
CONFIG_SQUASHFS_XATTR=y
CONFIG_AUFS_XATTR=y
CONFIG_CIFS_XATTR=y

🤔

@acj
Copy link
Member
acj commented Jun 27, 2021

The Yama security module was mentioned on another thread this week and might be relevant. I don't know whether/how it interacts with file capabilities.

The sysctl settings (writable only with CAP_SYS_PTRACE) are:

0 - classic ptrace permissions: a process can PTRACE_ATTACH to any other
process running under the same uid, as long as it is dumpable (i.e.
did not transition uids, start privileged, or have called
prctl(PR_SET_DUMPABLE...) already). Similarly, PTRACE_TRACEME is
unchanged.

1 - restricted ptrace: a process must have a predefined relationship
with the inferior it wants to call PTRACE_ATTACH on. By default,
this relationship is that of only its descendants when the above
classic criteria is also met. To change the relationship, an
inferior can call prctl(PR_SET_PTRACER, debugger, ...) to declare
an allowed debugger PID to call PTRACE_ATTACH on the inferior.
Using PTRACE_TRACEME is unchanged.

2 - admin-only attach: only processes with CAP_SYS_PTRACE may use ptrace
with PTRACE_ATTACH, or through children calling PTRACE_TRACEME.

3 - no attach: no processes may use ptrace with PTRACE_ATTACH nor via
PTRACE_TRACEME. Once set, this sysctl value cannot be changed.

If I set it to 3 in my VM, I can repro the same permissions error that we've been troubleshooting:

sudo sysctl kernel.yama.ptrace_scope=3

It defaults to 1, which allows rbspy to work normally:

$ cat /etc/sysctl.d/10-ptrace.conf
...
kernel.yama.ptrace_scope = 1

@jarthod
Copy link
Author
jarthod commented Jun 28, 2021

Nice finding! I just had a look but unfortunately I so have value 1 for both machines here (the one where it's working and the one where it's not):

desktop [rbspy] (master) > sudo sysctl kernel.yama.ptrace_scope
kernel.yama.ptrace_scope = 1
desktop [rbspy] (master) > ./target/debug/rbspy record -- ruby -e "sleep 5"
Press Ctrl+C to stop
Wrote raw data to /home/bigbourin/.cache/rbspy/records/rbspy-2021-06-28-E3bOi1qd4O.raw.gz
Writing formatted output to /home/bigbourin/.cache/rbspy/records/rbspy-2021-06-28-Ova9nNyyLB.flamegraph.svg
Warning: no profile samples were collected
Error. Causes: 
- Couldn't get ruby version: Couldn't determine Ruby version

Caused by:
    Permission denied when reading from process 12044. If you're not running as root, try again with sudo. If you're using Docker, try passing `--cap-add=SYS_PTRACE` to `docker run`
Couldn't get ruby version: Couldn't determine Ruby version

Caused by:
    Permission denied when reading from process 12044. If you're not running as root, try again with sudo. If you're using Docker, try passing `--cap-add=SYS_PTRACE` to `docker run`
deploy@db3 [~] > sudo sysctl kernel.yama.ptrace_scope
kernel.yama.ptrace_scope = 1
deploy@db3 [~] > ./rbspy record -- ruby -e "sleep 5"
Press Ctrl+C to stop
... (showing frames)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants
0