8000 need soft limits for volume quota and storage node's data file · Issue #401 · coreos/torus · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content
This repository was archived by the owner on Sep 22, 2020. It is now read-only.

need soft limits for volume quota and storage node's data file #401

Open
nak3 opened this issue Nov 20, 2016 · 0 comments
Open

77AD need soft limits for volume quota and storage node's data file #401

nak3 opened this issue Nov 20, 2016 · 0 comments

Comments

@nak3
Copy link
Contributor
nak3 commented Nov 20, 2016

version

issue

Since there are no soft limits for the volume quota and torusd's data file, it is difficult to notice the out of space.

For example, I have been running postgres on k8s with torus, then they got down suddenly, even though torus pod was running like below:

$ kubectl get pod
NAME                               READY     STATUS    RESTARTS   AGE
etcd-torus                         1/1       Running   1          2d
postgres-torus-1616493773-njtjp    0/1       Error     0          1d
postgres-torus-1616493773-njtjp2   0/1       Error     0          1d
torus-snqwd                        1/1       Running   1          2d

After checked torus log, it found that storage node's diskpace was full.

$ kubectl logs torus-snqwd
...
2016-11-19 19:38:01.432298 E | storage: mfile: out of space
2016-11-19 19:38:01.455505 E | storage: mfile: out of space
2016-11-19 19:38:01.575292 E | storage: mfile: out of space

$ kubectl get event
FIRSTSEEN   LASTSEEN   COUNT     NAME                               KIND      SUBOBJECT   TYPE      REASON        SOURCE               MESSAGE
1d          32s        5598      postgres-torus-1616493773-njtjp    Pod                   Warning   FailedMount   {kubelet fed-node}   Unable to mount volumes for pod "postgres-torus-1616493773-njtjp_default(73b274b2-ae3d-11e6-9746-5254007c7544)": attach command failed, status: Failure, reason: Couldn't attach
1d          32s        5598      postgres-torus-1616493773-njtjp    Pod                   Warning   FailedSync    {kubelet fed-node}   Error syncing pod, skipping: attach command failed, status: Failure, reason: Couldn't attach
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@nak3
0