Does anyone here understand cgroups?

I have to put up with running some really quite intensive processing
on an ubuntu machine (running 20.04 LTS, on the basis of uname -a)
that I do not admin (or have any sudo privileges). You may assume I
have zero influence over those who do admin it.

Just as some context, generally each processing run - managed by a
bash script - creates as many background jobs (via &, but not nohup)
as I need to saturate the cpu cores. Some processing is run manually
from a terminal, some automatically from cron.

Once upon a time I could use nice (and/or renice) to change priorities,
so that things I wanted finished sooner would run faster than ones I
was happy to wait for; but if the fast ones finished, I didn't have to
waste any cpu time as the others jobs could take up the slack.


Now we have "cgroups", which no doubt has many advantages, but seems
imo bizarrely complicated. And although I can still "nice" processes,
it never makes any difference at all -- not even between a set of jobs
created by the same shell or processing script, which I would have
thought of as sensible, and as seems to be implied by some doco as what
should happen (but ime does not).


Unfortunately, despite several attempts, I can't make head or tail of
what to do (if anything can be done) from any of: the cgroups
documentation, various blog posts, posts on stackoveflow/exchange, "askubuntu", etc. Many of the what-to-do" "explanations" seem to
involve being root and/or sudo; and the most useful looking questions
I have found are without answers.

Can anyone help?

#Paul



--- MBSE BBS v1.0.8.4 (Linux-x86_64)
* Origin: A noiseless patient Spider (2:250/1@fidonet)

news20k.noreply@threeformcow.myzen.co.uk (#Paul) writes:

Does anyone here understand cgroups?

I have to put up with running some really quite intensive processing
on an ubuntu machine (running 20.04 LTS, on the basis of uname -a)
that I do not admin (or have any sudo privileges). You may assume I
have zero influence over those who do admin it.

Just as some context, generally each processing run - managed by a
bash script - creates as many background jobs (via &, but not nohup)
as I need to saturate the cpu cores. Some processing is run manually
from a terminal, some automatically from cron.

Once upon a time I could use nice (and/or renice) to change priorities,
so that things I wanted finished sooner would run faster than ones I
was happy to wait for; but if the fast ones finished, I didn't have to
waste any cpu time as the others jobs could take up the slack.

Now we have "cgroups", which no doubt has many advantages, but seems
imo bizarrely complicated. And although I can still "nice" processes,
it never makes any difference at all -- not even between a set of jobs created by the same shell or processing script, which I would have
thought of as sensible, and as seems to be implied by some doco as what should happen (but ime does not).

I think it would be worth investigating why you can’t get ‘nice’
working. For example, start by checking what nice value your processes
get when you do or don’t nice them.

Unfortunately, despite several attempts, I can't make head or tail of
what to do (if anything can be done) from any of: the cgroups
documentation, various blog posts, posts on stackoveflow/exchange, "askubuntu", etc. Many of the what-to-do" "explanations" seem to
involve being root and/or sudo; and the most useful looking questions
I have found are without answers.

AFAIK directly createing a control group normally needs root
permission. But you can create a user slice with a CPU resource policy
attached and systemd will create a suitable one for you.

A very simplistic example:

richard@sfere:~$ cat .config/systemd/user/test.slice
[Slice]
CPUQuota=50%
richard@sfere:~$ systemctl --user daemon-reload
richard@sfere:~$ systemd-run --user --slice test.slice sh -c 'yes > /dev/null'
Running as unit: run-r2fa6a01b9fdb4ce49d208f7f58f8187c.service

[...]

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2504272 richard 20 0 5792 864 780 R 50.0 0.0 0:24.58 yes
805434 news 20 0 122264 96872 52212 S 1.0 0.6 40:13.91 innd

See ‘man systemd.resource-control’ for further options. (You should probably find an introduction to systemd first if you’re not already comfortable with it.)

--
https://www.greenend.org.uk/rjk/

--- MBSE BBS v1.0.8.4 (Linux-x86_64)
* Origin: terraraq NNTP server (2:250/1@fidonet)

On 12/5/23 13:19, #Paul wrote:

Does anyone here understand cgroups?

I have a vague understanding of them.

Once upon a time I could use nice (and/or renice) to change priorities,
so that things I wanted finished sooner would run faster than ones I
was happy to wait for; but if the fast ones finished, I didn't have to
waste any cpu time as the others jobs could take up the slack.

Sounds like you were leveraging (,re,io)nice to adjust the weight /
priority of your processes as you see fit. I would consider this to be
a very traditionally Unix thing to do.

Now we have "cgroups", which no doubt has many advantages, but seems
imo bizarrely complicated. And although I can still "nice" processes,
it never makes any difference at all -- not even between a set of jobs created by the same shell or processing script, which I would have
thought of as sensible, and as seems to be implied by some doco as what should happen (but ime does not).

My speculation is that more of your processes than you care for are run
in different (instances of a) cgroup(s). So when you're shuffling
priority as you had been doing, you're probably doing so against one or
maybe two other processes inside of a cgroup. Unfortunately this has no relation to much less impact on or between other cgroup(s).

Can anyone help?

I would work with the people that do have root / sudo access to the
system your on and get them to help enable you to do what you want.
Even if that means that all of your processes end up in one cgroup so
that you can shuffle things therein as you desire.

This seems to me like what a system administrator is supposed to do, as
in help their users achieve their desired goal.

But, sadly, it sounds like you might have more of a dictator form of
system administrator.

I'm sorry, but I don't have an answer for you.



--
Grant. . . .

--- MBSE BBS v1.0.8.4 (Linux-x86_64)
* Origin: TNet Consulting (2:250/1@fidonet)

Richard Kettlewell <invalid@invalid.invalid> wrote:
[...]

Grant Taylor <gtaylor@tnetconsulting.net> wrote:
[...]

Thanks both, the extra context helps. I'll try to remember to
post back here if I manage to get something to work out, but
atm the box is less heavily loaded atm so it's not such a
priority.

#Paul

--- MBSE BBS v1.0.8.4 (Linux-x86_64)
* Origin: A noiseless patient Spider (2:250/1@fidonet)