Is this possible?

Thu Oct 6 09:58:58 UTC 2016

** Reply to message from Joel Rees <joel.rees at gmail.com> on Thu, 6 Oct 2016
18:25:14 +0900

> (Acid warning. I'm sure you misread the question, Bob, but I'm feeling
> vitriolic at the moment.)
> 
> On Thu, Oct 6, 2016 at 5:24 AM, Bob <ubuntu-qygzanxc at listemail.net> wrote:
> >
> > ** Reply to message from Peter Silva <peter at bsqt.homeip.net> on Wed, 5 Oct 2016
> > 07:46:09 -0400
> >
> > > "swap is maxed out"
> > >
> > > uh... if that's true, it doesn't matter how much or what kind of cpu
> > > you have, it will crawl and die from time to time.   your machine is
> > > sitting in wait i/o.
> >
> > true
> >
> >
> > >   When "swap is maxed out" the kernel will kill
> > > processes randomly, (OOM Killer) you cannot expect a PC with it's
> > > memory (including swap space) entirely full to run correctly.
> >
> > If this is what Linux does that is a very bad design.  I would never have
> > thought the system would do that.
> 
> Oh?
> 
> And by what magic do you expect to avoid that?

No magic.  It is simple system options available to the programmer.

> Name me an OS that can run correctly when all of both swap and RAM are
> in use and some running program suddenly allocates more RAM.

As far as I know all OS's can do it.  I just do not believe the way Linux does
it is the correct way.

> Admittedly "killing randomly" is not what really happens, but it looks
> like that to the user because the user cannot control which processes
> the system puts on semi-permanent halt until the misbehaving mix
> settles down (and then it usually doesn't because the user gets
> impatient and starts clicking things).

Yes, normal user reaction when things do not respond normally.  I have even
done that.

> And many of the visible processes do have a habit of committing
> suicide when put on semi-permanent halt.

The program should be the one that decides how it is going to react to the
current problem.

> > >   One
> > > would routinely expect crashes as the processes that get killed might
> > > be important.
> >
> > No, one, would not expect crashes.  There is no reason for a crash even if the
> > system is thrashing because of excessive paging.  Slowing down to a crawl is
> > what I expect the system to do.
> 
> Very idealistic of you. How does a crawl so slow it looks like the
> system has frozen sound for a crawl?
> 
> > > Here is a healthy amount of swap use:
> > >
> > > KiB Mem : 11733384 total,  2327220 free,  3294000 used,  6112164 buff/cache
> > > KiB Swap: 24761340 total, 24761340 free,        0 used.  7137152 avail Mem
> > >
> > > You should be aiming for all your work to fit in memory. Swap is only
> > > efficient when it is used to swap out bits of memory that are not
> > > being used by running processes (say initialization code, or code that
> > > isn't used often.)
> >
> > This is why paging works and what it was designed to do.  Of course over
> > committing memory heavily is not a good idea.
> 
> How does paging work if there's no swap and a process is demanding
> more memory allocation?

It returns an error code to the program and lets the program decide what to do.
It can then terminate, wait and retry or something else?  Of course this
program may not be the program causing the problem but that is another
discussion.

> > > If your machine is going to swap often, well the
> > > swapping in from disk is 100x to 1000x slower than memory, so it's
> > > going to be slow.
> > >
> > > You either need more memory, or you need to run less stuff at once.
> > > There are ways of doing that in an organized way (batch queueing
> > > systems), but it might be more straight forward just to put some
> > > sequencing in your work.  That is you don't just fire up many
> > > background tasks at once, but rather a few at a time, planning them
> > > out so that all running tasks always fit in memory.
> >
> > Good advice.
> >
> >
> > > use top to look at your memory usage and keep swap down...  It doesn't
> > > have to be zero, but it will never work if it is 'maxed out'.
> >
> > Very true.
> 
> Dang. Didn't you say anything else I can be a stinker about?
> 
> ;-)
> 
> > > On Wed, Oct 5, 2016 at 4:52 AM, Liam Proven <lproven at gmail.com> wrote:
> > > > On 3 October 2016 at 16:35, rikona <rikona at sonic.net> wrote:
> > > >>
> > > >> My hope is to do things in parallel. I work with a fair amount of
> > > >> data, and a large data run may take 12 hours. Sometimes I can split
> > > >> that and run it as multiple processes. While that is running, I may
> > > >> have multiple browsers, each with perhaps 50 or more tabs open. This
> > > >> load makes my current box unusably slow, swap is maxed out, and
> > > >> something very often crashes - I may lose several days of work. And
> > > >> there's email, editing of docs, making diagrams, etc, etc. Perhaps
> > > >> Intel can do one 12 hour job in 4 hours, but I still have lots going
> > > >> on during that 4 hours.
> > > >
> > > >
> > > > I'm ignoring all the pointless advocacy here.
> > > >
> > > > If you have stability issues, you need to troubleshoot them properly.
> > > >
> > > > You need to profile your workloads and find the bottlenecks.
> > > >
> > > > And if it's background stuff and concerns with OSes struggling to
> > > > balance conflicting workloads then you should probably be looking at
> > > > VM solutions, and partitioning off the background  number-crunching
> > > > tasks.
> > > >
> > > > Throwing CPU cores at the problem is inane and a pointless waste of
> > > > cash. Throwing slower cores is burning banknotes. And throwing slower
> > > > cores *when you're not even sure it's CPU-bound* is just stupid.
> > > >
> > > > Sorry, but it is.
> > > > --
> > > > Liam Proven • Profile: http://lproven.livejournal.com/profile
> > > > Email: lproven at cix.co.uk • GMail/Twitter/Facebook/Flickr: lproven
> > > > Skype/MSN: lproven at hotmail.com • LinkedIn/AIM/Yahoo: liamproven
> > > > Cell/Mobiles: +44 7939-087884 (UK) • +420 702 829 053 (ČR)
> >
> > --
> > Robert Blair
> >
> >
> > A government big enough to give you everything you want, is strong enough to take everything you have.  -- Thomas Jefferson
> >
> 
> I like that quote.
> 
> -- 
> Joel Rees
> 
> I'm imagining I'm a novelist:
> http://joel-rees-economics.blogspot.com/2016/04/economics-101-novel-rough-draft-index.html

-- 
Robert Blair

Beer is proof that God loves us and wants us to be happy.  -- Benjamin Franklin