Is multipath essential on Server 20.04?

[Liam Proven]

On Fri, 2 Oct 2020 at 22:52, R C <cjvijf at gmail.com> wrote:

> I wonder why  that "bottom posting" is such an issue,  there was a time
> where no one really cared....    but anyway:

Well, no-one cared because it was the norm and everyone did it.

Then Microsoft came along with Outlook and broke email, and the world
has never recovered.

The big benefit is that with bottom-posting you can easily address
multiple points one-by-one, whereas with top-posting it's all at once.
Secondly, it's easier to follow a bottom-posted thread, because
English like most languages reads top-to-bottom.

> I used to develop large scale storage systems, like Lustre.

Okay. I just document them (he said, sitting here in a Ceph T-shirt.)

> Scality
> builds large "object stores", basically an
>
> object based storage system, I did some R&D work for that.

OK. I looked them up on Wikipedia. Probably because they're a
competitor to my $DAYJOB, it was new to me.

> Basically when you have multiple devices show up (/dev/sd*) and they are
> used in a virtual device, it would be benificial to use multipath,
> especially when the number of physical devices go up.

I was struggling to understand _why_.

> I assume you built a raid pool with your ZFS setup, and use multiple,
> actual, drives in your ZFS pool.

Yes. 5 x 2TB USB 3 drivers.

> When writing to the pool, a virtual
> device, at some point a stripe needs to end up on a physical device
> somewhere. (and vice versa, for reading from a pool,  a stripe needs to
> come from a physical device somewhere)

Sure. But that's what devicemapper does.

> That means you have multiple paths to a virtual device, at least one for
> each actual drive in that pool.

Um. Not if there is 1 server with _n_ drives on it, no. Not AFAICS. 2
servers with _n_ drives, possibly, depending on architecture.

>  That's what you'd want multipath for,
> especially under load when you are hammering that ZFS filesystem. It
> starts to even get more important when you  put a storage system on top
> of ZFS, for example put Lustre on top of ZFS pools, where each pool
> consists of multiple actual drives/devices.

I will take your word, but I am mainly sharing the single RAIDZ to an
iMac using Netatalk.

> Yes JBODs are big boxes that contain nothing but a boatload of drives,
> and they would all show up as devices/drives, in /dev.

Sure. I was running one on NT Server 4 as my home server in 1997 or
so: 7 full-height 5.25" 700MB SCSI-2 drives in a metre-tall 70kg cab.
In full flight, it sounded like a Harrier was hovering outside and I
couldn't take phone calls, but I had a quite fast 5GB RAID on my
dual-Pentium-100-server, backing up to an Exabyte drive using VHS-C
cassettes, and it cost next to nothing. Old surplus kit.

I'm not totally new to this stuff. :-)

> Typically  you'd
> make a bunch of pools of drives, array, often done with ZFS and then use
> these virtual/ drives, ZFS pools, for putting a storage system on top
> of.

OK, so this has changed since I last implemented it. I made an array,
formatted it and shared it. That was it: job done. I've done this on
NT 4 Server, Windows Server 2000/2003/2008 and various Linuxes. No
need for any layer above the RAID.

>  (It is done both for compression and for redundancy. JBODs are used
> for building filesystems many PBs in size.)

Hmmm. OK, not when I last did it.

> I don't think you need multipath simply because you have multiple
> drives,  multipath is needed for optimizing io on a virtual device,
> consisting of more than one actual physical drive/device, where you have
> multiple actual paths to the physical drives. Especially in parallel io
> it becomes a big deal.

This is the tricky bit to follow. I did some more Googling and found a
few articles full of bloviating -- such as this:
https://www.thegeekdiary.com/beginners-guide-to-device-mapper-dm-multipathing/

... and one that was comprehensible. This:
https://www.thegeekdiary.com/understanding-linux-multipath-using-dm-multipath/

IOW, this is valuable _if and only if_ there are multiple different
routes over separate comms channels between your server and your
storage.

But all my drives are local: USB 3 drives, on a single USB 3 hub, on a
single USB 3 port on a RasPi 4. AIUI the Pi 4 only has a single PCI-e
channel so all ports share a single 4Gb/s channel to the CPU.

So, IIUIC, there is no possibility of multipath here, and indeed, I
would guess that normally if the drives are all local to the server,
then there can't be, right?

Anyway, I removed it, the Pi's idle RAM consumption fell by about
150MB from nearly 400MB to about 250-something MB, and it all still
seems to work. :-)

It's a bit too soon to tell if performance is any better.

I can still see ssh sessions pausing -- freezing then unfreezing --
when I telnet into its wired GbE port, but it doesn't do this if I
connect to its WLAN port. That is not good. I am beginning to suspect
my network cable.  (See thread titled "Anyone running Server on a
Raspberry Pi 4?" on July 21.)

-- 
Liam Proven – Profile: https://about.me/liamproven
Email: lproven at cix.co.uk – gMail/gTalk/gHangouts: lproven at gmail.com
Twitter/Facebook/LinkedIn/Flickr: lproven – Skype: liamproven
UK: +44 7939-087884 – ČR (+ WhatsApp/Telegram/Signal): +420 702 829 053