Consideration of archive mirror requirements (filling up to 1TB now)

[Fumihito YOSHIDA]

Hi developers and mirror server admins (+Canonical IS team?),

I would like to raise a warning(#1) and discussion(#2) about archive
mirror capacity and
want to clarify requirements.

#1 warning for mirror admins
-------------------------------------------------------

Firstly, I would like to pull your attention immediately, if you are
admin of mirror
servers, and your server uses 2TiB physical storage, please check the capacity
of your storage free space that is used for archive mirrors.

Current archive size is filling up to approx. 1.8TB, and default ext4
reserved block is
5%, as a result, as a result, we are consuming approx 2TB (in typical,
buffer is 15GB),
like below:

$ df -m | grep extra1
/dev/sdc1                           1877613 1767423     14791 100% /extra1

- And, if your server has release mirrors (several GB), it can reach
to full and you
  might see something bad in your mirrors.

- I assume that 15TB is not enough for future development in kinetic
(in development
  phase, we will consume more storage.)

- So, if you are using 2TB storage, you might consider tune2fs -m0 for releasing
  reserved blocks for freeing mirror capacities. It can provide
additional storage
  as first aid (approx 80+GB).
  Note: You have to change "/dev/sdc1" section to fit your server's.

$ sudo tune2fs -l /dev/sdc1 | grep 'Reserved block count'
Reserved block count:     24418227

$ sudo tune2fs -m0 /dev/sdc1
tune2fs 1.46.5 (30-Dec-2021)
Setting reserved blocks percentage to 0% (0 blocks)

$ sudo tune2fs -l /dev/sdc1 | grep 'Reserved block count'
Reserved block count:     0


#2 discuss about old archives, such as trusty/xenial
-------------------------------------------------------------------------------------

Okay, acute care is finished, and we might have some discussion time for
discussion.  I would like to raise about that matter for considering future
archive mirror requirements.

At right now, we have great guide in w.u.c and it says:
https://wiki.ubuntu.com/Mirrors
> The Ubuntu archive, as of 2021-07-01, uses about: 1.5TB of disk space
> for the Ubuntu package archive.

However, today, we consumed 1.8TB storage and it might hit 2TB storage limit
shown above. And I assume that some of us mirror admins are using
2TB storage for this purpose.

And we have trusty (14.04) and xenial (16.04) in our archive and they consumed
some space.

$ ls -1d ubuntu/dists/*-updates
ubuntu/dists/bionic-updates
ubuntu/dists/devel-updates
ubuntu/dists/focal-updates
ubuntu/dists/hirsute-updates
ubuntu/dists/impish-updates
ubuntu/dists/jammy-updates
ubuntu/dists/kinetic-updates
ubuntu/dists/trusty-updates
ubuntu/dists/xenial-updates

- They (trusty, xenial) are already EOLed, but live with ESM, and consumed some
  capacity and reached to 2TB physical storage limit.

- hirsute is EOLed already and will be deleted in the near(?) future.

- Considering with rational viewpoint with some assumptions, the state
is balanced, and
  might not be so serious. Now is full, and we will have some spaces
in near future
  but the timing of removing old archives is a matter.

- 4 years rough estimation is below, and with optimistic projection,
freeing reserved
  space of ext4 (and getting 80+GB storage) is enough to handle.

(now) hirsute should be deleted.              (-1) sum: -1
2022-04 impish will be EOLed                  (-1) sum: -2
2022-10 K will release and L will start       (+1) sum:-1
2023-04 L will release and M will start       (+1) sum: 0
2023-07 K will be EOLed                           (-1) sum: -1
2023-10 M will release and N(LTS) will start   (+1) sum:0
2024-01 L will be EOLed                         (-1) sum: -1
2024-04 N(LTS) will release and O will start   (+1) sum: 0
2024-04 trusty ESM will be EOLed             (-1) sum: -1
2024-07 M will be EOLed                          (-1) sum: -2
2024-10 O will release and P will start       (+1) sum: -1
2025-04 P will release and Q will start       (+1) sum: 0
2025-07 O will be EOLed                          (-1) sum: -1
2025-10 Q will release R(LTS) start           (+1) sum: 0
2026-01 P will be EOLed                           (-1) sum: -1
2026-04 R(LTS) will release and S start      (+1) sum: 0
2026-04 xenial ESM will be EOLed              (-1) sum: -1

So, possible measures are:
a) update wiki for warning to 2TB storage admins.
b) a+ delete xenial/trusty (and future "dead" LTS releases)
c) a+ update mirror requirements (to 2+TB) in the future.

Any ideas?