[Bug 1970634] Re: FTBFS: mariadb fails to start due to low MEMLOCK limit
Andreas Hasenack
1970634 at bugs.launchpad.net
Wed Jul 6 13:09:54 UTC 2022
Jammy verification
Confirming the bug with the jammy packages:
root at j:~# apt-cache policy mariadb-server
mariadb-server:
Installed: 1:10.6.7-2ubuntu1
Candidate: 1:10.6.7-2ubuntu1
Version table:
*** 1:10.6.7-2ubuntu1 500
500 http://archive.ubuntu.com/ubuntu jammy/universe amd64 Packages
100 /var/lib/dpkg/status
root at j:~# systemctl status mariadb.service
● mariadb.service - MariaDB 10.6.7 database server
Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
Active: activating (auto-restart) (Result: core-dump) since Wed 2022-07-06 13:07:10 UTC; 595ms ago
Docs: man:mariadbd(8)
https://mariadb.com/kb/en/library/systemd/
Process: 2014 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
Process: 2015 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
Process: 2017 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= || VAR=`cd /usr/bin/..; /usr/bin/galera_recovery`; [ $? -eq 0 ] && systemctl set-environment _WSREP_ST>
Process: 2048 ExecStart=/usr/sbin/mariadbd $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=dumped, signal=ABRT)
Main PID: 2048 (code=dumped, signal=ABRT)
Jul 06 13:07:10 j systemd[1]: mariadb.service: Failed with result 'core-dump'.
Jul 06 13:07:10 j systemd[1]: Failed to start MariaDB 10.6.7 database server.
With the packages from proposed:
root at j:~# apt-cache policy mariadb-server
mariadb-server:
Installed: 1:10.6.7-2ubuntu1.1
Candidate: 1:10.6.7-2ubuntu1.1
Version table:
*** 1:10.6.7-2ubuntu1.1 500
500 http://archive.ubuntu.com/ubuntu jammy-proposed/universe amd64 Packages
100 /var/lib/dpkg/status
1:10.6.7-2ubuntu1 500
500 http://archive.ubuntu.com/ubuntu jammy/universe amd64 Packages
It starts just fine:
root at j:~# systemctl status mariadb.service
● mariadb.service - MariaDB 10.6.7 database server
Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2022-07-06 13:09:09 UTC; 20s ago
Docs: man:mariadbd(8)
https://mariadb.com/kb/en/library/systemd/
Process: 3626 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
Process: 3627 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
Process: 3629 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= || VAR=`cd /usr/bin/..; /usr/bin/galera_recovery`; [ $? -eq 0 ] && systemctl set-environment _WSREP_ST>
Process: 3668 ExecStartPost=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
Process: 3670 ExecStartPost=/etc/mysql/debian-start (code=exited, status=0/SUCCESS)
Main PID: 3658 (mariadbd)
Status: "Taking your SQL requests now..."
Tasks: 11 (limit: 1154)
Memory: 59.5M
CGroup: /system.slice/mariadb.service
└─3658 /usr/sbin/mariadbd
Jul 06 13:09:09 j mariadbd[3658]: Version: '10.6.7-MariaDB-2ubuntu1.1' socket: '/run/mysqld/mysqld.sock' port: 3306 Ubuntu 22.04
Jul 06 13:09:09 j systemd[1]: Started MariaDB 10.6.7 database server.
Jul 06 13:09:09 j /etc/mysql/debian-start[3672]: Upgrading MySQL tables if necessary.
Jul 06 13:09:09 j /etc/mysql/debian-start[3675]: Looking for 'mysql' as: /usr/bin/mysql
Jul 06 13:09:09 j /etc/mysql/debian-start[3675]: Looking for 'mysqlcheck' as: /usr/bin/mysqlcheck
Jul 06 13:09:09 j /etc/mysql/debian-start[3675]: This installation of MariaDB is already upgraded to 10.6.7-MariaDB.
Jul 06 13:09:09 j /etc/mysql/debian-start[3675]: There is no need to run mysql_upgrade again for 10.6.7-MariaDB.
Jul 06 13:09:09 j /etc/mysql/debian-start[3675]: You can use --force if you still want to run mysql_upgrade
Jul 06 13:09:09 j /etc/mysql/debian-start[3683]: Checking for insecure root accounts.
Jul 06 13:09:09 j /etc/mysql/debian-start[3687]: Triggering myisam-recover for all MyISAM tables and aria-recover for all Aria tables
Jammy verification succeeded.
** Tags removed: verification-needed-jammy
** Tags added: verification-done-jammy
--
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1970634
Title:
FTBFS: mariadb fails to start due to low MEMLOCK limit
Status in mariadb-10.6 package in Ubuntu:
Fix Released
Status in systemd package in Ubuntu:
Confirmed
Status in mariadb-10.6 source package in Jammy:
Fix Committed
Status in systemd source package in Jammy:
New
Bug description:
[Impact]
Jammy's MariaDB will fail to build, and also fail to start, if the
underlying kernel is 5.4.x (focal's) and if it's running in an
unprivileged container (lxd, docker). It will also fail to build in
launchpad builders.
Common scenarios where this combination exists is a focal host,
running an unprivileged jammy container (lxd or docker), or even a
chroot (the launchpad builders).
Jammy's MariaDB was built with io_uring support, and it tries to
enable it at runtime if it deems it's running on a supported kernel.
There is a range of kernel versions it checks, but of interest for
this SRU, the Focal (5.4.x) kernel is inside this range and io_uring
will be enabled. The jammy kernel (5.15) is not, so io_uring will not
be enabled there, and thus the bug is not manifested in that case.
If io_uring is enabled, a higher MEMLOCK limit is required than what
is set by default in focal or jammy (64Mb is what we get, 256Mb or
more is required).
The systemd unit file for mariadb tries to raise this limit, but in an
unprivileged container this won't work.
MariaDB has checks in place to catch when MEMLOCK is too low, in which
case it will not use io_uring, but these checks are failing because of
the LTO build flags that were used in the jammy build of mariadb. It's
unclear if it's a bug in gcc or something else. There is more
information in comment #1, comment #5 and later.
The suggested fix here is to disable LTO for the jammy build. This has
been done for kinetic already, and is also applied to the debian
packaging (inside a distro-specific conditional).
[Test Plan]
The test plan is to launch mariadb in a jammy lxd container running on a focal host.
lxc launch ubuntu:focal f --vm
lxc shell f
lxd init # just hit enter for all questions
lxc launch ubuntu:jammy j
lxc shell j
ulimit -l # confirm it's less than 256
apt update && apt install mariadb-server -y
After the installation, mariadb will not be running, and this can be
checked with:
systemctl status mariadb.service
or
journalctl -u mariadb.server
You will see something like this:
Jun 17 18:32:01 jammy-mariadb systemd[1]: Starting MariaDB 10.6.7 database server...
Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 2022-06-17 18:32:01 0 [Note] /usr/sbin/mariadbd (server 10.6.7-MariaDB-2ubuntu1) starting as process 1864 ...
Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 2022-06-17 18:32:01 0 [Note] InnoDB: The first data file './ibdata1' did not exist. A new tablespace will be created!
Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 2022-06-17 18:32:01 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 2022-06-17 18:32:01 0 [Note] InnoDB: Using transactional memory
Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 2022-06-17 18:32:01 0 [Note] InnoDB: Number of pools: 1
Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 2022-06-17 18:32:01 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 2022-06-17 18:32:01 0 [Warning] mariadbd: io_uring_queue_init() failed with ENOMEM: try larger memory locked limit, ulimit -l, or https://mariadb.com/kb/en/systemd/#configuring-limitmemlock under systemd (262144 bytes required)
Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 220617 18:32:01 [ERROR] mysqld got signal 6 ;
And a crash dump.
With the fixed version, the service will be running normally after
installation.
[Where problems could occur]
The proposed fix is not a surgical strike. It's unfortunate that we didn't get to the bottom of why LTO is causing this behavior. Reverting it is still the quickest and less risky change at the moment, though. This gets us on par with upstream binary builds, and debian builds, and these also have wide test coverage and ample user base.
The other regression possibility is that this is a rebuild of mariadb
in the current jammy environment, and the package that is currently in
jammy was built on March 10th, 2022. Most likely the reverse
dependencies were updated in jammy since then.
It's unclear how 10.6.7-2ubuntu1 migrated in jammy. I checked build
logs and dep8 logs, and can't tell why the tests passed. At least the
build log in jammy shows the host kernel was 5.4.x, so it should have
been affected. My only explanation is that at that time, the MEMLOCK
limit was higher in that environment for some reason, and didn't
trigger this bug. Then at some point later, launchpad builders
changed, and MEMLOCK was reduced to 64Mb.
https://irclogs.ubuntu.com/2022/04/28/%23ubuntu-devel.html#t15:00 has
some discussion about it.
[Other Info]
Not at this time.
[Original Description]
<rbasak> ahasenack: IIRC, originally Launchpad was FTBFSing on mariadb that included io_uring support because upstream were doing a build time test for io_uring (and I think still are), which is wrong because it should be done at runtime since the lack of io_uring availablity at build time doesn't tell us about its availablity at runtime.
<rbasak> But then the Launchpad builders got updated to a newer release and therefore a newer kernel that supported it.
<rbasak> AIUI, that's how we ended up with a successful build in the Jammy release pocket (of 10.6).
<ahasenack> I think the lp builders are using the focal hwe kernel
<ahasenack> 5.4.0-something
<ahasenack> let me check that build log
<rbasak> But then something changed that caused this current FTBFS, and I haven't tracked down what that is.
<ahasenack> hm, both are 10.6.7
<ahasenack> release and proposed
<rbasak> What puzzles me is that if the root cause is a memlock rlimit issue then why did it work before?
<rbasak> So since there's a contradiction somewhere, maybe one or more of my "facts" above is wrong.
<ahasenack> this is the current failure
<ahasenack> 2022-04-14 8:11:49 0 [Warning] mariadbd: io_uring_queue_init() failed with ENOMEM: try larger memory locked limit, ulimit -l, or https://mariadb.com/kb/en/systemd/#configuring-limitmemlock under systemd (262144 bytes required)
<ahasenack> and ulimit -l confirms that the limit is lower
<ahasenack> Max locked memory 65536 65536 bytes
<ahasenack> just 64kbytes
<rbasak> Yeah but then how did the release pocket build work?
<ahasenack> either the limit was different back then
<ahasenack> or ... stuff
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/mariadb-10.6/+bug/1970634/+subscriptions
More information about the foundations-bugs
mailing list