ACK: [SRU][MANTIC][PATCH 0/5] correct cephfs pull request for uidmap support

Fri Oct 27 17:10:13 UTC 2023

On Fri, Oct 27, 2023 at 07:53:42PM +0300, Dimitri John Ledkov wrote:
> BugLink: https://bugs.launchpad.net/bugs/2041613
> 
> [ Impact ]
> 
>  * During mantic dev cycle, we pulled all of cephfs testing patches,
> rather than uidmapshift patches alone
> 
>  * We also pulled in debug patches that generate a lot of dmesg and
> delays in the cephfs module
> 
> [ Test Plan ]
> 
>  * run ceph fs and experience less dmesg at info level
> 
> [ Where problems could occur ]
> 
>  * Hopefully the reverts were done correctly, and none of the messages
> reference non-existing things, in all the code paths.
> 
> [ Other Info ]
> 
>  * Story time how we got here:
> 
> The following pull requests
> 
> BugLink: https://bugs.launchpad.net/bugs/2038522
> BugLink: https://bugs.launchpad.net/bugs/2032959
> 
> generated the following commit list 1:
> 
> eddf9ed788 UBUNTU: [Config] disable shiftfs
> 72e8806959 UBUNTU: SAUCE: ceph: enable unsafe idmapped mounts by default
> 80b85e5d76 UBUNTU: SAUCE: ceph: allow idmapped mounts
> 9d341bb8aa UBUNTU: SAUCE: ceph/file: allow idmapped atomic_open inode op
> ddd1a8ee19 UBUNTU: SAUCE: ceph/acl: allow idmapped set_acl inode op
> 1b12f12987 UBUNTU: SAUCE: ceph: allow idmapped setattr inode op
> b1f695f0d1 UBUNTU: SAUCE: ceph: pass idmap to __ceph_setattr
> 8a9a3b9e88 UBUNTU: SAUCE: ceph: allow idmapped permission inode op
> e5471722b3 UBUNTU: SAUCE: ceph: allow idmapped getattr inode op
> f6b1d353fa UBUNTU: SAUCE: ceph: pass an idmapping to mknod/symlink/mkdir
> 5ecfa6f659 UBUNTU: SAUCE: ceph: add enable_unsafe_idmap module parameter
> 19724a984a UBUNTU: SAUCE: ceph: handle idmapped mounts in create_request_message()
> 6ec4e53238 UBUNTU: SAUCE: ceph: stash idmapping in mdsc request
> 8006617c5e UBUNTU: SAUCE: fs: export mnt_idmap_get/mnt_idmap_put
> 55f4f0afc5 UBUNTU: SAUCE: ceph: make num_fwd and num_retry to __u32
> 54e03068d6 UBUNTU: SAUCE: ceph: make the members in struct ceph_mds_request_args_ext an union
> f7ec09aaac UBUNTU: SAUCE: ceph: print the client global_id in all the debug logs
> b2831559c5 UBUNTU: SAUCE: ceph: add ceph_inode_to_client() helper support
> a0852434ee UBUNTU: SAUCE: ceph: move mdsmap.h to fs/ceph/
> cfda2c4b70 UBUNTU: SAUCE: ceph: rename _to_client() to _to_fs_client()
> 42924fda3a UBUNTU: SAUCE: ceph: pass the mdsc to several helpers
> 9f6e6f9ce9 UBUNTU: SAUCE: ceph: add the *_client debug macros support
> 8f61639667 UBUNTU: SAUCE: ceph: BUG if MDS changed truncate_seq with client caps still outstanding
> a53dba9297 UBUNTU: SAUCE: ceph: make sure all the files successfully put before unmounting
> 17fcf5d75d UBUNTU: SAUCE: mm: BUG if filemap_alloc_folio gives us a folio with a non-NULL ->private
> 834ebb591f UBUNTU: SAUCE: ceph: dump info about cap flushes when we're waiting too long for them
> 498d08b9bd UBUNTU: SAUCE: rbd: bump RBD_MAX_PARENT_CHAIN_LEN to 128
> 3ae75e2438 UBUNTU: SAUCE: libceph: do not include crypto/algapi.h
> d984763988 UBUNTU: SAUCE: ceph: switch ceph_lookup/atomic_open() to use new fscrypt helper
> 8e298c72fb UBUNTU: SAUCE: ceph: fix updating i_truncate_pagecache_size for fscrypt
> 288f935219 UBUNTU: SAUCE: ceph: wait for OSD requests' callbacks to finish when unmounting
> 435fc54954 UBUNTU: SAUCE: ceph: drop messages from MDS when unmounting
> 862a1e3c2a UBUNTU: SAUCE: ceph: update documentation regarding snapshot naming limitations
> 904f39f283 UBUNTU: SAUCE: ceph: prevent snapshot creation in encrypted locked directories
> a47bdc0f8a UBUNTU: SAUCE: ceph: add support for encrypted snapshot names
> e4202a1c3e UBUNTU: SAUCE: ceph: invalidate pages when doing direct/sync writes
> 9f781cfb0d UBUNTU: SAUCE: ceph: plumb in decryption during reads
> e76383977f UBUNTU: SAUCE: ceph: add encryption support to writepage and writepages
> 0b571b2ac0 UBUNTU: SAUCE: ceph: add read/modify/write to ceph_sync_write
> a81bd5f1c8 UBUNTU: SAUCE: ceph: align data in pages in ceph_sync_write
> edbe3d2a62 UBUNTU: SAUCE: ceph: don't use special DIO path for encrypted inodes
> efed7d1764 UBUNTU: SAUCE: ceph: add truncate size handling support for fscrypt
> 5d65284a6a UBUNTU: SAUCE: ceph: add object version support for sync read
> d76cff7c37 UBUNTU: SAUCE: libceph: allow ceph_osdc_new_request to accept a multi-op read
> 53d1c3958e UBUNTU: SAUCE: libceph: add CEPH_OSD_OP_ASSERT_VER support
> 7ecbaa5849 UBUNTU: SAUCE: ceph: add infrastructure for file encryption and decryption
> 4142d15305 UBUNTU: SAUCE: ceph: handle fscrypt fields in cap messages from MDS
> 53507ac6a8 UBUNTU: SAUCE: ceph: size handling in MClientRequest, cap updates and inode traces
> b9936d13f4 UBUNTU: SAUCE: ceph: mark directory as non-complete after loading key
> 906d57a83b UBUNTU: SAUCE: ceph: allow encrypting a directory while not having Ax caps
> 4a58d08c63 UBUNTU: SAUCE: ceph: add some fscrypt guardrails
> 83229b1f64 UBUNTU: SAUCE: ceph: create symlinks with encrypted and base64-encoded targets
> 6f10a0358e UBUNTU: SAUCE: ceph: add support to readdir for encrypted names
> 03584cc481 UBUNTU: SAUCE: ceph: pass the request to parse_reply_info_readdir()
> a3743d846e UBUNTU: SAUCE: ceph: make ceph_fill_trace and ceph_get_name decrypt names
> 6c9af9c3a8 UBUNTU: SAUCE: ceph: add helpers for converting names for userland presentation
> 8c360700dd UBUNTU: SAUCE: ceph: make d_revalidate call fscrypt revalidator for encrypted dentries
> 8344e0ee75 UBUNTU: SAUCE: ceph: set DCACHE_NOKEY_NAME flag in ceph_lookup/atomic_open()
> 867d49582b UBUNTU: SAUCE: ceph: decode alternate_name in lease info
> a4db7516c2 UBUNTU: SAUCE: ceph: send alternate_name in MClientRequest
> b98a9a17d8 UBUNTU: SAUCE: ceph: encode encrypted name in ceph_mdsc_build_path and dentry release
> 1013995dc1 UBUNTU: SAUCE: ceph: add base64 endcoding routines for encrypted names
> 9b5eadd138 UBUNTU: SAUCE: ceph: make ioctl cmds more readable in debug log
> 6a7040d723 UBUNTU: SAUCE: ceph: add fscrypt ioctls and ceph.fscrypt.auth vxattr
> 777d91ff3d UBUNTU: SAUCE: ceph: implement -o test_dummy_encryption mount option
> e5e9604b73 UBUNTU: SAUCE: ceph: fscrypt_auth handling for ceph
> fbbda6f972 UBUNTU: SAUCE: ceph: use osd_req_op_extent_osd_iter for netfs reads
> 68f68a42d2 UBUNTU: SAUCE: libceph: add new iov_iter-based ceph_msg_data_type and ceph_osd_data_type
> 1385eb1b29 UBUNTU: SAUCE: ceph: make ceph_msdc_build_path use ref-walk
> 8465a7391c UBUNTU: SAUCE: ceph: preallocate inode for ops that may create one
> 24f6b4cf5d UBUNTU: SAUCE: ceph: add new mount option to enable sparse reads
> e74feccd77 UBUNTU: SAUCE: libceph: add sparse read support to OSD client
> 9ecbe41d10 UBUNTU: SAUCE: libceph: add sparse read support to msgr1
> 35f0e93c0e UBUNTU: SAUCE: libceph: support sparse reads on msgr2 secure codepath
> fcf0320c64 UBUNTU: SAUCE: libceph: new sparse_read op, support sparse reads on msgr2 crc codepath
> c0a553f6ec UBUNTU: SAUCE: libceph: define struct ceph_sparse_extent and add some helpers
> e23e44a2e1 UBUNTU: SAUCE: libceph: add spinlock around osd->o_requests
> 
> But what actually was only needed, is commit list 2
> 
> eddf9ed788 UBUNTU: [Config] disable shiftfs
> 72e8806959 UBUNTU: SAUCE: ceph: enable unsafe idmapped mounts by default
> 24697c7f4e ceph: allow idmapped mounts
> 4d80ea0a86 ceph/file: allow idmapped atomic_open inode op
> d286bfced3 ceph/acl: allow idmapped set_acl inode op
> 6cf8458860 ceph: allow idmapped setattr inode op
> 011686a286 ceph: pass idmap to __ceph_setattr
> 153c324403 ceph: allow idmapped permission inode op
> 59a7485e9b ceph: allow idmapped getattr inode op
> 3298624765 ceph: pass an idmapping to mknod/symlink/mkdir
> 8912444672 ceph: add enable_unsafe_idmap module parameter
> 30e397e5b0 ceph: handle idmapped mounts in create_request_message()
> e41ba76459 ceph: stash idmapping in mdsc request
> 8c18089082 fs: export mnt_idmap_get/mnt_idmap_put
> d02306a1d4 ceph: make num_fwd and num_retry to __u32
> 
> To match the v10 submission for v6.6 on v6.5 kernel https://lore.kernel.org/all/20230807132626.182101-1-aleksandr.mikhalitsyn@canonical.com/
> 
> However luckily all the patches that got pulled from the testing tree are actually making their way into v6.6 and v6.7.
> 
> Some of the patches pulled however,are potentially harmful, since they were marked "DO NOT MERGE" which is commit list 3:
> 
> 8f61639667 UBUNTU: SAUCE: ceph: BUG if MDS changed truncate_seq with client caps still outstanding
> a53dba9297 UBUNTU: SAUCE: ceph: make sure all the files successfully put before unmounting
> 17fcf5d75d UBUNTU: SAUCE: mm: BUG if filemap_alloc_folio gives us a folio with a non-NULL ->private
> 834ebb591f UBUNTU: SAUCE: ceph: dump info about cap flushes when we're waiting too long for them
> 498d08b9bd UBUNTU: SAUCE: rbd: bump RBD_MAX_PARENT_CHAIN_LEN to 128
> 
> For v6.5 kernels - let's revert those, and keep the rest of ceph
> testing patches in place. And hope for the best maybe.
> 
> For v6.6+ kernels - let's drop all the commits from list 1, and pick
> up commits from list 2 whichever are not merged yet.

The extra patches (despite being marked as "do not merge") were applied
to provide additional debugging information in case of potential
bugs/regressions.

Now that the idmapped mounts feature in ceph has been tested we can
definitely drop them and keep only what is actually needed.

I also agree with the plan to pick commits from list 2 for v6.6,
but for now:

Acked-by: Andrea Righi <andrea.righi at canonical.com>