Gerrit Review Dashboard

LU-20626 lnet: cancel discovery Ping on peer down

An LND peer-down notification can arrive after the send event of a
discovery GET. Its response MD then remains PING_SENT until response
expiry, preventing prompt rediscovery after a connection reset.

Cancel an outstanding discovery Ping when an LND reports its peer
down, but only if no other peer NI is cached UP. A repeated DOWN
notification can be Old news to peer-NI state handling even though a
requeued discovery Ping is outstanding. Retain a peer reference for
cancellation before that return.

This does not cancel an in-flight Push or a Ping that may use another
cached-UP multi-rail path.

Requeue cancellation cleanup under LNET_LOCK_EX while discovery is
running. Wake the thread after moving it to the request queue. These
operations serialize global discovery-queue updates and clear the
failed Ping state promptly; a subsequent discovery request can retry.

Add a sanity-lnet case that drops a discovery GET, verifies the client
sent the GET and entered PING_SENT, then resets matching port-988
SockLND sockets with ss -K. The test records peer state, candidate
sockets, and ss -K status. It sets an LTT deadline and requires the
notification and cancellation before its response expires. It verifies
that peer-down clears PING_SENT and that a fresh discovery succeeds.

Tested with an OST HA failover and with the sanity-lnet case after a
SockLND TCP reset.

Fixes: 0f1aaad4c1b4 ("LU-9480 lnet: implement Peer Discovery")
Signed-off-by: Aboo Valappil <vadakkumuri.valappil.abooback@oracle.com>
Change-Id: I5e5d434895f4b3ec0f7fc520b25b0ddf8fca0ea0

LU-20211 mdt: trunc(0) layout replace on inactive stripes

When truncate(0) hits a file whose layout references inactive OSTs,
and the client advertises OBD_CONNECT2_LAYOUT_REPLACE, replace the
layout: move the old layout onto a volatile victim and instantiate a
fresh layout on the file from the existing pattern via QoS/RR.

Skip replace when any component has LCME_FL_NOSYNC.  That flag freezes
the mirror's existing objects; swap would discard the snapshot.
mdt_replace_unavailable_layout() re-checks this under LAYOUT lock and
returns 0 so setattr falls through to a normal trunc(0).

Extend the layout API with MD_LAYOUT_UNAVAILABLE (detect stripes on
inactive OSTs) and MD_LAYOUT_INSTANTIATE (allocate from a source
pattern).  Teach lod_generate_lovea() to emit uninstantiated LOVEA
templates (as_template), omit placement hints so new stripes use
QoS/RR, and trigger replacement from mdt_reint_setattr() via
mdt_replace_unavailable_layout().

Layout-type handling for as_template / instantiate:
- DoM: never carries OST objects; force stripe_count to 0 and skip
  the OST object array so LOVEA size matches INIT DoM.
- PFL/DoM: keep the DoM component and replace only OST components.
- SEL: coerce LOV_MAGIC_SEL to COMP_V1 for lod_qos_parse_config();
  keep LCME_FL_EXTENSION and restamp SEL when LOVEA is written.
- EC: re-emit bound data/parity as unbound (IS_LINK_ID + shared
  link id) so lod_qos_prep_create() can pair before component IDs
  exist.

Layout replace creates a volatile victim via mdt_orphan_open().
Zero mti_spec first so that internal create does not inherit
leftover create state from the setattr thread.

On success, return MS_LAYOUT_REPLACED (with OBD_MD_FLEASIZE) in the
setattr reply.  The client then applies the new LOVEA via layout
change / prune.  By default prune uses CL_FSYNC_LOCAL and would
writeback dirty pages still cached against the old layout to the
unavailable OSTs (fail or hang).  When MS_LAYOUT_REPLACED is seen
with size 0, set coc_prune_mode to CL_FSYNC_DISCARD so prune discards
that stale OSC dirty cache locally instead of writeback; VFS page
cache is still dropped by ll_truncate_inode_pages_final() as usual.

Add sanity/test_27xt (plain), 27xu (PFL/DoM), 27xv (SEL), 27xw (EC),
and sanity-flr/test_50e/50f.

Test-Parameters: testlist=sanity,sanity-flr serverversion=2.17 env=SANITY_EXCEPT=24D
Test-Parameters: testlist=sanity clientversion=2.17
Assisted-by: Cursor:Grok-4.6 llm_code_and_review_tools
Signed-off-by: Keguang Xu <kxu@ddn.com>
Change-Id: I07e0792a51e144d935cece4b028e17f659e4ccfd

LU-20354 quota: refresh QMT usage cache via MDT/OSP proxy

Wire qmt_refresh_lqe_usg() to pull stale or missing per-slave
usage through MDT: local OSD for the DOM MDT, OSP Q_GETOQUOTA
for OSTs. First refresh runs qmt_setup_lqe_usg() to snapshot
sarr (keeping DQACQ-warmed values) and set lqu_inited.

On fetch failure, keep a valid cached value when present,
otherwise fall back to granted from that slave's local index.

qc_id is already a filesystem ID. MDS/OSS NIDs belong in a
trusted nodemap, so OFD's CLIENT_TO_FS map is identity and
no skip-idmap handshake is needed.

qmt_get_usage() can sum/refresh when qmt_usg_serve is on;
the flag stays off so GETQUOTA still leaves usage unset.

Expose qmt.*.usage_cache_stats (hit/miss/fetch_rpc/fallback).
Add sanity-quota/310 (skipped until qmt_usg_serve is on).

Assisted-by: Cursor:Grok4.6 llm_code_and_review_tools
Signed-off-by: Keguang Xu <kxu@ddn.com>
Change-Id: Ied2f4f1e3c7ee6611a69a57e8064d46a59ba845f

LU-19276 llapi: add llapi_changelog_(de)register

Add llapi_changelog_register() and llapi_changelog_deregister()
to allow them to be used by external applications.

Move CHANGELOG_USER_NAMELEN and CHANGELOG_USER_NAMELEN_FULL to
lustre_user.h since applications need to know the buffer limits
when registering a changelog user.

Man pages llapi_changelog_register/deregister.3 are added and
lctl-changelog_register/deregister.8 are updated to reflect the
new changes. sanity.sh/test_160q is modified to verify this patch.

Move llapi_obdname2devno() and llapi_ioctl_dev() into internal
library functions so that they are more easily usable, but not
yet fully public API functions.

Since llapi_ioctl_dev() requires the OBD device to be registered
in order to call l_ioctl(), have llapi_obdname2devno() call
llapi_register_ioc_dev() internally to ensure it is done instead
of depending on the caller.  Change llapi_register_ioc_dev() so
that it can be called repeatedly without closing and opening the
OBD ioctl device each time.

Rename yaml_get_device_index() to llapi_yaml_get_device_index()
to avoid namespace clash with yaml library. Fix a bug with that
code which returned 'device 1' on error, and leaked resources
leaks on error cleanup paths.

Remove long unused jt_lcfg_device() wrapper function.

Test-Parameters: testlist=sanity serverversion=2.17 env=ONLY=160
Fixes: ba0d5ffc1c ("LU-9680 utils: new llapi_param_display_value().")
Fixes: 0b715b9a53 ("LU-2675 utils: remove loadgen")
Signed-off-by: Andreas Dilger <adilger@thelustrecollective.com>
Signed-off-by: Emoly Liu <emoly@whamcloud.com>
Change-Id: I732366b9972ba7c2d8ed5b49a405647e2d300c1e

LU-19783 lfsck: fix MDT orphan by LFSCK

This patch adds a new option "-O | --mdt-orphan" in "lfsck_start"
to check the possible MDT orphan, which is no "linkEA", owned by
non-root user and group, old enough since its creation time,
(the timeout can be specified by "-T | --mdt-orphan-timeout"), and
the size is 0. It will be moved into ".lustre/lost+found".

The "struct lfsck_bookmark" is modified, then this patch adds the
corresponding check during setup and load.

It introduces a new failure injection "OBD_FAIL_LFSCK_MDT_ORPHAN"
to create an orphan object in MDT, which will be fixed by LFSCK
in the newly added test in sanity-lfsck.

In order to distinguish the MDT orphan object from the active
volatile object, a new flag "LU_OBJECT_VOLATILE" is added to
the lu_object to identify the active volatile object.

It also adds check on "test_45" in "sanity-lfsck" to verify LFSCK
is completed, which should check on "SINGLEMDS" instead of "ost1".

Test-Parameters: testlist=sanity-lfsck serverversion=2.16.1
Signed-off-by: Hongchao Zhang <hongchao@whamcloud.com>
Change-Id: I3fee4e992f24399523362a5c25dcd1590a07f2be

LU-20357 obdclass: clear idle NID statistics on server

Introduce configurable nid_stats_idle_time per target device to
automatically garbage collect stale NID statistics structures that
remain idle with no active exports, preventing memory and debugfs
bloat in high-churn cloud environments.

High-Level Design:
- **Idle NID Stats Tracking**: When a client disconnects, its export
  is cleaned up. If no other exports are using the same NID, the
  corresponding NID statistics structure (`nid_stat`) is marked as
  "idle" and moved to an idle list.
- **Background Purging**: The ping evictor thread
  (ping_evictor_main()) periodically scans the idle list. The thread's
  wait loop is converted from wait_event_idle() to a 60-second timed
  wait to trigger periodic scans. If an entry has been idle for longer
  than the configurable `nid_stats_idle_time` timeout, it is evicted
  from the hash table and freed, along with its associated debugfs
  files.

Key Implementation Details:
- Added `obt_nid_stats_idle_time` configuration parameter per target.
- Added `nid_last_idle` timestamp to `struct nid_stat` to track idle
  duration.
- Idle entries are moved to `obd_nid_stats_idle` list during export
  cleanup.
- Target devices are marked in `obd_devs` xarray for efficient
  scanning by the ping evictor thread.
- Reimplemented the `exports.clear` debugfs interface to clear active
  stats and immediately evict idle stats.
- Added `idle_seconds` output to `exports/*/export` debugfs files to
  show how long the NID has been idle.
- Renamed `obd_nid_stats_get()/obd_nid_stats_put()` to
  `obd_nid_stats_insert()/obd_nid_stats_remove()` to better reflect
  their hash table operations.
- Fixed a potential Use-After-Free by moving NID stats hash
  destruction from class_cleanup() to class_free_dev().
- Fixed a race causing EEXIST on debugfs creation by destroying the
  debugfs entry before removing the nid_stat from the hash table in
  obd_nid_stats_remove().
- Fixed a bug where obd_nid_stats_insert() could sleep under RCU read
  lock when retrying by releasing the RCU lock before sleeping.
- Free `nid_stat` with OBD_FREE_RCU(), as entries are now freed while
  clients may be traversing the hash table, and a reader must not walk
  into reclaimed memory.
- Added `mdt.exports.clear`/`obdfilter.exports.clear` man pages for
  the pre-existing `exports.clear` interface reimplemented here.

Signed-off-by: Jinshan Xiong <jinshanx@google.com>
Change-Id: I511de0c1205105708e9aa89fe9f1cff915dbe2b8

LU-20695 utils: fix mount.lustre_tgt fallback

If target filesystem is mounted with `mount -t lustre_tgt` but the
kernel does not support the `lustre_tgt` fstype, fall back to using
the `lustre` fstype for the mount.

Fix deprecation check to work before 2.20.53 instead of the reverse.

Fix `update_mtab_entry()` to record actual fstype that was used.

Test-Parameters: trivial
Fixes: d3ea87a17f ("LU-16738 utils: fix auto mgsname= mount option")
Fixes: 4b2eca7095 ("LU-12514 utils: try lustre_tgt filesystem for mount")
Signed-off-by: Andreas Dilger <adilger@thelustrecollective.com>
Change-Id: I37914d70668899ec1fd3bdb2f45724b827b0ff18

LU-17957 mdd: POSIX interface for non-root immutable files

Add a POSIX-compatible interface allowing non-root file owners to
make their own files immutable, without requiring CAP_LINUX_IMMUTABLE
or root, by:

  1. Setting the file's atime at least 24 hours into the future
  2. Removing all write permissions from the file

When both conditions are met and mdd.posix_immutable is enabled,
the MDS automatically sets LUSTRE_IMMUTABLE_FL. A new LMA compat
flag, LMAC_RETENTION (wire flag LUSTRE_RETENTION_FL), marks the
file as under this auto-expiry scheme, distinguishing it from a
plain chattr +i file, which must never auto-clear.

lu_posix_immutable_expired() and lu_posix_immutable_owner_or_cap()
are shared inline helpers in md_object.h. They define the common
expiry rule used by MDD: a regular file with both IMMUTABLE_FL and
RETENTION_FL set, no write bits, and atime <= now.
lu_posix_immutable_check() combines these into a single
authorization rule used everywhere LUSTRE_IMMUTABLE_FL is enforced:
setattr, unlink, rename, migrate, open for write/trunc/append,
layout merge/split, layout swap, and mirror/component add. A caller
with CAP_LINUX_IMMUTABLE can always clear the flag directly via
chattr -i, at any time, regardless of expiry. Once the retention period
has expired, the owner or a CAP_LINUX_IMMUTABLE holder may also perform
the operations directly without first clearing the flag. Whoever acts
on an expired file also persists the flag clear via
mdd_clear_expired_posix_immutable() in its own transaction.
mdd_attr_get() applies the same rule on getattr, so a stale flag is not
returned to the owner or a cap holder.

Note, a getattr from a non-owner, non-privileged caller may still
see LUSTRE_IMMUTABLE_FL on an expired but not yet cleared file until
the owner or a privileged caller next touches it. The flag is not
filtered in mdt_pack_attr2body().

osd-ldiskfs and osd-zfs osd_attr_set() are reworked from a one-way
OR of LMA flags to a compute-and-diff set/clear update, since
RETENTION_FL, unlike ORPHAN/ENCRYPT, must be clearable.

On the client, llite gains LLIF_POSIX_IMMUTABLE and
ll_refresh_posix_immutable(), called from ll_revalidate_dentry()
before the VFS IS_IMMUTABLE() check, to re-fetch flags from the MDS
and drop S_IMMUTABLE once LUSTRE_IMMUTABLE_FL is gone. Since the
file stays mode a-w, writes still fail with EACCES until write
permission is restored. Only the immutable restriction is lifted.
Refresh RPCs are rate-limited to 5 per second per client (the
ratelimit state is shared by all such lookups on that client), and
only triggered for files carrying LLIF_POSIX_IMMUTABLE, so plain
chattr +i files never issue extra RPCs. fileattr_get() is refactored
to share ll_md_getattr_flags() with the revalidation path, and hides
LUSTRE_RETENTION_FL from userspace so chattr -i is not rejected as
an unrecognized flag.

Off by default, mdd.posix_immutable=0.

test_106d in sanityn.sh covers flag set, retention expiry,
mode-only failure after expiry, root denied before expiry and
allowed after it, and owner delete after expiry.
test_106e covers lfs migrate by the owner and by a
CAP_LINUX_IMMUTABLE holder after expiry, honoring the same
owner-or-cap rule.
test_106f covers write and unlink denied to a now-owner,
non-privileged caller on an expired but not yet cleared POSIX
immutable file.

Test-Parameters: testlist=sanityn serverversion=2.15.6
Test-Parameters: testlist=sanityn clientversion=2.15.6

Signed-off-by: Chakshu Kansal <ckansal@ddn.com>
Change-Id: Iab9c0f69dd8152a7eaa48e8454120af469ea4d63

LU-20505 lfsck: fix osd-scrub vs lfsck race

osd scrub and lfsck share same iterator, where current position is
stored in os_pos_current and corresponding fid/ino in oic_fid/oic_lid.
osd_scrub_next() advances os_pos_current, but oic_fid/oic_lid are
updated later in osd_iit_iget(). the race should go away if we update
os_pos_current after oic_fid/oic_lid.

Test-Parameters: env=ONLY=5,ONLY_REPEAT=50 testlist=sanity-lfsck
Test-Parameters: env=ONLY=5,ONLY_REPEAT=50 testlist=sanity-lfsck
Test-Parameters: env=ONLY=5,ONLY_REPEAT=50 testlist=sanity-lfsck
Test-Parameters: testlist=sanity-lfsck
Test-Parameters: testlist=sanity-lfsck
Test-Parameters: testlist=sanity-lfsck
Fixes: e2ad44de38a4 ("LU-9488 scrub: reset noslot for full speed scrub")
Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Assisted-by: ClaudeCode:Opus-5
Change-Id: I6f409b40173191efef6931c356a0271df1ae86e4

LU-20095 osd-ldiskfs: clean up orphan list on truncate error

If ldiskfs_truncate() fails for some reason, like journal aborted
or fs remounted read-only, we should remove the inode from the
in-memory orphan list, otherwise during umount we will trigger
J_ASSERT(list_empty(&sbi->s_orphan)) in ldiskfs_put_super().

Add an osd_ldiskfs_truncate() helper to handle this consistently.
Add an osd_obj_fid() helper to simplify FID resultion on error.

Add conf-sanity/265 to test this.

Fixes: 1f3e4df2fb ("LU-18562 osd: llog write batch block allocation")
Fixes: cf29a5e7bf ("LU-10048 osd: async truncate")
Signed-off-by: Li Dongyang <dongyangli@ddn.com>
Signed-off-by: Andreas Dilger <adilger@thelustrecollective.com>
Change-Id: I891de64ec9775e53b76391ddff18e20742e3fc3b

LU-19548 lfs: mirror extend support for EC

Add three EC modes to 'lfs mirror extend':

--mirror-id N --ec D+P   attach a parity mirror to data mirror N
-N [K] --ec D+P          add K data+parity mirror pairs (default 1)
--ec D+P                 attach parity to the file's lone data mirror;
                         2+ mirrors need --mirror-id or -N

As with a regular extend, the new mirrors are in sync on return. Data
mirrors are copied before the merge. A parity mirror has nothing to
copy, so it is merged stale (LL_LEASE_ALLOW_STALE, LU-18746) and the
resync computes it in place. An interrupted extend therefore leaves a
stale parity mirror for a later resync, never an up-to-date one with
no parity. The stale merge needs OBD_CONNECT2_FLR_EC negotiated with
the MDT, else the extend fails with EOPNOTSUPP. A stale data mirror is
refused, as (for now) are a nosync one and a second parity geometry on
one data mirror (LU-20616).

A parity mirror is merged as a layout of parity components only, built
by the new llapi_layout_comp_add_parity_only() and accepted by layout
sanity. lod_declare_layout_merge() stamps the data component with the
parity mirror id so 'lfs mirror split' sees the pair, and refuses a
dangling or already-used link. lod_comp_prep_create() sizes such a
parity from the data stripe count the client sends and leaves it
uninstantiated, as its data component is not in the volatile file;
the resync then allocates it against the full layout, where
lod_check_and_reserve_ost() keeps it off the OSTs of its own data
component on every pass and treats unrelated mirrors as a first-pass
preference only. sanity-ec test_8j asserts this with the QoS
allocator, as round-robin hides it.

lod_verify_striping() and lod_comp_prep_create() reject k == 0 and
m == 0, which the k + m bound alone let through to a divide by zero
and an LASSERT.

The new subtests are gated on MDS > 2.17.57, as an older MDS refuses
a parity-only create buffer.

Fixes: 8c5f5d3ec9b1 ("LU-12668 lod: bind ec mirror to data mirror during creation")
Test-Parameters: testlist=sanity-ec ostcount=8
Signed-off-by: Maximilian Dilger <mdilger@whamcloud.com>
Assisted-by: ClaudeCode:Fable-5.1 llm_code_and_review_tools
Change-Id: I8d670f0558521063af425e2c75a56b00cdd294d7

LU-18857 mgs: add timeout mechanism for allow_register

Add automatic timeout functionality to the allow_register
parameter to improve security by limiting the window during
which new filesystems or targets can register.

Values:
  0          registration disabled (initial default)
  unlimited  registration enabled indefinitely (no timer)
  N[s|m|h]   registration enabled for N seconds/minutes/hours
             (default unit is hours when no units is given)

When a timed value is set, a timer disables registration on expiry.
show() reports remaining time with explicit 's' units.

Timed windows above 7 days are rejected (-EINVAL).
Use 'unlimited' for an open-ended window.

Automatic default window:
 - allow_register defaults to 0 (disabled).
 - the first target ever to register for a brand-new filesystem on a
   freshly loaded MGS automatically opens a one-time 24-hour window,
   unless the admin already set allow_register explicitly (via module
   parameter or lctl set_param). It never re-arms for later filesystems
   or later mounts.
 - suppressing the automatic window entirely requires setting
   allow_register explicitly (e.g. 'options mgs allow_register=0')

Boot-time module parameter:
 - new mgs.allow_register=<value> module parameter applies the same
   policy at module load time via class_allow_register_set(), so it
   gets identical validation, locking, and timer arming as the sysfs
   attribute.

Compatibility changes from the original boolean allow_register:
 - initial default is now 0 (disabled, see automatic default window
   above) instead of 1 (enabled)
 - allow_register=1 set at runtime means "enabled for 1 hour",
   not "enabled"

Persistent set_param -P:
 - allow_register=0 and allow_register=unlimited may be persisted
 - timed values cannot use -P (replaying a duration would open a
   fresh registration window on every mount)
 - on config apply, bare 1 (old boolean "on") maps to unlimited;
   other non-steady values are rejected

Also:
  - Documentation/man4/allow_register.4
  - SEE ALSO allow_register(4) in mkfs.lustre(8) and mount.lustre(8)
  - conf-sanity 157a/157b: changed allow_register=1 to 'unlimited'
  - conf-sanity 157c: timeout window coverage
  - conf-sanity 157d: param persistence and legacy 1->unlimited
  - conf-sanity 157e: mgs module parameter boot-time enforcement
  - mutex around timer/value updates in store/show
  - class_set_global(): reject params with no '=' or with '=' as the
    first character, instead of deriving a garbage/zero offset from
    them, affects every global param, not just allow_register

Signed-off-by: Chakshu Kansal <ckansal@ddn.com>
Change-Id: I8ce969db3e06f82d97da47b11421a2b8833c5656

LU-18746 flr: allow merge stale mirror

Let a mirror be merged into a file without copying data into it
first, so that mirrors can be appended cheaply and resynced later,
by a data mover or by an EC parity extend that computes the mirror
content in place.

The client asks for it with LL_LEASE_ALLOW_STALE, a modifier on
LL_LEASE_LAYOUT_MERGE. It travels to the MDT as CD_MERGE_STALE in
the new close_data field cd_merge_flags and becomes the
LU_XATTR_MERGE_STALE modifier on LU_XATTR_MERGE, on which
lod_declare_layout_merge() marks every merged component
LCME_FL_STALE. An MDT without this change ignores cd_merge_flags
and would merge the mirror as up to date, so the client refuses
the request with -EOPNOTSUPP unless the MDT advertises
OBD_CONNECT2_FLR_EC. That flag is only negotiated with EC enabled
on both the client and the MDT, so a stale merge needs that too.

lfs still merges in sync. The stale path gets its callers in
"LU-19548 lfs: mirror extend support for EC" and "LU-17531 lfs:
add --no-resync option in mirror extend".

Also return the error from the LL_IOC_LEASE unlock argument checks
instead of overwriting it with the lease type from the close.

Fixes: f172b116885 ("LU-10092 llite: Add persistent cache on client")
Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Signed-off-by: Maximilian Dilger <mdilger@whamcloud.com>
Assisted-by: ClaudeCode:Fable-5.1 llm_code_and_review_tools
Change-Id: I0d730566ea9b238aeac5d2e27e77c719a1c73308

LU-12187 lov: MDS layout pattern negotiation

Add obd_connect_data::ocd_layout_pattern_types so a client and the MDS
negotiate the layout patterns and LOV_PATTERN_F_* flags both support.
The client sends what it knows, the MDS replies with the intersection
when it grants OBD_CONNECT2_FLR_EC, and the client falls back to the
pre-2.18 baseline LOV_LAYOUT_PATTERN_TYPES_217 otherwise.  FLR-EC uses
this for LOV_PATTERN_PARITY.  The mask comes from MDT0000 through the
LMV and is refreshed on every MDC reconnect; the client's own request
is restored for each reconnect like the connect flags, so a pattern
the MDS re-enables comes back without a remount.  OSTs are not
involved.

A component whose pattern or flag was not negotiated is left
uninitialized by lov_init_composite() like an unknown pattern, so IO
reads another mirror instead; writes still stale it so that a later
resync rebuilds it.  The client refuses to create such a component
with EOPNOTSUPP on both the setstripe ioctl and lustre.lov xattr
paths.  Parsing stays pattern-agnostic so lsme alloc/free remain
symmetric.

Add mdt.*.layout_pattern_types to change the advertised types at
runtime (mdt_enable_flr_ec only seeds it at MDT start), read-only
lov.*.layout_pattern_types on the client, and man pages for both.
The value is filesystem-wide, so set it on all MDTs.

Rename llite_enable_flr_ec to enable_flr_ec, reached as
lustre.enable_flr_ec via 'lctl {get,set}_param --module' by adding
the lustre module to the libcfs parameter path list.  The test
framework's MODOPTS_LLITE never matched the module name, so
enable_flr_ec=1 was silently dropped; MODOPTS_LUSTRE now applies it,
and sanity-ec negotiates parity explicitly before it starts.

Add conf-sanity 163a-163d: the negotiated mask in the import, the
tunable and reconnect renegotiation, write staling from a client
without EC, and an un-negotiated layout flag injected with
OBD_FAIL_LOV_COMP_FLAG.

Assisted-by: ClaudeCode:Fable-5.1 llm_code_and_review_tools
Test-Parameters: testlist=sanity-ec
Test-Parameters: testlist=conf-sanity env=ONLY="163"
Test-Parameters: testlist=sanity serverversion=2.17 env=SKIP27D=37
Test-Parameters: testlist=sanity-pfl clientversion=2.17
Test-Parameters: testlist=sanity-pfl serverversion=2.17 env=SANITY_PFL_EXCEPT="18 28"
Test-Parameters: testlist=sanity-flr clientversion=2.17
Test-Parameters: testlist=sanity-flr serverversion=2.17 env=SANITY_FLR_EXCEPT="0a"
Signed-off-by: Maximilian Dilger <mdilger@whamcloud.com>
Change-Id: Iaef716e28014be5b91dd50b117dd0881f5c37f37

LU-12064 obdclass: auto-scale at_min based on expected clients

On servers, the global at_min default can now be automatically
derived from the expected_clients tunable as the number of connected
clients grows, using the formula
(ilog2(expected_clients+7) - 3) * 5/2, clamped to [5, 60] seconds and
never exceeding at_max.

The expected_clients and global at_min tunables now use
module_param_cb handlers, and the sysfs at_min attribute gets a
matching store callback, so both entry points validate updates and
coordinate automatic tuning through shared helpers.

class_expected_clients_update() gains an allow_lower flag to
distinguish administrator writes from automatic updates: manual
changes may both increase and decrease expected_clients, while
automatic updates from tgt_update_max_clients() only increase it as
the maximum observed client count grows.

If at_min is written manually to a non-zero value, either via
lctl set_param or through the corresponding kernel module parameter,
the value is considered pinned and automatic scaling from
expected_clients no longer updates it. Writing at_min=0 unpins it,
immediately recalculating at_min from the current expected_clients
value. The pin/unpin logic is consolidated in the class_at_min_set()
helper, called from both the sysfs and module parameter write paths.

Signed-off-by: Chakshu Kansal <ckansal@ddn.com>
Change-Id: I5c42da4343c1ca62b2fd20f7f40e997a356e6b4f

LU-20184 nrs: extend TBF rate to support bandwidth (BPS) mode

The Lustre NRS TBF scheduler currently enforces rate limits in
term of IOPS (operations per second).
This model works well for controlling metadata or small,
relatively uniform I/O. However, on OSS servers with large and
variable-sized I/O, IOPS is often a poor proxy for bandwidth of
how much resource a class is consuming:
- A client issuing 100 X 4 KiB writes is much less demanding than
  one issuing 100 X 4 MiB writes.
- Admins often want to bound bandwidth (bytes/s) per class bucket
  (i.e, per user/job/nodemap, et al).

To address this, we propose extending TBF to support bandwidth
based performance control (BPS) in addition to the existing IOPS
mode.
- Rules can specify rates in bytes per second (or equivalent
  units).
- Enforcement is based on the actual I/O size of read/write
  requests.

The bandwidth cost of an OST_READ or OST_WRITE request is a page
count. The niobuf count comes from the size of the niobuf buffer
that the request carries. The code does not read the client field
ioo_bufcnt. The cost rounds each niobuf up to whole pages. This
model counts the partial pages at the fragment boundaries. The OSS
pins those pages.

The following command will start a rule in BPS mode:
nrs_tbf_rule="start r1 projid={100} bps=5MB/s"
nrs_tbf_rule="start r1 projid={100} bps=5MB"
nrs_tbf_rule="start r1 projid={100} bps=5MiB/s"
nrs_tbf_rule="start r1 projid={100} bps=5MiB"
nrs_tbf_rule="start r1 projid={100} bps=5"
nrs_tbf_rule="start r1 projid={100} rate=5MiB/s"

The default unit for the BPS mode is MiB/s.
And it can also set rate explicitly in IOPS mode:
nrs_tbf_rule="start r1 projid={100} iops=1000"

A rule can also limit both the IOPS and the BPS dimensions:
nrs_tbf_rule="start r1 projid={100} iops=1000 bps=5MB/s"

The rule dump appends a "bps=" field to the line of a rule in BPS
mode. The prefix of the line does not change. A tool that reads the
dump of an IOPS rule therefore keeps its behavior.

A start command that pairs "realtime=" with "bps=" returns
-EOPNOTSUPP. The BPS dequeue path reads neither the real time flag
nor the token depth, so the pair has no meaning. A change command
with "rate=" on a BPS rule replaces the byte rate with an IOPS rate.
A change command also accepts the "iops=" and "bps=" keys.

A rate of zero, or a rate at or above LPROCFS_NRS_RATE_MAX, now
returns -ERANGE. The old code returned -EINVAL for both.

A new test case sanityn/77u{a,b,c,d} is added to verify it works as
expected. Test 77uc asserts the -EOPNOTSUPP refusal. Test 77ud
asserts the rate replacement of a change command.
We set the following rules to limit the read BPS to 5 MiB/s
and write BPS to 15 MiB/s:
"start nidrd nid={0@lo}&opcode={ost_read} bps=5"
"start nidwr nid={0@lo}&opcode={ost_write} bps=15"
The I/O size is varying from 4KiB to 1MiB, the performance is
shown as follows (The unit is MB/s):
blksz	4KiB	64KiB	128KiB	256KiB	1MiB
write	5.7	14.6	14.8	15.3	15.7
read	4.6	5.0	5.1	5.2	5.3

Test-Parameters: trivial testlist=sanityn env=ONLY=77
Signed-off-by: Yingjin Qian <qian@ddn.com>
Change-Id: I359e23faedd2a09d3249c1a36038396d9780293a

LU-18461 layout: join file implementation.

The "join file" feature in S3 is similar to the "merge" feature that was
available in previous versions of Lustre. This feature allows users to
combine multiple files into a single file, with the merged file being
placed at the end of the original files.

This functionality is useful for S3's multi-part upload process, as it
allows users to upload individual parts of a file (known as "chunks")
separately, and then merge them into a single, complete file without having
to copy the data multiple times. The "zero-offset" feature used in this
process allows each chunk to be stored as a separate, movable object, and
then merged together as components of the final file.

Signed-off-by: Alexey Lyashkov <alexey.lyashkov@hpe.com>
Change-Id: Ia2accfd843aa362edc4dc51a114ee3ef47282439

LU-19916 obd: rename OBD_CONNECT_ATTRFID to OBD_CONNECT_FTM

Clients have assumed OBD_CONNECT_ATTRFID since 2.11 (LU-8402) and no
server has checked it since, so stop requesting and requiring it and
give the bit its next use, the fault tolerant MGS of LU-19916.

MDTs keep granting it until 3.1.53, since a pre-2.18 client
still lists it in CLIENT_CONNECT_MDT_REQD and will not mount without
it.  It is granted back only when the client asked for it: obd_connect()
asserts that what a target grants is a subset of what was requested.

Test-Parameters: testlist=sanity clientversion=2.17
Test-Parameters: testlist=sanity serverversion=2.17 env=SANITY_EXCEPT=27D
Signed-off-by: Oleg Drokin <green@whamcloud.com>
Assisted-by: ClaudeCode:Opus-5 llm_code_and_review_tools
Change-Id: I69fe531cab83aaa978a2a3be39618aec9383276e

LU-20281 tests: skip sanity/27Ke on read-only root

if a test system uses read-only root, then you can't modify
/etc/passwd, /etc/group -- it's OK to jus skip the test then.

Test-Parameters: trivial
Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I0973e6be0375bb715e661dd2013cacbae1e98f8c

LU-17000 target: handle short IO write copy properly

Verify lnb_bytes fits in RMF_SHORT_IO buffer in tgt_shortio2pages().
Reject the request if not, the way the read side tgt_pages2shortio()
already does, and use unsigned lengths while we are at it.
A malformed request cannot be fixed by resending it, so reply with
an error rather than dropping the reply, and log the lengths so that
a client sending one leaves a record on the server.

Test-Parameters: trivial clientversion=2.17 testlist=sanity
Fixes: 4c3864cf97 ("LU-12600 tgt: shortio size should be unsigned")
Signed-off-by: Oleg Drokin <green@whamcloud.com>
Assisted-by: ClaudeCode:Opus-5 llm_code_and_review_tools
Change-Id: I5fa8f0c58a040f2c689fb74ca0efb6abd61f4053

LU-20523 target: don't scale cached statfs by tgd_blockbits

tgd_blockbits is sampled once in tgt_init() and never updated, but
os_bsize is not fixed for the life of a mount: osd_recordsize_changed_cb()
follows ZFS_PROP_RECORDSIZE, so "zfs set recordsize=" on a mounted target
moves od_max_blksz and with it the units of os_blocks/os_bfree/os_bavail.

Commit 8ee6e1c8825c ("LU-14565 ofd: Do not rely on tgd_blockbit") fixed
the statfs reporting paths in mdt_statfs() and ofd_statfs(), which is why
df and lfs df survive a recordsize change.  The grant code was not
covered and still pairs the cached tgd_osfs counts with the old shift in
tgt_grant_space_left(), tgt_grant_sanity_check(), tgt_grant_create(),
tgt_grant_commit(), tgt_statfs_internal() and tgt_grant_statfs().

Both directions are harmful.  Lowering the recordsize from 128K to 16K
multiplies the block counts by 8 while tgd_blockbits still says 128K, so
tgt_grant_space_left() reports eight times the space the target actually
has; tgt_grant_alloc() bounds grant by that value, so a target close to
full hands out grant it cannot honour and the clients hit ENOSPC on
writeback for space they were promised.  Raising it divides the counts
instead, and tgt_grant_sanity_check() then computes a maxsize smaller
than the grant already outstanding and LBUGs the server on the next
statfs.

Convert the cached counts with the os_bsize that came with them, and
read both under tgd_osfs_lock in tgt_grant_sanity_check() and
tgt_grant_create(), which took the counts without it: the size and the
counts have to come from the same statfs or a racing refresh simply
reintroduces the mismatch.  The tgt_grant_create() precreate check is
done in bytes, where it no longer wraps once the export's grant exceeds
the free space, and its trace prints the free space, that grant and the
device size instead of a product of two block counts.

The sites that use tgd_blockbits as the device's allocation unit - the
extent tax, the grant chunk rounding, tgt_grant_inflate()/deflate() and
ocd_grant_blkbits - are left alone: those are what was negotiated with
the connected clients and must not move underneath them.

This is the first of two patches on the ticket.

Rename test_805 to test_805a.

Add sanity test_805b, which emulates the recordsize change with
OBD_FAIL_TGT_STATFS_BSIZE so it also runs on ldiskfs, and compares the
free space the grant code computes against what df reports, with a
reported block size of 1K and of 1M - below and above both the ldiskfs
block size and the 128K a ZFS MDT is formatted with.

Fixes: bc9eae65f088 ("LU-6152 osd-zfs: ZFS large block compat")
Signed-off-by: Oleg Drokin <green@whamcloud.com>
Assisted-by: ClaudeCode:Opus-5 llm_code_and_review_tools
Change-Id: I03ff09d3218c3527c49e129b1814c72eaf747f93

LU-20523 target: don't LBUG when the device shrinks

tgt_grant_sanity_check() panics the server whenever the space granted
to a client exceeds the size of the backing device, assuming that such
a value can only come from counter corruption.  The device size is not
an invariant: a ZFS dataset quota can be lowered at runtime, and
os_blocks is (used + avail), so the reported total collapses to little
more than the used space.  Space granted while the device was large is
still legitimately outstanding, and the next statfs takes the server
down:

  lustre-MDT0000: cli 2665b580.../... ted_grant(29622272) +
                  ted_pending(0) > maxsize(6946816)
  (tgt_grant.c:209:tgt_grant_sanity_check()) LBUG

sanity test_805a does exactly this to the MDT: it lowers the MDT dataset
quota to (used + 5MB) to reach ENOSPC quickly, so any client holding
DoM grant from earlier tests panics the MDS - in test_805a itself, or in
a later test whose statfs runs the check (test_807a).

This is not new: the same panic has been recorded in test_805a (and in a
later statfs of test_806/807a) since 2022-01, on master, b2_15 and in
review runs, with per-export grants from 3MB to 355MB against a maxsize
of 1MB to 64MB.

Compare against the largest device size seen since mount rather than
the current cached one, so grants that were legitimate when they were
issued are not mistaken for corruption, while corrupt counters are
still caught.

Only the fatal check moves: the aggregate warnings at the end of
tgt_grant_sanity_check() keep comparing against the size the device
reports now, so an over-committed target still says so in the log
instead of going quiet - they just repeat now, rather than being cut
short by the LBUG() on the first statfs.  CERROR() is CDEBUG_LIMIT() on
D_ERROR, so repeating does not flood the console.

The sample is converted with the os_bsize that came with it, which is
what the preceding patch made tgt_grant_space_left() use as well, so the
recorded size and the space the allocator is willing to hand out stay in
the same units across a ZFS recordsize change.

The check stays fatal.  What was wrong is the threshold, not the
reaction to crossing it: gating these LBUG() calls on
lbug_on_grant_miscount would also stop catching the genuine counter
corruption the check exists for, which is a separate decision.

Add sanity test_805c, which takes DoM grant on the MDT and then shrinks
the size reported to the grant code with OBD_FAIL_TGT_STATFS_SHRINK.
The shrink is injected rather than done with "zfs set quota" as test_805a
does, so that the case also runs on an ldiskfs MDT; the code being fixed
is target-generic and ldiskfs is what most of CI runs.

Fixes: 71d68757c76c ("LU-1406 ofd: grant support")
Signed-off-by: Oleg Drokin <green@whamcloud.com>
Assisted-by: ClaudeCode:Opus-5 llm_code_and_review_tools
Change-Id: I3df1560b83bb3c76f89dc11c9281ed1f7d4116bf

LU-19602 osc: account LRU pages on the page's home OSC

The per-OSC LRU counters (cl_lru_busy/cl_lru_in_list) intermittently
underflow and LBUG the client in interop configs (master client / b2_15
server) running FLR fsx workloads. Proven with a per-page accounting
tracker: at the underflow the page's bucket is correct but it was
counted on one OSC's client_obd and de-counted on another's.

osc_lru_del() (page teardown) always operates on the page's home
osc_object, osc_cli(osc_page_object(opg)) - it is the only object
reference available at destroy time. But the increment side keyed off
the *IO's* object instead: osc_page_cache_add() passed the committing
sub-io's object (cl2osc(ios->cis_obj)) to osc_lru_use(), and
osc_lru_add_batch() used the finishing extent's object. In normal
operation these are the page's home. Under a mid-IO FLR layout-version
change (a b2_15 OSS bounces -EINPROGRESS on OST_PUNCH so the client
resends and re-resolves the layout; master's ofd_verify_layout_version
no longer does this since LU-14642) the committing sub-io can target a
different mirror component than the page's home, so a page is counted on
component B and removed on its home A: A underflows, B leaks. The damage
is not only arithmetic: osc_lru_use() unlinked the page from A's
cl_lru_list while holding B's cl_lru_list_lock, racing A's own shrinker
over that list.

Make the increment side use the page's home OSC too, matching the
decrement side, so a page lives on exactly one OSC's LRU list and its
counters cannot split. osc_lru_use() and osc_lru_del() derive the home
internally, leaving no caller able to name an OSC at all;
osc_lru_add_batch() accounts each page on its home, keeping the batched
fast path for the common case and handling a cross-OSC page
individually. That page needs its own shrinker wake-up, since the
batched one only ever covers the extent's own OSC, and gets a
rate-limited warning so the mismatch stays visible now that it no longer
announces itself as an LRU underflow. osc_lru_alloc() already uses the
home (page-init object).

Grant needs no equivalent change: osc_queue_async_io() reserves it on
the IO's OSC and osc_extent_finish() releases it on that same extent's
OSC, so it cannot split. The LRU is asymmetric because its release site
is page destruction, where the IO is gone and the home is the only
object reference left.

This is EC/parity safe: each osc_page (data or parity) has a single home
component and is accounted there regardless of which sub-io commits it.

The interop runs below are narrowed to sanity-flr test_70a, the FLR fsx
workload that reproduces this bug, because the full suite gives no
usable interop signal: test_0a fails on any master client against a 2.15
or 2.16 server since LU-19826 moved the mirror-count limit to a
server-side lod.*.mirror_count_max tunable that exists on neither, and
the run then dies in test_21 on a bulk-transport failure between the
master client and the old OSS (server_bulk_callback status -113,
target_bulk_io rc = -110) after which the OSTs evict the client and the
write retries forever.

Test-Parameters: testlist=sanity-flr serverversion=2.15 env=ONLY=70a
Test-Parameters: testlist=sanity-flr serverversion=2.16 env=ONLY=70a
Fixes: 1d8e1b31d547 ("LU-3321 osc: Adjustment on osc LRU for performance")
Fixes: 4c98a76dea48 ("LU-13814 osc: remove "osc_page_transfer_add" wrapper")
Signed-off-by: Oleg Drokin <green@whamcloud.com>
Assisted-by: ClaudeCode:Opus-5 llm_code_and_review_tools
Change-Id: I939cb69cff3b223073fedfec38ab9a7609a31f22

LU-19932 osc: allow killing a write waiting on an extent

A process writing to a file can block indefinitely in
osc_extent_wait() while an extent it needs is in flight to an
unresponsive server, with no way to kill it.  Make that wait abortable
on the two paths that queue a page, osc_extent_find() and
osc_queue_async_io(), which hand the error back to the writer.

The wait stays uninterruptible wherever the caller goes on to flush,
discard or truncate the pages, which must not happen on top of an
incomplete drain: osc_cache_wait_range() for osc_lock_flush() and the
fsync drain, osc_cache_writeback_range(), osc_object_invalidate(), and
the truncate handshake between osc_extent_release() and
osc_cache_truncate_start(), which has to reach OES_TRUNC before
osc_extent_truncate() asserts on it.  That is what LU-2779 made this
wait uninterruptible for.  osc_queue_dio_pages() stays uninterruptible
for its own reason: returning early would leave its pages counted in
the DIO sync anchor.

The hang reported in LU-19932 is in that fsync drain and is not one of
the ones this makes killable.  Making the drain killable would not fix
it either: the writer returns into filemap_write_and_wait_range(),
which then waits uninterruptibly in folio_wait_writeback() for the
same pages, since cl_page_make_ready() marks every page of a write RPC
PG_writeback.  Measured against an OST stalled with
OBD_FAIL_OST_BRW_PAUSE_BULK, SIGKILL does not free a writer from that
second wait.

Only the second wait becomes abortable, the one entered after the
initial 600s wait has already reported the extent as stalled.  An
ordinary wait for an extent that is merely still in flight must not be
cut short by a SIGALRM or a SIGINT.

An mmap store reaches osc_queue_async_io() through page_mkwrite(),
which maps an interrupted wait to VM_FAULT_SIGBUS.  Use a killable
wait for mkwrite so a caught SIGTERM cannot cause an unexpected
SIGBUS. After the initial 600s timeout, fatal signals can still abort
this wait; caught signals remain pending until the wait completes.
Normal writes retain the abortable wait.  This avoids introducing a
refault, which needs the kernel-mode fault case worked out: a fault
taken from inside a syscall does not dequeue the signal and would
re-execute rather than end the syscall.

Release the current page's dirty-cache reservation if the wait in
osc_queue_async_io() fails.  The page has not joined the extent yet,
so RPC completion cannot release its per-OSC and global dirty-page
accounting.  This also covers the existing RPC-error exit.

osc_extent_wait() returns only 0 or a negative errno.  The leftover
jiffies from wait_event_idle_timeout() must not reach the callers:
osc_queue_async_io() only calls osc_extent_find() when rc is zero, and
on a non-zero rc walks the extent pointer it has just released.

Simplify ldlm_completion_ast() to only call l_wait_event_abortable()
once instead of in two separate branches to avoid code duplication.

Simplify osc_extent::oe_rc usage to only hold 0 (success) or a
negative errno.  Storing the number of pages there isn't used for
anything, and it complicates its usage elsewhere.

Fixes: 26345bee6b04 ("LU-2779 osc: osc_extent_wait() shouldn't be interruptible")
Signed-off-by: Andreas Dilger <adilger@thelustrecollective.com>
Signed-off-by: Oleg Drokin <green@whamcloud.com>
Signed-off-by: Vitaliy Kuznetsov <vkuznetsov@ddn.com>
Assisted-by: ClaudeCode:Opus-5 llm_code_and_review_tools
Change-Id: I140bdd6cdea3ccd985b457ee4b6bf415a542c669

LU-16832 lnet: add nanosecond delay latency support

Introduce nanosecond-resolution latency for LNet delay
fault injection. The previous interface accepted latency
only in seconds (u32). This change promotes the field to
u64 nanoseconds and plumbs it so that sub-second delays
(s/ms/us/ns) can be expressed precisely. Since this
changes the Netlink output we bump the LNet netlink
family version. The user land code is updated to accept
and process the new latency_ns option for the non ioctl
approach. The internal timer is moved to hrtimer for
finer level of time resolution.

Test-Parameters: trivial
Signed-off-by: Sohei Koyama <skoyama@ddn.com>
Change-Id: I9ed1bd34cef0f0bf647f87d17dd68f05274e2b28

LU-19712 utils: Add 'lfs pool pin/unpin/pin_info' commands

Introduce lfs pool commands to manage pool pinning via lustre.pin
xattr.  Three new commands added: "lfs pool pin" to pin
files/directories or FIDs to an OST pool, "lfs pool unpin" to unpin
files/directories or FIDs from an OST pool, and "lfs pool pin_info" to
report whether files/directories or FIDs are pinned to an OST pool.
"lfs pool pin" and "lfs pool unpin" take the pool name as an optional
argument: when omitted, it is inferred from the primary mirror layout
of each file or directory. "lfs pool pin_info" requires --pool
explicitly, since a query has no layout-derived default to fall back
to. They also support operating on FIDs via --fid option.

Add the lfs-pool-pin.1, lfs-pool-unpin.1 and lfs-pool-pin_info.1 man
pages documenting the new commands, extend lfs-pool.1 to list them,
and wire them into Documentation/man1/Makefile.am and lfs.1.  Also add
bash-completion support for --pool/--not-pool pool-name completion,
and cover the new commands in sanity.sh test_157c.

Also extend racer's file_setxattr.sh to exercise "lfs pool pin/unpin"
against a real pool name on about one iteration in ten (governed by
RACER_PIN_FREQ) when running as root, alongside its existing random
user.* xattr stress testing, and switch the pre-existing hsm_cmdlist
comment to the same kdoc-style /** block as the new pool_cmdlist one
just above it, for consistency.

Signed-off-by: Nikos Papakonstantinou <npapakonstantinou@ddn.com>
Change-Id: I23d3679713702f83eb2faa0ab47b7e5a01079831

LU-20434 build: strncpy() to strscpy() non-obvious cases

Some cases of strncpy() want to overwrite some bytes
of the destination and do not want to truncate the destination.

mdt_path_current() and mgs_replace_mdtname() use memcpy() to
overwrite segments.

mdd_xattrs_migrate_prep() can use strscpy().

nodemap_test_nid() should use strscpy_pad() as it is always passed
the sizeof() the buffer and the zero padding is needed.

Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Change-Id: I8b70127fb5e4f870ff856bb2344f402eb2bd5f68

LU-20434 build: strncpy to strscpy simple cases

Most users of strncpy want a nul terminated destination.
These case be transformed easily to use strscpy().

In obd_str2uuid() use snprintf() to copy and calculate the
bytes written and number of bytes of padding needed for
memset().

In obd_uuid2str() use memcpy() instead of strncpy()

In obd_uuid2fsname() strncpy can be replaced with snprintf()

In osd_get_name_n_idx_compat() and osd_get_name_n_idx() buf is passed
to zap_lookup() which treats the value a nul terminated string.

In osd_scan_dir() ozi_name is treated as nul terminated string.

In upcall_cache_set_upcall() uc_upcall is passed as a command line
argument and should be nul terminated.

In check_gssd_socket() sun_path should be a nul terminated string.

In qmt_pool_alloc() qpi_name should be a nul terminated string, it is
treated as string in some places.

In server_put_super() extraname must be a nul terminated string.

Test-Parameters: trivial
Fixes: e95e4d3227f ("LU-10186 osd-zfs: move LAST_ID OI mapping out of oi.xx")
Fixes: f2d44192278 ("LU-7898 osd: remove unnecessary declarations")
Fixes: 0c1ae1cb9c1 ("LU-13124 scrub: check for multiple linked file")
Fixes: 7869bb320e7 ("LU-17718 obdclass: potential string overflow upcall_cache.c")
Fixes: 8d828762d18 ("LU-17015 gss: support large kerberos token for rpc sec init")
Fixes: f6819c90c85 ("LU-11023 quota: remove quota pool ID")
Fixes: eba2a78067e ("LU-6600 obdclass: race lustre_profile_list")
Signed-off-by: Shaun Tancheff <shaun@tancheff.com>
Change-Id: Id397f4d9a4812d25f530f00926a5f2906d91f06a

LU-18838 statahead: fix refcounting on statahead error paths

ll_ioctl_ahead() unwound every failure through a single out: label and
told the cases apart with "if (fd_sai)" / "if (sai)" / "if (ctx)" tests.
Two things were wrong with that.

First, a use-after-free of @ctx. When ll_sax_get() finds no context and
another task installs lli->lli_sax with a non-ADVISE pattern before this
one takes lli_sa_lock -- "lfs ladvise -a ahead" racing an "ls -l" on the
same directory -- the freshly allocated context is released with
ll_sax_free(), but @ctx still points at it and out: then ran
ll_sax_put(ctx) on freed memory, reading ctx->sax_inode and, at refcount
zero, iput()ing and freeing it a second time. Note that the release here
has to stay ll_sax_free(): the context was never published in
lli->lli_sax, and ll_sax_put() at refcount zero clears lli_sax,
lli_sai and lli_sa_pattern -- which at this point belong to the context
the racing task just installed.

Second, ll_ioctl_ahead() and start_statahead_thread() both increment
sbi->ll_sa_running once, and both released @sai on error with
ll_sai_put(), which itself decrements ll_sa_running when it drops the
last reference to @sai. The error paths then also called an explicit
atomic_dec(&sbi->ll_sa_running), decrementing a second time for the same
increment. Under racer load kthread_create() fails repeatedly with
-EINTR, so this fires often and drives ll_sa_running negative (observed
-14 and -1 in a timeout vmcore). That turns statahead off for good on
that mount, rather than merely miscounting: ll_sa_running_max is an
unsigned int, so in

	if (unlikely(atomic_inc_return(&sbi->ll_sa_running) >
				       sbi->ll_sa_running_max))

the signed side is converted, a counter of -14 increments to -13 and
compares as 4294967283, and every later statahead attempt is refused
with -EMFILE -- decrementing straight back to -14 on the way out.

Give ll_ioctl_ahead() one label per resource -- out_fd:, out_ctx:,
out_sai:, out_dec: -- so each failure jumps to the point matching what it
has actually acquired and no conditional cleanup is needed; the accounting
then falls out of the control flow, with out_dec: reached only before
@sai exists and ll_sai_put() doing the decrement everywhere else. In
start_statahead_thread(), which already distinguishes the two cases with
@fail_free, move the decrement into the fail_free branch, the one that
frees @sai with ll_sai_free() and so has nothing else to account for it.
The "if (rc)" it was guarded by goes with it: out: is only ever reached
with a non-zero rc, since every GOTO sets one and the success path
returns -EAGAIN before the label.

Test-Parameters: optional testlist=racer
Fixes: 85fe3cfff866 ("LU-17625 statahead: avoid to use @sai after its has been freed")
Fixes: 1288681bb8aa ("LU-14361 statahead: add statahead advise IOCTL")
Signed-off-by: Oleg Drokin <green@whamcloud.com>
Assisted-by: ClaudeCode:Opus-5 llm_code_and_review_tools
Change-Id: Ia0bb57db9275f589ad48d631ee6f8184927e8112

LU-6301 llite: clear rq_replay in ll_open_cleanup

ll_open_cleanup() closes the MDT-side handle of an open the client
abandoned after the MDT had granted it, but leaves the open request
itself replayable. Nothing ever clears rq_replay for such a request:
it has no och and therefore no mdc_commit_open() commit callback, so
once the transaction commits ptlrpc_free_committed() moves it to
imp_committed_list, where it stays -- pinning a reply buffer sized to
max_mds_easize, hundreds of KB with wide/PFL/DoM layouts -- until the
next reconnect. A workload that takes the open-error path often enough
grows that list without bound and can OOM the client.

Clear rq_replay before sending the close, exactly as mdc_close() does
for a normal close, and drop the request from the replay list with
ptlrpc_request_committed() afterwards, as mdc_free_open() does. Clearing
it after md_close() would be too late: after_reply() for the close runs
ptlrpc_free_committed() on the same import, and if the open has committed
by then the request is moved to imp_committed_list with rq_replay still
set -- and that list is only rescanned when the import generation changes.
Both happen even if op_data cannot be allocated and no close goes out,
since that is the low-memory case this is meant to help.

Make ll_open_cleanup() self-guarding while here. It takes the
lookup_intent instead of a req_capsule and does nothing unless
DISP_OPEN_OPEN is set, i.e. the MDT granted an open and the reply holds a
handle to release. Which reference on that reply it may use then depends
on how the open ended. With DISP_ENQ_OPEN_REF set the intent still owns
the reference mdc_finish_intent_lock() took, so it->it_request is safe.
Without it, and with the open itself successful, ll_file_open() or
ll_release_openhandle() has taken the handle over and cleared the bit; an
och owns the handle (and with it rq_early_free_repbuf) and the request
may already be freed -- on a non-replayable open, such as one against a
read-only export, mdc_finish_enqueue() clears rq_replay, nothing retains
the request, and their ptlrpc_req_put() was the last reference. Without
it and with the open failed, the bit was never set at all, because
mdc_finish_intent_lock() only sets it when it_open_error(DISP_OPEN_OPEN)
is zero; the MDT can install the mfd in mdt_finish_open() and then fail
the request in mdt_refetch_lovea(), and that handle still has to be
released. Nothing retains the reply there either -- mdt_reint_open()
zeroes the transno on error -- so the caller's own reference is the only
one, and callers that have one pass it in. ll_lookup_it_finish() and
ll_prep_inode() do; ll_atomic_open()'s out_release passes NULL, since
ll_lookup_it() has already dropped it by then. DISP_OPEN_OPEN is cleared
once the close has been issued, so a second cleanup of the same abandoned
open is a no-op; DISP_ENQ_OPEN_REF must survive for ll_intent_release()
to drop the reference.

ll_atomic_open() gains a cleanup call on out_release, which previously
only released the intent. That label is reached both by the error exits
and by the two finish_no_open() branches, which return 1 and leave the
VFS to open the dentry itself; either way the handle granted for this
intent is abandoned, and until now it leaked until umount or eviction.
It passes no request of its own, since ll_lookup_it() has already
dropped its reference by then, so it covers the intents that still carry
DISP_ENQ_OPEN_REF and not an open the MDT granted and then failed --
that one arrives here through ll_file_open()'s out_och_free with the bit
already clear, and freeing it would need ll_lookup_it() to hand the
reply reference back. As at the other two call sites, the intent lock is
dropped first, since the close waits for a mod RPC slot and holding the
lock across that wait can get the client evicted (LU-13101).

The leak is reachable today via ll_prep_inode() and
llcrypt_prepare_readdir() failures. It becomes
easy to hit with "LU-9868 llite: use d_splice_alias for directories",
which lets ll_splice_alias() fail with -ELOOP or -ESTALE where the
current d_add() cannot: a racing rename/create workload then drives
ll_lookup_it_finish() into this error path thousands of times per run.

Fixes: d2b203692e24 ("LU-6301 llite: cleanup open handle for client open failure")
Signed-off-by: Oleg Drokin <green@whamcloud.com>
Assisted-by: ClaudeCode:Opus-5 llm_code_and_review_tools
Change-Id: Iea44872f5507d829c1baa791ed179ee0f13f09c1

LU-20501 llite: do not derive the dirent from the emitted name

ll_get_name() recovers the directory entry it is examining with
container_of() on the name pointer its filldir gets, which is only valid
while that name points into the lu_dirent.  ll_dir_read() has not
guaranteed that since encrypted directories were added: for those it
emits the name out of the separate llcrypt_fname_alloc_buffer()
allocation, so the derived "lu_dirent" starts 32 bytes before an
unrelated slab object and lde_fid is read out of bounds.

Tracing the two pointers on an encrypted directory shows the derived
entry landing at name - 0x20 for every entry, with the FID read back out
of the neighbouring allocation -- in one run
[0xffff998306137fe0:0x6137fe0:0xffff9983], a kernel pointer being
compared as a FID.  A plain directory in the same mount never diverges.

Pass the lu_dirent being emitted down from ll_dir_read() so the filldir
works from the entry itself rather than from pointer arithmetic on a name
it does not own.  The whole entry rather than just its FID, because the
companion llite patch also has to read lde_attrs to tell whether lde_fid
means anything.  The pointer is live only for the duration of the
dir_emit() call, which is where its only consumer runs; the folio holding
it is unmapped once ll_dir_read()'s loop is done.

Drop the (filldir_t) cast on .ctx.actor while here:
ll_nfs_get_name_filldir() already has that prototype under both
FILLDIR_TYPE variants, so the cast only hid the type check, and no other
.actor assignment in the tree casts.

This does not fix the failure LU-20501 was filed for.  ll_dir_read()
emits ent->lde_name for a plain directory, so container_of() already
recovered exactly the entry this patch passes down; only the
IS_ENCRYPTED() path changes behaviour, and sanityn test_102 (the existing
open-by-handle test) uses unencrypted directories.  This fixes the
out-of-bounds read only.

NFS export of an encrypted directory works after this only when the
parent inode is already warm on the mount doing the reconnect.
ll_get_name() calls ll_dir_read() without llcrypt_prepare_readdir(),
which ll_iterate() does before the same call, so the crypt info has to
have been loaded by something else -- search_inode_for_lustre() starts
with an ilookup5() and llcrypt_setup_filename() loads the info for every
lookup inside the directory.  Measured on a dummy-encrypted mount: with
the parent walked first, check_fhandle_syscalls passes 3/3; with
drop_caches and the LDLM LRU cleared so the inode is instantiated cold,
0/3, because ll_fname_disk_to_usr() then emits the no-key base64 form
that reconnect_path()'s lookup_one_len() cannot resolve.  Adding
llcrypt_prepare_readdir() there returns -ENODATA on a cold
NFS-instantiated inode, so the cold case needs its own fix.

sanityn test_102b covers it: it remounts both clients with
test_dummy_encryption, walks the directory on the second mount so the
parent's crypt info is loaded, and resolves a handle for a subdirectory
inside it.  It fails without this patch ("check_fhandle_syscalls on an
encrypted dir failed") and passes with it.  The cold-inode case is left
uncovered on purpose, since it still fails for the reason above.

The new test goes in beside a bare test_102 rather than renaming that one
to test_102a here, so that autotest's review-*-subtest-change groups do
not put test_102 -- which fails on ZFS DNE roughly one session in ten on
master, the failure this ticket is about -- into a 30-minute repeat loop
against a patch that does not fix it.  The companion llite patch, which
does fix it, carries the rename.

Fixes: 4d38566a004f ("LU-13717 sec: filename encryption")
Signed-off-by: Oleg Drokin <green@whamcloud.com>
Assisted-by: ClaudeCode:Opus-5 llm_code_and_review_tools
Change-Id: Ic0380a640740ced90acd41004f8ff9652f25b8bd

LU-19848 compat: move GDS support to lustre_compat/

The upstream kernel can not depend upon an out-of-tree
driver. Hence, move the legacy GDS support to
lustre_compat/ in preparation for p2pdma support.

Test-Parameters: ignore
Signed-off-by: Timothy Day <timday@amazon.com>
Change-Id: I81573a06b42a5daa70691f93ba6fe89128d07103

LU-19768 obd: Create new framework for obd_device based Netlink

The netlink API is very complex and each implementation has
repeated code. Create a framework to make implementation
easier and this allows more code sharing. To demostrate this
new API move the target_obd to this new framework.

This new framework actually uses the struct genl_info for its
family field which exposed some issues with older kernels.
Those issues have been fixed.

New features:

1) Enforce setup of min_dump_alloc so we never have sk_buff
   overflow. Common sizes like OBD_GENL_HDR_SIZE and
   OBD_GENL_SOURCE_SIZE are done for you. ctx->setup_keys()
   is used to do implementation specific sizing calculations.

2) Besides ensuring the obd still exist only collect obd
   devices that are attached, set up and not stopping.

3) Each source attribute is per obd device so only emit
   when the obd changes, not once per message.

4) Use Xarray instead of genradix since it has built in
   cursor handling.

5) Use of obd device's obd_minor value instead of the
   obd location string.

Test-Parameters: trivial
Signed-off-by: James Simmons <jsimmons@infradead.org>
Change-Id: I06931716758cb02b353ba3ec37d08a73fb332c69

LU-19712 utils: Add only and not-pool options to lfs pool pin

Introduce --only and --not-pool options to lfs pool pin. The --only
option makes the pin exclusive to the set of pools the target is
already positively pinned to (including the one this operation itself
pins), while the --not-pool option prevents the file from being
migrated or mirrored to the specified pool.

Also add a matching --not-pool option to lfs pool unpin, via a new
llapi_pool_undeny_fd(), to remove a deny rule without disturbing a
same-named positive pin (a pool can never be both, since --not-pool on
pin already refuses to deny an already-pinned pool). --not-pool always
takes an explicit pool name; unlike --pool, it is never inferred from
the target's layout, since a denied pool cannot appear there.

Also rename verify_pin_xattr_object(), dump_pin_object(),
read_pin_xattr_object() and read_pin_xattr_object_fd() to their
llapi_-prefixed equivalents, since these are symbols exported from the
shared library and the original unprefixed names were too generic for
that; --only/--not-pool themselves go through the new
llapi_pool_pin_only_fd()/llapi_pool_deny_fd()/llapi_pool_undeny_fd()
entry points instead.

Signed-off-by: Nikos Papakonstantinou <npapakonstantinou@ddn.com>
Change-Id: Icc35bf87444edd74c1fcda927cead5e3b34a0fcd

LU-15373 llog: rework changelog_remap_rec()

Originally, changelog_remap_rec() expanded missing changelog fields,
but now we use it only to trim unsupported fields for remote clients.

So the fields need to be move in-order from the first record to last
one to avoid overwrite.

Moreover, changelog_remap_rec() is not trivial, so adding new
changelog field can be tricky.

This patch renames & improves changelog_remap_rec() to
changelog_trim_rec(). llapi_changelog_repack_rec() is updated to
reduce their dependency on field types.

Add a regression test sanity 160y to check if changelog_trim_rec() is
able to trim a field in the middle of the record.

Fixes: a98690be90 ("LU-5862 changelog: Proper record remapping")
Test-Parameters: clientversion=2.15 testlist=sanity env=ONLY=160
Test-Parameters: serverversion=2.15 testlist=sanity env=ONLY=160
Test-Parameters: clientversion=2.15 testlist=sanity-hsm env=ONLY=222
Test-Parameters: serverversion=2.15 testlist=sanity-hsm env=ONLY=222
Signed-off-by: Etienne AUJAMES <eaujames@ddn.com>
Change-Id: I2a25e8131304e985187cb2c644822d1946455bf9

LU-19917 mgc: dynamic MGS NIDs discovery

Enable fault tolerant MGS on the client: the client MGC can connect to
any server running an MGS follower, and it learns about those servers
on its own.

. the client mount command takes a list of MGS NIDs, shuffled so that
  clients spread the MGS load; a server MGC still tries the primary
  MGS first, in the order given.  mount.lustre(8) says so now
. the imperative recovery log adds every server NID to the client MGC
  import at the tail, so this works with no configuration while the
  NIDs from the mount line stay preferred.  That is independent of
  mgc.*.dynamic_nids, which covers the target imports
. the server MGC device is renamed "mgc<NID>", since it only ever
  connects to the primary MGS, while the client MGC stays "MGC<NID>".
  Userspace and the tests name the one they mean, and it is now what
  "lctl dl" and "lctl get_param mgc.*" show on a server
. a client MGC pings at max(PING_INTERVAL_SHORT, obd_timeout / 20),
  capped at PING_INTERVAL, so it notices a dead MGS in time to not
  miss an IR update.  At the default obd_timeout that is 7s rather
  than 25s, on client MGC imports that negotiated OBD_CONNECT_FTM
. a follower builds the fsdb on the first MGS_CONFIG_READ when the
  filesystem is in its CONFIGS directory, and serves the nidtbl it
  has.  An MGS whose nidtbl is still empty answers -EAGAIN, on the
  leader too: the version is persisted but the target list is not, so
  an empty table would claim a version the client never got entries
  for, and -ENOENT would make the client stop asking for good
. the IR log parser reports why it stops instead of breaking silently.
  Its "datalen < 0" test after the subtraction goes, the bound just
  above it already covers that, and the revoke that ends the log drops
  its IS_SERVER() guard because the rename leaves a client MGC unable
  to carry an observer at all
. a mount line names at most MGC_MGS_NODES_MAX MGS nodes, where it had
  no bound before; mount.lustre(8) says so and the rest are dropped
  with a console warning
. the tests that read a server MGC match either case, since the server
  they talk to may be an older one; an except file cannot express "the
  server has the rename", because that file is read from the server.
  An old client running its own recovery-small test_141 against a new
  server is the other way round and is excepted: its "*MGC*" globs
  match no server MGC namespace any more
. conf-sanity test_163 is new; test_160, test_161 and recovery-small
  set_ir_status() are adjusted for a node that can run both an MGS
  follower and a client MGC
. conf-sanity test_73e waits for the targets to re-register after it
  restarts the MGS, and needs LU-19692 on the server to do so: the
  client now mounts as soon as an MGS answers, so it no longer waits
  out the whole reconnect and would otherwise get an empty OSC import
. conf-sanity test_250 updates the MDS version check to match the
  actual version in which the test and functionality landed

Test-Parameters: mdtcount=4 testlist=conf-sanity serverversion=2.17
Test-Parameters: mdtcount=4 testlist=conf-sanity clientversion=2.17
Test-Parameters: mdtcount=4 testlist=recovery-small serverversion=2.17
Test-Parameters: mdtcount=4 testlist=recovery-small clientversion=2.17
Fixes: 86904ef943 ("LU-19740 utils: Restrict {tunefs,mkfs}.lustre to one device")
Fixes: f5e198dc68 ("LU-18951 utils: lctl lcfg_erase erase fsname params")
Signed-off-by: Lai Siyao <lai.siyao@whamcloud.com>
Signed-off-by: Oleg Drokin <green@whamcloud.com>
Assisted-by: ClaudeCode:Opus-5 llm_code_and_review_tools
Change-Id: Ib6d2b1ff6bce8d1a51a54982356c859448cf7f05

LU-19916 mgs: start MGS follower on MDTs/OSTs

Start an MGS follower on every MDT/OST, to serve the configuration it
has a copy of - the params and sptlrpc logs - to the clients attached
to it; imperative recovery is wired up in the next patch.  MGS0 is
the leader; MDTs and OSTs still connect to it, and a follower rejects
them with -ENODEV.  A combined MGS/MDT0 takes over from a follower
another target on the host already started.  The follower is optional:
an MDT or OST mounts without it.  Once the MGS is stopped the MGC
requeue thread starts a follower on another target if it can.

A target running a follower gets LDD_F_SV_TYPE_MGS in its lsi_flags,
so IS_MGS() covers it too, and IS_MGS0() is what now means "holds the
MGT".

A follower is a readonly backup of MGS0: no ioctls, and no nodemap
config file - it refuses a nodemap config read rather than serving one
it does not have.  Replicating the nodemap configuration, so that it
too survives the loss of MGS0, is left to a later patch, and so is
replicating <fsname>-client, which a follower has no copy of and
cannot serve, and _mgs-sptlrpc, without which a follower applies no
mgs.srpc.flavor.* restriction to the clients it accepts.

A follower writes no config log into CONFIGS/, which on a follower
belongs to the MDT or OST it runs on and holds that target's local
copy of the logs: building an fsdb there skips reading
<fsname>-client, which the target does not have and which
llog_open_create() would otherwise leave behind empty.  For the same
reason it leaves that target's lu_site cache alone, and only the
leader sends LCFG_PRE_CLEANUP, which would stop the OI scrub of a
target that is still serving.

"nomgs", "nosvc" and a read-only mount all keep the follower off a
target; the last so that an lsnapshot mount does not serve the
snapshot's configuration.

That target is already serving and the site is shared, so
mgs_fs_setup() no longer publishes itself as ls_top_dev and clears it
afterwards: that left the live MDT/OST with a NULL top device,
crashing the next mdt_object_find() or ofd_object_find().  Nothing
there needs it - local storage resolves through ls_locate() and the
root through dt_locate_at() - which is also why mgc_fs_setup() does
the same work without it.  For the same reason mgs_fs_setup() and
nidtbl_read_version() resolve their objects through the local storage
device rather than through the MGS layer, which would leave an
MGS-layer object pinned in a site only that target can purge.

The follower registers as an observer of the MGC so it hears about
config changes; obd_notify() takes a union obd_notify_data to carry
the payload.  It is unregistered before it is freed, since the MGC is
shared by every target on the node and outlives any one of them, and
the MGC holds the observer link for read across every notify:
unregistering takes it for write, so an unmount waits out a notify
already inside the MGS rather than tearing its fsdbs down underneath
it.

OBD_CONNECT_FTM, set only by a client mount, is what a follower
accepts a connection on.  The follow-on patch gives the server MGC its
own device name, so a client and a target on one node never share a
connection's flags.

server_mgs_lock() stays a separate call rather than being taken inside
server_start_mgs_or_follower(), because server_fill_super() holds it
across the follower start and the obd_register_observer() that follows
it, so the requeue thread cannot start a second one in between.  It
nests outside cld_lock, which cld_start_mgs() takes under it.

Add debugfs entry mgs/MGS/state, "leader" or "follower", which
is_mgs() in liblustreapi checks because a follower handles no ioctl.

conf-sanity test_5b starts its OST with "nomgs": it checks that a
client cannot mount with no MGS, and an OST that starts a follower
here has no MGS0 to feed it, so the OST's own MGC is refused by that
follower for as long as the test runs.  replay-single test_61d drops
CFS_FAIL_ONCE from its llog_setup failure, which a follower started
on another target of that node would otherwise spend before the mount
under test reaches it.

Skip conf-sanity test_104a during old client interop testing.

Test-Parameters: mdtcount=4 testlist=conf-sanity serverversion=2.17
Test-Parameters: mdtcount=4 testlist=conf-sanity clientversion=2.17
Test-Parameters: mdtcount=4 testlist=replay-single env=ONLY=61,ONLY_REPEAT=20
Fixes: 2924d62f60 ("LU-20214 osd-ldiskfs: no support for noacl in 6.0+")
Signed-off-by: Lai Siyao <lai.siyao@whamcloud.com>
Signed-off-by: Oleg Drokin <green@whamcloud.com>
Assisted-by: ClaudeCode:Opus-5 llm_code_and_review_tools
Change-Id: I7c4778763ec18ce11d513bc6e2b2cce51acb5a70

LU-20541 lfs: Implement migrate_stripe command and add test

Add lfs migrate_stripe to move OST objects of a file onto another
OST using a single-stripe volatile partner and llapi_fswap_layout_fids().
Accepts multiple FILE operands, migrating each one in turn with the
same options, to make draining many files off an OST convenient.

-i selects a source OST index rather than a stripe index: with
multiple FILE operands the stripe holding data for a given source
OST differs from file to file, so identifying it by index would not
make sense. Instead every stripe currently on the source OST is
migrated to the target OST, including repeats of the same OST within
one component or across components, unless -I restricts the scan to
a single component.

Copy follows absolute LOV stripe numbering and object offsets, keeps
holes sparse, and ftruncates the volatile so trailing holes still
carry file size without SOM. Match source uid/gid on the volatile for
mdd_swap_layouts(). Default path is optimistic (DV check at swap);
--lock-copy holds a group lock across copy and swap.

Clamp the copy end from st_size after reading dv_src (under the group
lock for --lock-copy) so a concurrent append cannot leave the new
stripe short while CHECK_DV1 still matches.

llapi_layout_comp_flags_get() always reports LCME_FL_INIT clear for
a plain (non-composite) layout, so skip that check unless the layout
is actually composite; otherwise every stripe of a plain-layout file
would be silently skipped as "uninstantiated".

Document in lfs-migrate_stripe.1. sanity-pfl 30-39 cover content,
holes, PFL offsets, size, absolute striping, root on user files,
past-EOF components, an OST reused multiple times in a component,
and migrating multiple FILE operands in one invocation. Test 39
covers migrate_stripe on a plain (non-PFL) layout.

Test 32 is skipped on ZFS as we do not yet have FIEMAP support. LU-1941

Test-Parameters: trivial
Signed-off-by: Ronnie Sahlberg <rsahlberg@whamcloud.com>
Change-Id: I48be7e597544c6856fdcf8edbd181b9f8d3db263

LU-19916 mgs: do not offer a target with no MGS as an MGS

A client adds every server NID from the imperative recovery log to
its MGC import as somewhere to reach an MGS.  Not every server runs
one: "nomgs", "nosvc" and a read-only mount all keep the follower
off, and nothing listens on MGS_REQUEST_PORTAL there, so an attempt
costs a full connect timeout rather than being refused.

A target already reports LDD_F_SV_TYPE_MGS to the MGS when it runs
one, and mgs_nidtbl_write() used to discard it.  Keep it instead and
pass it on in mne_type, so a client can tell which nodes are worth
trying; it now reads mne_type as a mask rather than comparing it
with LDD_F_SV_TYPE_OST.

What a target reports is now whether an MGS that serves its own
filesystem runs on the node, rather than what its mount line asked
for.  mount.lustre adds "mgs" from the on-disk label of a device
formatted with --mgs and "nomgs" does not take it away again, and the
MGC requeue thread can start a follower long after a target mounted -
which now also has the targets notify the MGS of their NIDs again, so
the bit reaches the clients.  tgt_add_notifier() does the same once it
has installed the upcall, because a follower started before that has
nothing to notify through.

One node can hold targets of two filesystems with different MGSes, and
each of those has its own MGC, so "an MGS is running here" would tell
the wrong MGS: a follower replicates only the MGS of the MGC that
feeds it, which is the MGC it observes, and MGS0 serves only what its
own MGT holds.  A target reports the bit when the MGS here is one of
those two.

An older client makes that comparison for equality, and would read a
flagged OST entry as an MDT, look for an obdname that does not exist
and drop the entry, losing that target's NID update.  So the bit is
masked out for a peer without OBD_CONNECT_FTM.

A follower is fed by a server MGC, which does not set that flag, so
it records no MGS bit and offers no extra MGS NIDs to the clients
attached to it.  They keep the ones from their mount line.

Test-Parameters: mdtcount=4 testlist=conf-sanity clientversion=2.17
Test-Parameters: mdtcount=4 testlist=conf-sanity serverversion=2.17
Signed-off-by: Oleg Drokin <green@whamcloud.com>
Assisted-by: ClaudeCode:Opus-5 llm_code_and_review_tools
Change-Id: Icb445ea972522c2773b5afc1f11a80c285ef728c

LU-20169 tests: stress stats read during module unload

Add conf-sanity.sh test_180 to stress test stats reading during module
unload and reload.  This is a frequent source of problems when system
monitoring tools are actively accessing stats during mount/unmount.
Configure a nodemap to include those parameters into the testing.

Test-Parameters: trivial
Test-Parameters: testlist=conf-sanity env=ONLY=166,ONLY_MINUTES=60
Test-Parameters: testlist=conf-sanity env=ONLY=166,ONLY_MINUTES=60
Test-Parameters: testlist=conf-sanity env=ONLY=166,ONLY_MINUTES=60
Test-Parameters: testlist=conf-sanity env=ONLY=166,ONLY_MINUTES=60
Test-Parameters: testlist=conf-sanity env=ONLY=166,ONLY_MINUTES=60
Test-Parameters: testlist=conf-sanity env=ONLY=166,ONLY_MINUTES=60
Test-Parameters: testlist=conf-sanity env=ONLY=166,ONLY_MINUTES=60
Test-Parameters: testlist=conf-sanity env=ONLY=166,ONLY_MINUTES=60
Test-Parameters: testlist=conf-sanity env=ONLY=166,ONLY_MINUTES=60
Test-Parameters: testlist=conf-sanity env=ONLY=166,ONLY_MINUTES=60
Test-Parameters: testlist=conf-sanity env=ONLY=166,ONLY_MINUTES=60
TLC-bug-id: TLU-202
Signed-off-by: Andreas Dilger <adilger@thelustrecollective.com>
Change-Id: Ib08cea28aa5fe9e33f07d12562af5c624e8f44d8

LU-15504 utils: fix the 'lfs find -ls' output format

The `lfs find -ls` output was using tabs for field alignment,
but this resulted in misaligned output when fields like the
blocks count or file size were large.  Instead, use the field
size option added later in https://review.whamcloud.com/57395
("LU-16561: find: support width in -printf directive") to do
the field alignment.

Update sanity.sh test_56Eab to sort the output files so
'lfs find' and 'find' are comparing the same filenames,
in case they ever change output order (e.g. parallel find).
This will also run additional iterations of this subtest.

Test-Parameters: trivial
Fixes: 1d8164fa16 ("LU-15504 utils: lfs find -ls function")
Signed-off-by: Andreas Dilger <adilger@thelustrecollective.com>
Change-Id: Iba325742923b4b0ffbcb2c06454c51ab82500c1e

LU-19778 osd-ldiskfs: don't unlock DIO folios in osd_read_prep cache-hit path

DIO folios (PagePrivate2) are per-thread state kept locked in
oti_dio_folios[] for their lifetime.  osd_get_page() only calls
folio_lock() on first allocation; reused slots skip the lock.

When any path marks a DIO folio uptodate before osd_read_prep() sees
it — e.g. OBD_FAIL_OST_FAKE_RW (0x238) calling SetPageUptodate(), or
a cached read hitting the same folio — the cache-hit branch fires and
unlock_page() is called on a folio that must remain locked.

On the next RPC the same thread reuses the (now unlocked) folio without
re-locking it.  Two crash sites follow:

  - Read path: if the file offset maps to a sparse region, osd_do_bio()
    submits no bio, the early-release loop runs, and
    LASSERT(PageLocked(page)) panics the server.

  - Write path: if a subsequent write RPC obtains the same DIO folio,
    osd_write_commit() hits LASSERT(PageLocked(lnb[i].lnb_page)).
    This is the scenario reproduced by toggling read_cache_enable=0
    while concurrent writes are in flight (test_155g + test_63a).

Guard the unlock_page() call with !PagePrivate2(), mirroring the
identical guard already present in the early-release loop below.

Fixes: e790df5fa38d ("LU-13309 osd-ldiskfs: speedup osd_bufs_get/put")
TLC-bug-id: TLU-184
Signed-off-by: Minh Diep <mdiep@thelustrecollective.com>
Assisted-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Change-Id: I58cecf2cdf89f41a01fdeb14e2e304e9b6588f94

LU-10499 pcc: Add dio support for data copy during attach

PCC attach performance is bottlenecked by single threaded
buffered I/O performance.  We could do multi-threading, but
multi-threaded buffered I/O to one file has a very low
performance ceiling.  In order to significantly speed up
PCC attach performance, we need to switch to DIO.

DIO cannot be done from kernel memory due to various
restrictions, so we call out to a usermode helper.

Note that the helper uses open by fid because given a
file pointer, it's not possible to reliably generate the
path to a file on Lustre due to container namespace issues.
Specifically, the path used by the user may not work for
our helper program due to namespace differences.  So we
must use open by fid for the Lustre side of the copy.

This patch improves attach performance from about 1 GiB/s
to about 5 GiB/s.  This performance figure includes time to
read the data from Lustre *and* to write it out to PCC.
---
EX-5014 pcc: avoid deadlock during DIO open attach on rhel7

The Maloo testing fails with sanity-pcc/45 due to the following
deadlock on rhel7 kernel:

    ll_fid_path_cop D ffff9a32db5eb180     0 10783  10782 0x00000080
    Call Trace:
    schedule_preempt_disabled+0x29/0x70
    __mutex_lock_slowpath+0xc7/0x1d0
    mutex_lock+0x1f/0x2f
    lookup_slow+0x33/0xa7
    link_path_walk+0x80f/0x8b0
    path_openat+0xae/0x5a0
    do_filp_open+0x4d/0xb0
    do_sys_open+0x124/0x220
    SyS_open+0x1e/0x20

    dd              D ffff9a32fb5b6300     0 10779  10755 0x00000080
    Call Trace:
    wait_for_completion+0xfd/0x140
    call_usermodehelper_exec+0x179/0x1a0
    call_usermodehelper+0x40/0x60
    pcc_copy_data_dio+0x267/0x340 [lustre]
    pcc_attach_data_archive+0x6ff/0xe80 [lustre]
    pcc_readonly_attach+0x3d2/0xad0 [lustre]
    pcc_readonly_attach_sync+0x205/0x260 [lustre]
    pcc_file_open+0x798/0xdd0 [lustre]
    ll_atomic_open+0xd80/0x1780 [lustre]
    do_last+0xa53/0x1340
    path_openat+0xcd/0x5a0
    do_filp_open+0x4d/0xb0
    do_sys_open+0x124/0x220
    SyS_open+0x1e/0x20

This only happened on el7 kernel which uses mutex for inode locking.
During ->ll_atomic_open(), the kernel will take this mutex on the
parent inode. However, when copy data via the user space helper
program ll_fid_path_copy, it will also try to obtain this mutex
lock on the parent inode during lookup, resulting in deadlock.

Was-Change-Id: I384c7b1979d93183b86bbde311d29a50346a8d56

EX-5014 pcc: minor fixes for parameter checks

Improve console message when out-of-range pcc_dio_attach_size_mb
values are supplied.

Fix sanity-pcc test_49b to allow future limit changes

Was-Change-Id: I2bf7d0bf564c954318980f7a09d8713a70f37db9

EX-6193 pcc: dio attach failed on non-blksz-aligned file

PCC attach failed due to do DIO copy on files with blksz unligned
file size.
The reason is that the copy tool ll_fid_path_copy fails on
non-blksize-aligned file for PCC backend (such as a local Ext4
file system) using direct I/O.
In this path, it fixes this bug by falling back from direct I/O to
buffered I/O mode when copy the tail non-blksize-aligned file
part.

Was-Change-Id: I5287563029269032a91397c0094e2ccede73b9b1

EX-6399 pcc: add tunable parameter for PCC attach thread

Currently the max number of kernel threads doing asynchronous
attach is a hard code value (1024 by default).
In this patch, we make it a tunable parameter:
	llite.*.pcc_max_attach_thread_num

Was-Change-Id: Ic59c15af935dd8dff586fa6be3939d4322c136d5

EX-8236 pcc: abort data copy via ll_fid_path_copy

For data copying via ll_fid_path_copy in direct I/O mode in user
space, the client calls llapi_pcc_state_fd() to obtain the file
PCC state. If it is marked with PCC_STATE_FL_ATTACH_ABORTING, the
data copy process ll_fid_path_copy exits immediately.
To reduce the overhead of these check, we do not check for each
data copy iter, instead, we do a check for certain times of I/Os
(32 times by default). For I/O size of 32MiB, it will be checking
1 times per second at 1GiB/s. There should be some time-lag
before the copy tool quits finally.

Was-Change-Id: I20631e5481a7e97d7a1ed0729bcd269ef6248a2c

Test-Parameters: clientdistro=el8.9 mdscount=2 mdtcount=4 testlist=sanity-pcc env=ONLY=45,ONLY_REPEAT=10
EX-bug-id: EX-5014 EX-6399 EX-6193 EX-8236
Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Signed-off-by: Qian Yingjin <qian@ddn.com>
Signed-off-by: Andreas Dilger <adilger@thelustrecollective.com>
Change-Id: Idb2a12296c3e4778763c9b576bbb0ecd2570a458

LU-19420 tests: add conf-sanity/101a debugging

There are intermittent timeouts in conf-sanity.sh test_101a,
possibly due to OST0000 running out of inodes during the test.
This can happen in single-OST systems like Janitor if the only
OST is offline and has run out of precreated objects, but that
should still not cause files to be created with bad OST objects.

Limit the runtime of the test if it is taking a long time by
creating files in batches rather than a huge number in advance.

Add some debugging to get more information about the failure.
Make the cleanup more robust in case of errors.

Run `ls -l` on newly created files so it actually does something,
since a previous update broke this check by running on wrong dir.
Use $DIR consistently instead of assuming it matches $MOUNT.

Test-Parameters: trivial testlist=conf-sanity env=ONLY=101a,ONLY_REPEAT=50
Fixes: d295847d94 ("LU-8562 osp: fix precreate_cleanup_orphans/precreate_reserve race")
Fixes: ed44caee9a ("LU-11196 tests: clean up after conf-sanity test_101")
Signed-off-by: Andreas Dilger <adilger@thelustrecollective.com>
Change-Id: Ia138f1d87418f11f5145bcaa5ed1067d143ebbe5

LU-20169 tests: stress stats read during module unload

Add conf-sanity.sh test_180 to stress test stats reading during module
unload and reload.  This is a frequent source of problems when system
monitoring tools are actively accessing stats during mount/unmount.
Configure a nodemap to include those parameters into the testing.

Test-Parameters: trivial
Test-Parameters: testlist=conf-sanity env=ONLY=166,ONLY_MINUTES=60
Test-Parameters: testlist=conf-sanity env=ONLY=166,ONLY_MINUTES=60
Test-Parameters: testlist=conf-sanity env=ONLY=166,ONLY_MINUTES=60
Test-Parameters: testlist=conf-sanity env=ONLY=166,ONLY_MINUTES=60
Test-Parameters: testlist=conf-sanity env=ONLY=166,ONLY_MINUTES=60
Test-Parameters: testlist=conf-sanity env=ONLY=166,ONLY_MINUTES=60
Test-Parameters: testlist=conf-sanity env=ONLY=166,ONLY_MINUTES=60
Test-Parameters: testlist=conf-sanity env=ONLY=166,ONLY_MINUTES=60
Test-Parameters: testlist=conf-sanity env=ONLY=166,ONLY_MINUTES=60
Test-Parameters: testlist=conf-sanity env=ONLY=166,ONLY_MINUTES=60
Test-Parameters: testlist=conf-sanity env=ONLY=166,ONLY_MINUTES=60
TLC-bug-id: TLU-202
Signed-off-by: Andreas Dilger <adilger@thelustrecollective.com>
Change-Id: Ib08cea28aa5fe9e33f07d12562af5c624e3ebbe5