Gerrit Review Dashboard

LU-20637 llapi: name a device scan's objects

A scan of a target holds names and parent FIDs but no paths, so lfind
prints FIDs and a user has no way to turn them into pathnames.  The
HLD names an Output Format for exactly this, beside the FID printing
that is already there.

Add llapi_scan_rec_path(), which uses the path a namespace walk
already put in the record and resolves the FID through a client mount
when there is none.  lfind gets --fid2path MOUNT for it.

The cost is one ioctl per object, so it is the consumer's to ask for
rather than the scanner's to pay: it falls on the objects a search
kept, not on every object it saw.  That is also why the HLD prices
pathname generation separately from the scan rate.

An object with no pathname is counted and reported at the end rather
than printed: the objects a target keeps for itself have none, their
FIDs being below FID_SEQ_NORMAL, so a search under --internal is
expected to report some.  An OST data object does have a pathname --
its FID resolves through the OSC to the file that owns it, which
sanity test_154C asserts -- so a scan of an OST names that file once
per stripe it has on the target.  A hardlinked object is printed once,
under its first name.  A FID that cannot be looked up at all -- no
CAP_DAC_READ_SEARCH, which ll_fid2path() checks first -- is an error
and not a missing name, so a broken search cannot read as a complete
one.

Neither half of the resolution asks anything of the target being
scanned, which is what lets it name objects on a target that is out of
service -- and a ZFS target can only be scanned out of service, its
pool being exported.

An OST data object is named by the file that owns it, read from the
object's own trusted.fid.  Its own FID is an OST-sequence one, which
__ll_fid2path() answers only by asking the OST holding it, through the
OSC: exactly the target the scan is reading.  ff_parent is the owner's
MDT FID, and the stripe index filter_fid keeps in its f_ver is
dropped,
that field being an index and not a version.  An object no file owns
yet -- precreated and never written, of which a target holds many --
carries no trusted.fid and is reported as having no name rather than
being chased through the OSC.

trusted.fid is not the only place that FID lives.  An OST whose inodes
are 256 bytes or smaller has none at all: osd_xattr_set_pfid() packs
the parent into the LMA as a struct lustre_ost_attrs and removes the
separate xattr, saying so with LMAC_STRIPE_INFO.  Reading only
trusted.fid there leaves every object on the target without an owner,
so --fid2path prints nothing and reports them all unnamed -- measured
on an OST formatted -I 256: 33 objects, no owner on any of them,
against four files whose paths it should have printed.  mkfs.lustre
defaults to -I 512 now, but -I 256 is still accepted and older OSTs
carry it.  The LMA buffer is already in hand for the classification,
so the fallback costs no second xattr, and ll_decode_filter_fid(8)
reads the same two places.

An MDT object is named from its trusted.link and a map of the target's
directories, FID to parent and name, built by a pass over the target
before the search.  A pass of its own because objects arrive in inode
order, so an object is usually delivered before its ancestors are;
resolving inline would mean holding every match until the scan ended.
The pass asks for no layout, no SOM and no HSM, and a filesystem holds
far fewer directories than files.  On an OST it stops at the first
object and leaves the map empty.

There is no fall back to the lookup once the map is in use.  The map
is
used precisely when the target cannot answer, so one object it could
not place would stop the whole search there.  Two things are nameless
in consequence and counted with the rest: an ownerless OST object, and
on DNE an object whose ancestors live on another MDT, the map covering
only the target scanned.

--paths is the same composition with no mount at all.  --fid2path
still needs one, not to resolve anything but as the filesystem the
paths hang from and for the comparison below, and a filesystem whose
only MDT is the target being scanned has none to give -- which for a
single-MDT ZFS filesystem is every scan of its MDT, the pool having
to be exported.  --paths prints filesystem-root-relative names
instead, asks nothing of any running service, and is refused on an
OST, whose objects are named by files that live on an MDT.

A mount whose own filesystem cannot be read is refused rather than
run unguarded: --fid2path was asked for explicitly, and the fallback
was the very outcome the comparison exists to prevent, another
filesystem's pathnames printed with a zero exit.

llapi_scan_rec_path.3 says the same thing to a consumer pairing that
call with llapi_scan_device() directly, because nothing in the
library can make the check for them: the call is handed a descriptor
and a record and never sees the target the record came from.  lfind
compares the mount's fsname with the target's label; anyone else has
to do that too, or know by construction that the two agree.

A MOUNT of another filesystem is refused when the target is opened,
its fsname compared with the one the target's label names.  The
comparison sits inside the scan's own open rather than in a probe
ahead of it: a second open is not free on every backend, and opening
a target twice buys nothing the one open cannot answer.  It cannot be
caught later: FID sequences are not unique across filesystems.
FID_SEQ_ROOT and FID_SEQ_DOT_LUSTRE are constants, so the root of
every filesystem has the same FID, and normal sequences overlap
because each filesystem's controller starts from FID_SEQ_NORMAL
(LUSTRE_SEQ_SPACE_RANGE).  fld_client_lookup() on the wrong mount
therefore finds a range and __ll_fid2path() answers with a pathname
from it, so the run would have printed another filesystem's paths as
though they were the target's and exited 0.

The mount point is resolved before use: a bare fsname, a
subdirectory and a trailing slash all open the same root, and none of
them names a file when prefixed to what fid2path answers.

conf-sanity test_302 stops an OST, exports its pool where the backend
is
ZFS, and compares the names a scan of it gives against what the client
saw before it went down -- a scan that resolved nothing and one that
resolved everything both exit 0, so the set is what is asserted.  Its
files are written rather than merely created, a precreated object
being
rightly nameless.

conf-sanity test_301 diffs the resolved paths against what the client
sees, and checks that a hardlinked object is named once.  It mounts a
client on the MDS for the duration, because --fid2path resolves
through a mount on the node running the scan and a server is not a
client; a configuration that cannot mount one skips.  It is ldiskfs
only: reading a target in service is what ZFS refuses.

scan_device_run()'s comment says what a second open costs rather than
calling it a deadlock, because it is not one: llapi_find_device()
builds the directory map in a pass of its own, so every --fid2path and
--paths
run already opens and closes the target twice in one process, and
conf-sanity test_302 does exactly that against a stopped OST with its
pool exported and passes on conf-sanity4@zfs.  The reason to test the
fsname inside the scan's own open is the cost of the second import and
export, not a hang.  The wasted pass on an OST -- where
scan_dirmap_cb() stops at the first object -- is left as it is;
folding the map into the search's own open would serve both from one
open and is worth doing separately.

The resolve-failure message names the FID that was looked up and the
mount only when there is one.  An OST object is resolved by its
owner's FID rather than its own, so the message named the wrong one;
and under --paths there is no mount at all, yet -ERANGE and -ELOOP
still arrive here and reported a failure "through the mount given".

lfind(8) says what --fid2path costs, which differs by target and is
easy to state as one number and be wrong. One lookup per kept object
is the OST's cost; on an MDT there is no lookup and a whole extra pass
instead, to build the map the page describes four blocks further down
-- the more expensive of the two, and paid before the search rather
than per match.  Where a name cannot be composed the page points at
lfs-fid2path(1), which is a tool that exists, rather than at a
lookup-only mode of this one, which is not.

--paths and --fid2path together goes through the same exit as every
other error in that function.  Returning CMD_HELP from main() exits 4
and prints no usage, where "ret = CMD_HELP; goto out" prints it and
exits EXIT_FAILURE.

fp_paths takes fp_unused_bit1.  struct find_param keeps those bits for
exactly this, and a trailing byte would have cost 8 with padding.

The directory map's names stay inline.  At NAME_MAX + 1 an entry is
~288 bytes, so a million directories is ~288 MB and 1.5x that while
scan_dirmap_grow() holds both tables, and interning them in an arena
would cut it several-fold.  Left for now: it is a change to the map's
memory layout for a case no measurement here has reached, and a
calloc() failure already fails the search cleanly rather than
corrupting it.  Worth revisiting when --paths is pointed at a target
big enough to need it.

conf-sanity test_303 covers --paths, where 301 and
302 both exercise only --fid2path, so the composition --paths exists
for was never run.  It scans an MDT in service with no mount given,
which is the case a filesystem whose only MDT is the target has to be
named in, and asserts the answer is the client's set: one path per
object rather than one per name, the hardlinked object once, no FID
where a path was asked for, and no mount point in the answer -- root-
relative being what separates it from --fid2path, and a mounted prefix
meaning the wrong composer ran.  It also asserts the two refusals:
--paths on an OST, whose objects are named by files that live on an
MDT, and --paths together with --fid2path.  ldiskfs only, an imported
ZFS pool answering EBUSY by design.

lfsp_fsname makes the check a public caller can ask for.
llapi_scan_rec_path.3 tells one they must establish that the mount and
the target are the same filesystem, and nothing in
<lustre/lustreapi.h> let them: scan_device_run() is internal,
llapi_scan_device() passed want_fsname NULL, and the target's label
lives in a private header.  The field is the same machinery under a
public name -- the scan refuses a target whose label names another
filesystem with -EXDEV -- and it has to be the scan that does it,
because only the scan reads the label and a lookup on the wrong mount
succeeds and answers with that filesystem's pathname.

The usage agrees with its own option list.  "The output is a
FID because a target has no paths" sat two lines below --paths and
--fid2path, both of which print pathnames; it now says the output is a
FID unless one of them is given.

test_301 and test_303 sync the MDS, not only the client.
lustre_super_operations has no .sync_fs, so a client sync writes back
page-cache data and leaves the MDT's ldiskfs to commit on its own
schedule -- and both tests scan that ldiskfs directly while it is in
service, so an object still only in memory is one the scan cannot see
and the comparison against the client's answer is a race.  test_302
needs none of this: its target is stopped, which flushes it.

test_302 also tells the scan where the OST's pool devices are.  An
exported pool is found by reading vdev labels under a search path,
which defaults to /dev, and the test framework's ZFS vdevs are files
in $TMP -- so without --search the pool is not found at all and the
test failed on every ZFS configuration with ENOENT.  On ldiskfs the
argument is empty and nothing changes.

test_301's one bare lfs becomes $LFS, as every other call in the file
is.  An interop run that points LFS at another binary would have used
whatever PATH found.

The EXAMPLES fragment in llapi_scan_rec_path.3 escapes its newline as
n, not n: troff reads the single form as a register reference,
which rendered the line as printf("%s0, path).  llapi_fid_parse.3
escapes it the same way.

The three places that said a target scan needs no client now say what
--fid2path changes.  lfind.c's file header and llapi_find_device()'s
block comment both stated it flatly, and llapi_find_device.3
documented neither fp_fid2path_mnt nor the -ENAMETOOLONG an over-long
mount path answers with; its SEE ALSO gains llapi_scan_rec_path(3),
which is the call that does the resolving.

llapi_scan_rec_path.3's NOTES point a consumer at lfsp_fsname rather
than telling it to compare the mount and the target itself.
lfsp_fsname is that comparison -- the scan refuses a target of another
filesystem with -EXDEV -- and saying otherwise on the page that
introduces it would send a caller to write what the library already
does.

test_303's one bare lfs becomes $LFS, as test_301's did.  It was added
after that fix and reintroduced it.

Signed-off-by: Hiroshi Nishida <hnishida@thelustrecollective.com>
Assisted-by: ClaudeCode:Opus-5 llm_code_and_review_tools
Change-Id: I6476abfd28c91b2ccc04fb08c4e5860aef8fe009

LU-20649 llapi: a changelog as an Object Stream

The two scanners we have enumerate what exists.  A changelog
enumerates what happened, and llapi_scan_changelog() delivers it in
the same record, so a consumer written against llapi_scan_namespace()
or llapi_scan_device() can be fed by a delta without being rewritten.

A changelog record carries the FID, the parent FID, the name, the
event and its time, and -- where the server recorded them -- the uid
and gid of whoever caused it.  That pair is lfsr_event_uid and
lfsr_event_gid, not lfsr_uid: mdd_changelog_ns_store() fills the
uidgid extension from lu_ucred(env), the acting client's credentials,
so it says who acted and not who owns the object, which is what
lfsr_uid means in every scanner that fills it and what lfs find -uid
matches. Delivering it as lfsr_uid would have answered "-uid alice"
from the identity of whoever last touched her file.  The record
carries no size, mode, owner, times, layout or HSM state at any
setting, so those arrive absent with their lfsr_valid bit clear.  With
LLAPI_SCAN_CL_F_RESOLVE they are filled by opening the object through
a client mount, which is one open and one stat each, or two of each
for a regular file whose size was asked for; the demand mask decides
whether that happens at all, so a consumer reading only what an event
carries pays nothing.  Resolution fills what the stream lacks and
never restates what it has, which is what lets a caller ask what an
event recorded rather than what is true now.

Two modes.  Event mode delivers one record per event, in order.
Object mode holds events in a FID-keyed cache and delivers one record
per object once it has been quiet, so a file written a hundred times
is one record and one lookup rather than a hundred of each.

Clearing is off by default and never runs ahead of the callback: it is
irreversible and it purges the registered user's whole backlog, not
this reader's alone.  Registering that user stays the caller's, since
registering without consuming is how an MDT fills up.

The record gains lfsr_event_type, _flags, _time, _index and _prev, the
rename source, and a job id, behind three new validity bits.  An event
time is not an object time, so it does not land in lfsr_mtime: a
search asking for one must not be handed the other.

Only an event that implies a type answers for it -- the three
creations, and CL_RMDIR, whose target is a directory by definition --
so the mode and its validity bit are set together, and a later event
leaves both alone.  Setting them as a pair is what keeps "cannot
answer" apart from "the answer is zero": a mode written on every
event, with the bit only ever OR'd in, would leave a coalesced object
claiming a type while carrying zero for it.  An object's type does not
change, so the first answer stands for every event after it.

lfsr_parent_fd and lfsr_fd are set to -1 rather than left as the
memset found them.  Zero is stdin, and a consumer written against
llapi_scan_namespace() is told to test lfsr_parent_fd and then
fstatat() on it.

The Changelog reader is finished on every path out, the error one
included.  llapi_changelog_start_user() opens the device and sets
*priv before it issues OBD_IOC_CHANGELOG_FILTER, so a failure there --
an unregistered user, an unreachable MDT -- hands back an error with
the reader live, and the fd, its buffer and the kernel-side reader
state would stay pinned for the life of the process if nothing closed
them.

sc_user, sc_mnt and sc_stats are read from the bounded copy rather
than the caller's structure, and the advertised minimum sc_size
reaches through sc_mdtname.  A minimum stopping short of it would let
a caller declare a structure without the field the call cannot work
without, and reading that field off the caller's own memory would go
past its end before the size check had a chance to refuse it.

Clearing stops at the oldest record any cached object still needs, not
at the last index delivered.  Coalescing puts those apart: create B at
10, create A at 11, touch B at 1000, and A can age out and be accepted
while B is still held with a record at 10.  Clearing to 11 there and
then stopping the scan loses B entirely -- no callback saw it, and a
restart from sc_startrec cannot read it back, because the record is
gone.

An event with no target object is delivered per event even in object
mode.  mdd_changelog_ns_store() zeroes cr_tfid when its target is
NULL, which mdd_rename() passes whenever the destination name did not
already exist, and cr_markerflags -- a CL_MARK's union with the same
bytes -- is not a FID either.  Coalescing keys on the FID, so all of
them shared one cache entry and only the last survived: an ordinary
`mv a b` was enough to lose the one before it, and with clearing on,
the record it came from is gone as well.

The parent FID is taken only from an event that carries one.  CL_CLOSE
does not -- mdd_changelog_data_store() is called with pfid NULL -- so
assigning it from every event while only ever OR-ing the bit in left
create+write+close reporting LLAPI_SCAN_PARENT over a zero.  A name or
job id whose copy failed leaves its bit clear for the same reason: a
bit that promises a string must not be set over a NULL.

A resolve is skipped only for a demand mask every event can answer,
which LLAPI_SCAN_TYPE is not: scan_cl_mode() answers it for the four
record types that imply one and no others, so counting it there would
leave a consumer asking for FID and type with neither a lookup nor a
type on a CL_CLOSE.

Resolution opens O_PATH and takes a real open only for the regular
file whose size it came for.  llite gives a special file
init_special_inode(), so opening every object for read opened the
driver behind a device node stored in the filesystem; O_NONBLOCK holds
off the wait, not the rest of what an open does.

The uidgid extension is asked for when sc_want is 0, and that case is
tested on its own rather than by a mask.  Zero means "everything", so
it is the caller that most needs the extension -- and `0 &
~LLAPI_SCAN_EVENT` is 0, so any test of that shape would exclude
precisely the caller that wanted it, leaving LLAPI_SCAN_EVENT_UID out
of lfsr_valid.

A size is reported strict only where the OSTs were actually glimpsed.
The second open runs for a regular file whose demand mask names a
size and can still fail, so a consumer asking for an mtime alone
reaches the fill on the O_PATH descriptor; that is the MDT's lazy
answer, and LLAPI_SCAN_LAZY_SIZE and LLAPI_SCAN_LAZY_BLOCKS are what
say so.  liblustreapi_pfind.c maps the two to OBD_MD_FLSIZE and
OBD_MD_FLLAZYSIZE, so they are not interchangeable.

LLAPI_SCAN_CL_F_RESOLVE is one open and one fstat, so the page says
which six of the ten fields it fills.  The project id, the layout,
the directory stripe and the HSM state each cost an ioctl or an xattr
of their own and are not filled at any setting; the page said
otherwise by listing all ten and then naming the flag.

The checkpoint gets its caveat, in the header and the page both.
Under coalescing a record naming no object goes out as it arrives
while an object whose earliest record sits lower is still held, so
the delivered indexes are not a watermark.  A run that reaches the
end of the log flushes what it holds and the highest index is safe;
a run the consumer stopped drops it, so that run must leave the
checkpoint alone.

sc_startrec and sc_endrec are both inclusive and the page now says
so, the example resuming from the saved index plus one rather than
re-delivering the record the job already processed.  lfsr_event_flags
is cr_flags & CLF_FLAGMASK, so CLF_RENAME, CLF_JOBID and
CLF_EXTRA_FLAGS are masked off -- they say which extensions a record
carries, not what happened.

Three smaller things.  The second fstat() is taken only when the
glimpse replaced the descriptor.  scan_cl_bucket() drops a modulo
that cannot change llapi_fid_hash()'s value, that being hash_long()
on the same shift.  The sc_mdtname test after the memcpy goes: the
minimum sc_size reaches through that field and the caller's own
struct was tested at the top, so it could never fire.  Four comments
described an earlier revision of this patch rather than the code and
now describe the code.

The COALESCE flag says which event's fields the one record carries. It
promised one record per object and left open which of the object's
events that record spoke for, so a consumer matching on lfsr_name had
no way to know it sees one name per object rather than every name the
log recorded -- an object created under one name and hardlinked under
another arrives carrying only the last of them.  lfs find --changelog
-name is that consumer, and the gap showed there first.

sc_type_mask without sc_user is refused.  The mask reaches the server
only through llapi_changelog_start_user(), which needs a registered
user to filter for; on the other branch llapi_changelog_start() never
sees it, so a caller asking for CL_UNLINK alone read every event type
and nothing said so.  Refused rather than applied client-side, because
that is a different thing -- the records still cross the wire -- and
sc_filter is already the client-side spelling.  The man page said only
that the mask is intersected with the user's own, which does not tell
a caller what happens when there is no user.

llapi_scan_changelog_test covers the five things the shape of this
stream turns on: that a record describes the event it came from, that
object mode collapses a burst into one record, that the demand mask
keeps a lookup from happening at all, that a consumer's own return
value comes back unchanged, and that bad parameters are refused rather
than crashed on.  It takes -m for the MDT, -u for a registered user
and -d for a client mount, since three of those need a live log.
Nothing in lustre/tests drives it yet, unlike its two siblings, which
land their suite case with the scanner: that case wants a changelog
registered and cleared around it and a skip where the MDT records
nothing, and is left for its own patch rather than grown here.

sanity 157d runs it, so the binary is not merely built.  Its events
have to be on the MDT whose log it reads, which under DNE a mkdir
below a striped root does not guarantee, so the test makes the
directory on MDT0 itself rather than leaving the binary to notice and
refuse.  -u passes the registered Changelog user the clearing cases
need; without one they skip themselves.

Signed-off-by: Hiroshi Nishida <hnishida@thelustrecollective.com>
Assisted-by: ClaudeCode:Opus-5 llm_code_and_review_tools
Change-Id: I12efd3bd2150592a4507533129ab56573ef924d9

LU-20650 llapi: fill a scan record for one FID

llapi_scan_namespace() walks what exists, llapi_scan_device() reads
what exists on a target, and llapi_scan_changelog() reads what
happened.  A consumer holding a FID had no way in, so the changelog
module resolved privately and with fstat() alone: mode, nlink, uid,
gid, three times, size and blocks, and nothing else.  A record filled
that way answers -size and -mtime but not -stripe-count, --hsm-state
or -projid, so "lfs find --since -stripe-count 4" would answer
differently from a plain find on the same filesystem.  That is a
defect rather than a difference of provenance: --since promises what
is true now, verified through the mount.  This patch adds the call;
the search moves onto it in "lfs: find --changelog, the log as
source", which is where that example starts answering.

The default demand mask is the walk's, so --hsm-state and -projid
still want naming in lfsp_want; what the call fixes without being
asked is the layout, which is what the example above turns on.

llapi_scan_fid() is the fourth way into the record and fills it the
way the walk does, because it is the walk's own gather.  It resolves
the FID to a pathname and gathers there rather than gathering from a
descriptor opened by FID: scan_rec_gather() is path-oriented
throughout -- get_lmd_info_fd() wants a parent descriptor and a name,
get_projid() opens by path, and the HSM and MDT-index branches open
the path themselves -- so the alternative is an fd-shaped twin of
every one of those, which is the second copy of the knowledge
LU-20611 existed to remove.  Resolution is not an added cost on the
caller that wants this: --since prints pathnames and prefix-matches
them for its subtree restriction, so it needs the pathname anyway.

What that costs is stated rather than buried: per object one fid2path,
one type-only statx and one open of the object or its parent, and
CAP_DAC_READ_SEARCH unless llite.*.user_fid2path is set.  The statx
asks for STATX_TYPE alone deliberately: anything touching size, blocks
or mtime keeps need_glimpse set in ll_getattr_dentry() and buys an OST
glimpse per regular file that the gather then discards.
An object unlinked since the caller learned of it no longer resolves
and comes back -ENOENT, which for a consumer asking what is true now
is the answer and not an error.

The call fills a fresh record and never reads one in.  A consumer
merging it into a record it already holds decides which fields it
lets this one supply, which is what will keep --changelog --resolve
filling only what the stream lacks.

A filter that returns a positive value skips the object, as it does
for the other three scanners, and the call returns 0 rather than
handing that value back to a caller testing `if (rc)`.  The gather
runs against a pathname, so the FID it found is checked against the
one asked for and a name unlinked and recreated underneath it answers
-ESTALE rather than a record for a different object.

Only lfsp_want and lfsp_filter are read from the parameter block.
lfsp_max_depth, lfsp_stats, lfsp_search and lfsp_fsname describe a
traversal there is none of and are ignored; lfsp_thread_count above 1
is refused rather than ignored, because a caller setting it has
misunderstood the call.

It widens a lfsp_want of 0 through LLAPI_SCAN_WANT_DEFAULT, the
definition llapi_scan_namespace() already uses, so the two mount-side
entry points mean the same thing by 0.

A trailing slash on mnt_path is trimmed before the pathname is
composed.  The leading slashes of fid2path's answer were already
stripped, but "/mnt/lustre/" with the root FID composed
"/mnt/lustre/" and left lfsr_name on the terminating NUL -- which
lustreapi.h says means NULL, not empty -- and every other FID got a
doubled separator in the published lfsr_path.

The kernel-doc no longer lists lfsp_flags among the fields that are
ignored.  Every flag it defines is, but a bit outside them is refused
with -EINVAL, as the code does and llapi_scan_fid.3 already says.

llapi_find_device() clears lfsp_got.  scan_param_copyin() copies the
caller's block whole, so the caller's pointer came through and was
answered with find's own demand mask -- a mask for a question the
caller never asked.  The field's kernel-doc is rewritten with it, to
say that it is per scan rather than per library and why it is reached
through a pointer.

The syscalls go through mnt_fd, not through the composed pathname.
The FID was already resolved through that descriptor; the statx, the
object's open and the parent's now use it too, at the name fid2path
answered, so mnt_fd alone says which filesystem is read.  An fd pins
its mount where a path string is resolved afresh every time, and a
mount replaced between the open and the scan would otherwise have the
resolve read one filesystem and the gather another, with rc 0 and
nothing said.  It also takes the mount prefix off every lookup, three
per object, which over a changelog's worth of FIDs is what this entry
point is for.

mnt_fd has to be the mount root now.  The ioctl behind
llapi_fid2path_at() takes any descriptor in the filesystem, and the
page always named llapi_root_path_open(), which returns the root; a
caller that passed something else answered before and answers -ENOENT
now.  mnt_path stays required and stays what lfsr_path is built from,
so a wrong one still mislabels the answer -- it just no longer decides
what was read.

llapi_scan_test gains a case for it, so sanity 157c runs it: one
record for a known FID, checked against the pathname and the basename
the client sees; a filter that skips, which is a success with nothing
delivered; the argument refusals, including the -EBADF that a
negative mnt_fd has to answer rather than the -EINVAL that also means
"this FID has no name"; and a FID that resolves to nothing, which
must fail and deliver no record whichever way it fails.

Signed-off-by: Hiroshi Nishida <hnishida@thelustrecollective.com>
Assisted-by: ClaudeCode:Opus-5 llm_code_and_review_tools
Change-Id: I796db84d3fe596b1cfa93658dcfeb240db8e33a5

LU-20650 lfs: find --since, from the changelog

The three pieces were already here.  llapi_scan_changelog() says which
objects had an event, llapi_scan_fid() fills each one's record the way
a walk fills it, and find_decide() judges it with the predicates a
walk uses.  --since composes them, so it answers over fewer objects
and judges each the same way: the objects are a strict subset,
narrowed and never different.  The lines are not a subset of the
lines -- a walk reaches a hardlinked file once per directory entry
and prints it once per link, while --since reaches it once per FID
and prints it once, under the first of its names that is under the
root and matches.  No new predicates and no second copy of the
vocabulary.

The argument is never guessed.  A bare integer is an index, which is
what "lfs changelog MDT <startrec>" already takes; a single-unit
duration (30m, 2h, 7d) is that far back from now; a timestamp or
@<seconds> is that time; anything else is an error naming the four
spellings.  A leading sign is refused rather than wrapped through
strtoull(), and a duration reaching past the epoch is a typo rather
than a window.

cr_index belongs to one MDT, so a bare index is refused on a
filesystem with more than one and the message says which time
spellings do work.  That count has to be right, and the obvious source
is not: LL_IOC_GETOBDCOUNT answers with the size of the LMV's target
array, so a single-MDT filesystem reports 64.  It is used as an upper
bound and each index confirmed with llapi_search_tgt() before its
changelog is read.

A changelog covers the whole filesystem, so the subtree the caller
named is enforced against the pathname llapi_scan_fid() resolved, and
the depth the walk would have counted is recovered from it for
--maxdepth and --mindepth.  Without that, a find rooted at one
directory would answer for its siblings.

Against every name the object has, not just the resolved one.
llapi_scan_fid() resolves linkno 0, so a hardlinked file arrives under
whichever name fid2path answered with first: with /a/f and /b/f the
same inode, "lfs find /a --since 2h" would be handed b/f, reject it,
and lose a file a walk of /a prints.  -name has the same shape, which
is why a target scan walks every linkea name.  The names come from
fid2path here, one ioctl each, and only a file whose link count says
it has another name pays for any of them.

A time anchor is cut on the record rather than at the start index,
because a time cannot be turned into a per-MDT index without reading
the log; coalescing has already reduced an object to its latest event,
so an object whose latest event is older than the window did not
change in it.  An object unlinked between its event and now cannot be
verified, so it is not in the answer -- that is what the flag gives up
-- and the run says how many were dropped that way.

--since is refused for a target scan rather than ignored.  The option
table is compiled into lfind as well, so `lfind --local --since 2h`
parsed and then scanned the whole target: llapi_find_device() has no
changelog to anchor on and never reads fp_since_kind.  It joins the
list find_device_supported() already keeps for --maxdepth, --threads
and -printf.

llapi_find_since() refuses fp_thread_count above 1 for itself, before
it looks at anything else: this source reads one MDT's changelog at a
time and has no work to divide.  That is why lfs_find() now computes
the default thread count on the walk's arm of the dispatch rather than
above it -- computed for every search, the default alone was above 1
and the refusal fired on every --since run, which is not a setting the
caller made.

The mount point is found from the resolved root and not from the
spelling, which are the same string only when the caller already gave
a canonical absolute path.  /home/user/data -> /mnt/lustre/data
answered -ENODEV, llapi_search_mounts() matching mnt_dir literally,
and a relative path went through the fsname to the *first* mount of
that filesystem -- so under $MOUNT2 every object fell outside the
subtree and the search printed nothing and exited 0.

What is printed is then the caller's spelling of that root with the
resolved remainder appended, because a walk prints the path as it was
given -- param_callback() copies the argument and
llapi_semantic_traverse() appends to it.  Named from the mount every
time, "lfs find . --since 2h" would answer /mnt/lustre/sub/f where
"lfs find ." answers ./sub/f, and --since promises the same answer as
the search without it.

ONLY=56 is the regression check that the shared paths still answer as
they did; the subtests for the new options arrive with the tests
patch at the end of this series.

Two things the page is careful about.  An MDT with no registered user
does not record at all, rather than recording and purging:
mdd_changelog_off() clears CLM_ON when the last user deregisters and
mdd_changelog_enabled() requires it.  And the usage string spells the
argument TIME|INDEX rather than TIME, because a bare integer is a
changelog index -- the one distinction the parser goes out of its way
to make, and the one a reader is most likely to get wrong.

Test-Parameters: testlist=sanity env=ONLY=56
Signed-off-by: Hiroshi Nishida <hnishida@thelustrecollective.com>
Assisted-by: ClaudeCode:Opus-5 llm_code_and_review_tools
Change-Id: I13426ef1ed12cd08d78aa5a5da9f8ba4465690d2

LU-20650 lfs: find --changelog, the log as source

--since treats the changelog as a candidate set and verifies each hit
against the live object, so it answers as it is now.  --changelog
replaces the source: it reads the log and nothing else, and answers as
it was recorded.  That is why -name here matches the name in the event
rather than the object's current one, and why an object unlinked since
the event is in this answer and cannot be in --since's.

It takes an MDT name or "all", required rather than optional because
getopt_long() accepts an optional argument only as --changelog=VALUE.
One named MDT is also what makes a bare --since index unambiguous, the
index being paired with the log it came from.

Every predicate a record cannot answer is refused naming itself,
rather than answered from a zero.  Without --resolve that is -size,
-blocks, -perm, -links, --attrs, --xattr, the layout options, -projid,
-printf, --mdt-count and --mdt-hash; --xattr was the loud one,
reaching llistxattr(NULL, ...) once per object, and the rest were
quiet and worse, "--perm 644" matching nothing and its negation
everything.  -uid and -gid are refused for a different reason and at
every setting short of --resolve: a changelog records who caused an
event, from the acting client's credentials, which is not who owns the
object.  An object time is refused outright, a record carrying an
event time and the two being different questions; the message says
--since 1d is how to ask.  Only --mdt, --maxdepth and --mindepth are
refused at every setting -- they describe a walk, or need something a
per-MDT log does not have.  --ost and the object times are refused
without --resolve alone, the lookup supplying both.

-type is the one field with a third answer.  A record says what kind
of object was created and only for four record types, so an object
whose event does not carry one is counted undecided rather than
compared against a mode of zero, which reads as "not a file" and "not
a directory" at once.

Output is a pathname where the FID still resolves to one and the FID
where it does not, which is the whole point of asking: an object
unlinked since its event is the case --changelog exists for, and
counting it as "no pathname" the way a target scan counts its own
objects would drop exactly that case in silence.  The count of those
is reported at the end, so a caller knows some lines are FIDs.

The path argument names the filesystem, not a subtree, a changelog
knowing nothing about where an object sits; a path below the mount is
refused unless --resolve makes a pathname to match against.

--resolve looks each object up through the mount and hands
find_decide() what it found -- the path, the parent descriptor and the
object's own -- so -size glimpses the OSTs, -projid opens through a
descriptor of ours, and -printf gathers as a walk gathers.  It fills
what the stream lacks and never restates what it has: the merge starts
from the looked-up record and puts every field the event answered for
back over it, taking only the type bits of the mode, so -perm reads
the live one.  The subtree is tested on that pathname, over every name
the object has rather than the one linkno 0 answered with.

An object that has gone between its event and the lookup keeps the
answer honest in the other direction.  It is still in this answer --
resolution fills and never removes -- but only where the record can
carry the question: where the search asks something only a lookup
supplies, or names a subtree the object can no longer be placed in, it
is counted undecided instead.

find_device_supported() refuses --changelog for the same reason it
refuses --since: the option table is compiled into lfind too, so it
parses there, and llapi_find_device() never reads it -- `lfind
--changelog all` would otherwise have run the target scan, printed
everything the other predicates kept, and exited 0.

-name joins -type as a predicate the record cannot always answer.
cr_namelen is 0 for an event on a FID whose name the log never held,
so CL_CLOSE, CL_SETATTR, CL_TRUNC and CL_LAYOUT all
arrive nameless -- a file created before the window and written inside
it is the ordinary case.  Matching those against "" decided them both
ways and said nothing about it, and is not even reliably a non-match,
fnmatch("*", "") succeeding.  They are counted undecided instead.

The first resolve failure that is not "this object has no name"
decides the exit status, as it already does for a target scan, so a
run that could resolve nothing cannot report itself complete.

fc_d is set only for a directory, as find_since_rec_cb() already had
it.  printf_format_lustre() reads d == -1 as "not a directory", and
sr_fd is not always -1 for a regular file: under -printf the gather
widens, scan_rec_gather() opens the object for the MDT index and
leaves that descriptor in sr_fd.  Passing it through sent every
striped file down the directory branch and onto an lmv buffer
find_rec_to_lmv() had just zeroed, so %Lc and %Li printed 0, %Lo
[0] and %Lp nothing.

-name is answered against the recorded name under --changelog.
find_since_pick() walks the object's other links to place it in the
subtree, and was substituting each link's basename into the record
before testing: right for --since, which asks what the object is now,
and wrong here, where the record is the answer.  The alternate path
still settles the subtree; only the name stays as recorded.

The pathname a match prints goes through the caller's spelling
whichever way it was composed.  Without --resolve it comes from
llapi_scan_rec_path(), which builds an absolute name from the mount,
so "lfs find ." answered /mnt/lustre/sub/f there and ./sub/f
everywhere else.  find_since_respell() splits into a find_respell()
taking the two roots, and the print site uses it.

The demand mask -printf widens is the one handed to llapi_scan_fid():
find_want() builds fss_want with the -printf flag set and the resolve
passes it as sp_want.  llapi_scan_changelog()'s own sc_want is a
separate literal and does not depend on -printf.  What the two sources
share is fss_want, which is why --since and --changelog widen
together.
The comment above it claimed %s, %b and %LP as the reason; those come
from find_decide()'s own stat and find_get_projid(), gated on
fc_gather_all, and find_want() clears LLAPI_SCAN_PROJID on its way
out regardless.  What the widening actually buys is LLAPI_SCAN_LMV,
for %Lc and its neighbours on a directory.

The cookie is not mentioned any more.  fp_since_cookie arrives with
--since-cookie in the next patch, and this one documented it in
llapi_find_since()'s Return: block and in llapi_find_since.3 --
an -EINVAL for giving both an anchor and a cookie, and an -ESTALE,
neither of which exists here.  -ENOTSUP does exist and was missing
from both.

A CL_MARK is dropped rather than delivered.
mdd_changelog_write_header() writes the marker flags into the union
with cr_tfid, so cr_tfid.f_seq would read back as CLM_ON|CLM_START,
0x10001 -- and fid_seq_is_igif() accepts anything in [12, 0xffffffff],
so the FID gate would pass a value that is not a FID and find_decide()
would print it as a FID with no pathname.

No such record can arrive today, so this is a guard and not a fix for
an observed answer: CHANGELOG_MINMASK is BIT(CL_MARK) and
mdd_changelog_mask_seq_write() hands it to cfs_str2mask() as the
floor, so CL_MARK cannot be masked out -- and
mdd_changelog_write_header() returns early whenever CL_MARK is in the
current mask, so no marker record is ever written.  Asking for "creat
mkdir" gives back "MARK CREAT MKDIR", and deregistering every user and
re-registering leaves an empty log.  The guard costs one test per
record and stops the flags being read as a FID if that ever changes.

It sits at the top of find_since_cand_cb(), above the FID gate.  The
anchor is not taken there but in find_since_filter_cb(), which runs
first on every record the log delivered, marks included: it has to see
them, or it lags the end of the log by whatever trailing run of them
the scan finished on.

-printf's widening now reaches --since.  find_want() clears
LLAPI_SCAN_PROJID on its way out and find_decide() gates
find_get_projid() and the OST glimpse on fc_gather_all rather than on
the demand mask, so a flag this patch sets for --changelog and not for
--since meant "--since -printf %LP" answered DEFAULT_PROJID for every
object where a walk answered the real one, and %s/%b answered from the
MDT instead of a glimpse.

The comment on the nameless case said cr_namelen is 0 for a MARK.  It
is not: mdd_changelog_write_header() sets it to strlen(obd->obd_name)
and copies the device name in, so a mark arrives named "lustre-
MDT0000" -- which is also why "-name 'lustre-MDT*'" matched one before
the record was dropped above.  A data event on a FID whose name the
log never held is the real case, and the sentence now says only that.

The undecided warning no longer claims a missing field.
find_since_cand_cb() also counts an object undecided when it has gone
and the search needed a lookup or named a subtree, and there the
record answered -- the object is what is missing.  "could not be
decided from their changelog record" covers both.

LLAPI_SCAN_CL_F_ONCE is set for --since and not for --changelog.  Once
means once per scan, which is what "as it is now" wants; a source
answering as it was recorded has a second burst to report, so an
object touched in two of them prints twice.  The man page says so.

The dropped-objects warning names what the count is of -- "%llu of
%llu changed objects could not be read" -- rather than leaving a bare
number beside the errno it belongs to.

Test-Parameters: testlist=sanity env=ONLY=56
Signed-off-by: Hiroshi Nishida <hnishida@thelustrecollective.com>
Assisted-by: ClaudeCode:Opus-5 llm_code_and_review_tools
Change-Id: If9ead118b5fc1dee86cfac6646106e982841d8e2

LU-20650 lfs: find --since-cookie, per-MDT anchor

cr_index belongs to one MDT, so "where I got to" on a DNE filesystem
is one number per MDT and not one number.  --since <time> is the human
spelling and means the same thing everywhere, but only as far as the
clocks agree; a cookie is exact on every MDT, and is what a repeated
job should use.

The file is a line per MDT, "<mdtname> <index>", read as the anchors
and rewritten with the indexes this run reached.  It is written whole
and renamed over, so an interrupted run leaves the previous anchors
rather than half of the new ones -- a half-written cookie is worse
than a stale one, because the stale one is caught.

Which is the reason a cookie is worth having.  If an MDT has purged
past its anchor the records in between are gone, and reading from the
oldest one left would return a short answer with nothing to say so.
So the oldest surviving index is read first, by a one-record scan that
stops at the first callback, and an anchor older than it is refused
naming both numbers.  Every MDT is checked before any of them is
scanned: refusing from inside the scan would leave the earlier MDTs'
matches on stdout with no cookie written, and the next run would
repeat them.  An MDT with no anchor is not probed at all -- 0 has
nothing to be stale against, and that is every MDT of a first run.

Zero is reserved for "the log is empty", which is why a record that
arrives without an index is reported rather than recorded as index 0.
Changelog indexes start at 1, so storing 0 for a record that has one
would make a log holding records look like a log holding none, and 0
compares false against every anchor -- the check would go on being
skipped and say nothing about it.  An empty log does proceed: from a
client, "nothing happened since the anchor" and "everything since it
was purged" look the same, and refusing on that would fail on every
idle filesystem.

A cookie from another filesystem would anchor this one at indexes that
mean nothing here, so a line whose name does not match the filesystem
is ignored, and a file whose header names another filesystem is
refused.  Ignoring every one of its lines instead would leave no
anchors at all, rescan this filesystem's whole log, and then drop the
other one's anchors in the rewrite -- three wrong answers, none of
them said out loud.

The rest of the file is input the caller does not control either.  The
scanf width is not optional: %s into a fixed buffer is how such a file
overruns it.  Nor is the type of the bound test, since %x accepts a
sign: "<fsname>-MDT-1" reads back as 0xffffffff, which passes a test
made against an int and then indexes the array far past its end.  So
the sign is refused at the parse and the bound is compared against the
value that does the indexing.

--since and --since-cookie are two ways to say where to start, so
giving both is refused rather than one of them silently winning.  More
than one path is refused for a related reason: lfs find runs the
search once per path and every run advances the anchors, so the second
path started past the end of the log, printed nothing, and left the
records it should have matched behind the anchor for good.

The cookie is proved writable in the same pre-pass that proves the
anchors are not stale, and for the same reason -- a cookie that cannot
be written leaves the matches on stdout and the next run repeating
them.  A missing directory component reads as ENOENT, which
find_cookie_read() takes for a first run, so the typo was not noticed
until every log had been scanned.

The temporary file is fsync()ed before the rename, and the directory
after it.  The rename makes the replacement atomic against another
process, which is not the same as durable: after a host crash the
rename can be on disk while the contents are not, and an empty cookie
parses as no anchors at all.

The pre-pass probes the temporary name the write will use rather than
the cookie itself, because the write creates "<cookie>.new" and
renames over: what it needs is permission on the directory, which
opening an existing cookie for append does not test.  A cookie owned
by the job user in a root-owned directory passed and then failed
EACCES with every match already printed.

The anchors are left where they were when a run did not finish.  A
resolve failure that is not "this object has no name" drops matches,
and writing the cookie anyway would advance past the very records a
rerun needs, so the exit status is settled before the rewrite rather
than after it.

An index is read with strtoull() and a leading sign refused, the MDT
number having been refused for the same reason: "%llu" negates, so
"-2" would anchor a run at 0xfffffffffffffffe, match nothing, and
write that value back.

find_device_supported() refuses --since-cookie along with --since:
the option table is compiled into lfind too, so it parses there, and
llapi_find_device() never reads it -- a target scan would have
printed every object the other predicates kept and exited 0, without
even reading or writing the cookie it was given.

ONLY=56 is the regression check that the shared paths still answer as
they did; --since-cookie's own subtests are in the tests patch at the
end of this series.

The cookie names the search as well as the filesystem, and a
mismatch is refused the way a wrong fsname already was.  The anchors
say how far each log was read, not how far a subtree was: a record is
counted before the FID and the subtree are tested, so a run over one
subtree carries every anchor to the end of the log and a second
subtree sharing the cookie starts past its own changes and prints
nothing.  That is the collision lfs_find() refuses when two paths are
given to one command, reached by running the command twice.  A cookie
written before this carries no root and is still accepted.

A cookie that exists and is not a regular file is refused up front.
The probe settles the containing directory, not the name: the write
ends in rename(), which answers EISDIR over a directory, while the
read succeeds and returns no anchors -- so naming a directory scanned
every log to the end, printed every match, and only then said it
could not rewrite.  An empty name is refused with it.

The comment on the file format said a comment or a blank line costs
nothing.  It costs the read nothing; the rewrite emits the header and
the anchors and nothing else, so it does not survive the run.  Said
so, the sentence otherwise reading as an invitation to annotate the
file.

The documentation for fp_since_cookie arrives here rather than in the
previous patch, which described the field, its -EINVAL and its
-ESTALE before any of them existed.

The root is the rest of the header line rather than a field.  A
pathname can hold a space and %s cuts it at the first one, so a search
under "/mnt/lustre/my data" was recorded whole and read back as
"/mnt/lustre/my": the comparison then failed for every run after the
first, and the message named a path the caller never gave.  The root
is whatever realpath() returned for the caller's path, so a directory
with a space in it was enough to reach it.

And the root is escaped rather than written raw, because the rest of a
line still cannot hold a newline.  Only '/' and NUL are excluded from
a name, so "/mnt/lustre/anb" was written whole, read back as
"/mnt/lustre/a", and refused for every run after the first naming a
path the caller never gave -- the space case one character further
out, with the remainder becoming its own line and being dropped.  Both
sides escape and the reader compares the escaped forms, so nothing
decodes: the mapping only has to be injective for a comparison, which
is also why backslash is escaped.  The refusal names both sides in
that spelling, or a newline would break the line meant to name it.

find_cookie_check()'s two other refusals say so as well.  An empty
--since-cookie and a name too long to append ".new" to both returned
the errno alone, beside a "is not a regular file" case that reports
itself, so the run ended at the caller's "failed for '<path>': Invalid
argument" with nothing naming the argument at fault.

Two diagnostics that this patch makes reachable are corrected with it.
The refusal of --resolve without --changelog said "--since already
reads the object", which names an option the caller need not have
given once --since-cookie exists; it names both anchored spellings
now.  And a cookie refused for its filesystem or its root was reported
twice, the second line saying "cannot read" over a file that read
perfectly well -- find_cookie_read() has already named which refusal
it was, so the wrapper is left with what fopen() passed up.

The MDT number is exactly the four hex digits find_cookie_write()
emits.  %4x converts the leading digits and ignores what follows, so
"<fsname>-MDT00001" and "<fsname>-MDT0000_UUID" both anchored MDT0000
at an index never written for it -- the short answer with nothing said
that the stale check exists to catch, from a file this function's own
comment says it ignores anything but "<mdtname> <index>" in.  The span
refuses the sign as well, so the separate test for it goes.

ERRORS gains -ESTALE, which the previous patch dropped on the
understanding it would arrive with the cookie and which the kernel-doc
Return: block has listed since.  It is the error the whole option is
built around, and the man page was the one place a caller of the API
could not find it.

Test-Parameters: testlist=sanity env=ONLY=56
Signed-off-by: Hiroshi Nishida <hnishida@thelustrecollective.com>
Assisted-by: ClaudeCode:Opus-5 llm_code_and_review_tools
Change-Id: Ia8ce93368a2c2c8c2ea2ee88c8e7f7e099b1bfa5

LU-20650 tests: sanity cases for find's changelog flags

lfs-find.1 gains --since, --since-cookie, --changelog and --resolve:
what each answers, that --since is a strict subset of the same search
without it, that a bare index belongs to one MDT and is refused where
that is ambiguous, that a number which could be an index or an epoch
second is never guessed at, and that reading a changelog neither
consumes it nor needs a registered user while records exist only while
some user is registered.  TIME's spellings are given in full, since
set_since() takes eight of them and its error text offers one the page
would otherwise not list.

The page says what --resolve does not answer for -- an object gone
between its event and the lookup is counted undecided and dropped, so
--resolve can leave out an object the same search without it reports
by FID -- and which name -name is tested against.  The log is
coalesced to one record per object before the search sees it, so the
name that can match is the one the object's latest event to carry one
carried: a file created as f and hardlinked as g is matched by "-name
g" alone, and the pathname printed is its first name rather than the
one that matched.  A plain walk meets each name in turn and matches
all of them.  The behaviour is deliberate -- substituting a current
name would answer -name against something the record never said -- but
a reader had no way to know it.

Four sanity cases.  160aa asserts the promise --since is worth having
-- that its answer is a subset of the same find without it, and a
smaller one -- rather than asserting a particular file, so it does not
depend on what else the filesystem is doing.  It names the files it
expects rather than only counting them, since a count that merely
shrank is also satisfied by returning nothing, and it widens the gap
around its window so a node that spends seconds in sync does not
change what the test means.  Its locals stay separate from the
assignments that fill them, and it sorts off the pipeline rather than
in it: "local x=$(...)" takes local's own exit status, which is always
0, and a command piped into sort reports sort's, so either shape would
let a failing lfs find read as the empty answer the index case
asserts.

160ab walks the refusals: -size, an object time, --maxdepth, a path
below the mount, an unknown MDT, -perm, -links, --mdt-count,
--mdt-hash and --xattr are each refused, --resolve on its own is
refused rather than dropped, and a named MDT is accepted as the other
half of --changelog's argument.  It asserts the --resolve answer
rather than the exit status alone: -size needs a strict size, the MDT
holds only the lazy one, and a resolution that did not glimpse left
every object "could not be decided" while still exiting 0.  It also
asserts -printf's %s, which reads the same size through a widened
gather, that --projid is answered rather than failing on the first
record, and that a subtree search under --resolve answers for that
subtree and not the whole filesystem.

160ab covers the two things that make --changelog worth having over
--since, which the man page leads with: an object unlinked since its
event is still in the answer, and its FID names it once no pathname
does.  It ends on -name, which is answered from the record: the log is
cleared, one file that predates the clear is written to -- data
stores, which carry no name -- and another is created.  The named one
has to come back and the nameless one must not, so a -name that
matched everything fails the second assertion and one that matched
nothing fails the first.

160ac runs the cookie three times: the first writes it, the second
returns the new object and not the old one, the third returns nothing,
and then a stale anchor and both anchors together are refused.  It
asks for creations only.  A wide mask records the test's own reads,
and then "nothing changed since" is never true -- which is what hid an
off-by-one in the cookie until a narrow mask made every run repeat the
last object of the one before it.

A test that asserts a precondition instead of creating one only holds
where that precondition happens to be true.  160ac's stale anchor is
made rather than assumed -- records the cookie has not consumed, then
a purge, then one more past the hole -- because writing index 1 is
only older than the oldest surviving record on a filesystem that
already has changelog history.  On a fresh one index 1 IS the oldest
record, nothing was purged, and refusing would be wrong.

160ac also feeds the cookie two files it is never written with: one
whose MDT index is negative, and one whose header names another
filesystem.  Neither is a bad anchor to ignore -- the first indexes
the anchor array far past its end, and the second is a whole-log
rescan of this filesystem plus the loss of the other one's anchors --
so one has to be ignored and the other refused, and the file it
refuses has to still be there afterwards.

160ad hardlinks one file into two directories and searches under each.
A changelog names an object by FID, and fid2path answers with one of
its names -- the first linkea entry, the one the file was created
under -- so a search rooted at the other directory is answered about a
name that is not under it, and -name about a basename the object does
not go by there.  Without the links being walked it finds the file
under a and loses it under b, which is a silent miss and the worst
shape a wrong answer can take.

The tag is tests: rather than doc: because that is where the bulk of
the change is; the man page rides along because the options it
describes are the ones these cases exercise.

There is no version gate.  Nothing in this series touches server code
-- it is lustreapi.h, lustre/utils/ and Documentation/ -- so the
options work against any server old enough to have changelogs, and a
gate on the MDS would skip interop runs that pass.  A client gate
would say nothing either: this script ships with the client whose lfs
has the options.  The third Test-Parameters line below is that claim
under test rather than only asserted: an old server, a new client, and
these four cases.

Test-Parameters: testlist=sanity env=ONLY=160aa,160ab,160ac,160ad
Test-Parameters: testlist=sanity env=ONLY=160aa,160ab,160ac,160ad mdscount=2 mdtcount=4
Test-Parameters: testlist=sanity env=ONLY=160aa,160ab,160ac,160ad serverversion=2.15.6
Signed-off-by: Hiroshi Nishida <hnishida@thelustrecollective.com>
Assisted-by: ClaudeCode:Opus-5 llm_code_and_review_tools
Change-Id: Id4001c6821e72f15b83a9733bbb28cef4248b822

LU-20603 llapi: pull a scan's records in batches

Add llapi_scan_namespace_open(), llapi_scan_next() and
llapi_scan_close(): the namespace scan turned around, so a consumer
pulls records in batches on its own thread instead of being called
once per object on the scan's.  A layer over llapi_scan_namespace()
and not a second scanner -- the scan runs on a thread of its own, its
callback copies each record into the batch being filled, and
llapi_scan_next() hands out a full batch while the scan fills the
other.

The copy is what a batch costs.  A callback record points into
buffers the scan reuses for the next object, so a record that
outlives its callback has to own what it points at: the path, the
name, the layout, the directory stripe, the linkea, the rename source
and the jobid are copied into an arena the batch owns and the
pointers rewritten.  The descriptors are not carried and read -1.
Two batches suffice, the scan blocking when the one it fills is full,
so what is in flight is bounded by the batch size and not by the
tree.  The caller sees an array of record pointers rather than of
records, so its stride is a pointer's and a record that grows does
not move its neighbours under a caller built against the older size.

A filter in lfsp_filter runs on the scan's threads with the caller's
data, which the layer keeps beside its own.  An error is reported
after every record produced before it has been handed out, so the
batches before it are whole; a close on a running scan stops it, and
that is not an error.

A handle belongs to one consumer thread, and that is enforced rather
than only stated: llapi_scan_next() and llapi_scan_close() refuse a
second caller with -EBUSY.  Without it a second consumer resets the
batch the first is still reading and installs one whose sb_count is
already sb_cap, and the next append writes past the end of sb_recs --
reproduced under ASan on the unfixed build, five runs out of five, and
gone on the fixed one, which serves every object once.  It is the
opposite of the callback API's rule, where the callback runs on every
scan thread at once, so it is said in the header and the man page too.

The arena keeps every chunk across a reset rather than the largest.
A batch needing two chunks -- about 320 bytes of payload per record at
the 1024 default, which a path plus an 8-stripe layout reaches --
otherwise malloc'd and freed the second on every batch, which is what
the comment said it did not do.

Measured on a 20k-object tree off Lustre, where per-object work is a
cached lstat() and the copy is at its largest share: a batch of 1024
costs 76 ns per object over the callback at 4 threads (+8%), 135 ns
at one (+5%); a batch of 1 costs 2-4 us, the lock round trip per
record.  On a scan whose per-object work is an ioctl the same copy is
a fraction of a percent, which the man page says.

test16 rides along: it asks llapi_scan_namespace() what lfsp_got
reports, which is the callback API and not this one, and belongs with
whichever change lands lfsp_got.

llapi_scan_next.3 says when a batch stops being the consumer's, not
only that it does.  The release happens at the top of the following
llapi_scan_next(), before that call waits for the scan rather than
when it returns, so a consumer that hands a batch to another thread
and calls again to overlap the next fill has released what that thread
is reading -- and the -EBUSY that refuses a second caller does not
refuse a reader racing one.  The arena is rewound rather than freed,
which is what makes a steady-state scan allocate nothing per batch, so
that reader finds the following batch's bytes and not a fault.

The -EBUSY case is made rather than waited for.  A second consumer is
refused only while the first is inside llapi_scan_next(), and a test
that calls from between two of them never reaches the guard it is
about -- the ASan overflow above cannot fail it.  A filter that sleeps
holds the scan back so the third call blocks; two calls come first,
the layer filling one batch ahead and the consumer not blocking until
it has drained that.  The overlap is then measured, not assumed: a
machine slow enough to put the second call outside the window makes
the case unreachable, and an unreachable case is reported rather than
failed.

Test-Parameters: trivial testlist=sanity env=ONLY="56El 157c"
Signed-off-by: Hiroshi Nishida <hnishida@thelustrecollective.com>
Assisted-by: ClaudeCode:Opus-5 llm_code_and_review_tools
Change-Id: I29aaa31a4c0bd34ca591316601a653f9b5ae70c2

LU-20564 utils: probe md RAID geometry for OST mkfs/mount

mkfs.lustre now probes the md array underneath an OST target (through
stacked dm devices) and seeds -E stride/stripe_width so mballoc can
align and size allocations to full stripes.  mke2fs can derive these
from the blkid topology, but that is absent on some stacked
configurations.  raid0, raid4/5/6, raid10 and raidkm (level 71) are
decoded; the replication factor of raid10 and the declustered g/m of
raidkm come from the md layout word.  User-supplied
stride/stripe_width values are respected, misaligned partition starts
draw a warning at mkfs time, and mount.lustre now warns when an OST on
a striped array has no (or mismatched) fs geometry.  A stripe_width
that is a multiple of the array row is still stripe-aligned and is not
reported.

mkfs.lustre also warns when the array's data row (k * chunk) is not a
power of two.  Such a row cannot be written whole by any client: bulk
RPC sizes are powers of two in practice, so a row divides some RPC
size if and only if the row is itself a power of two.  A geometry like
k=5 at 64K (320KiB) or k=14 at 64K (896KiB) leaves a partial row at
the tail of every RPC, which the array services as a read-modify-
write, at every chunk size and with no tuning available.  Unlike
stride/stripe_width this cannot be corrected afterwards -- it is a
property of the array -- so the warning is issued at mkfs time, while
the geometry can still be chosen, and is not suppressed by a
user-supplied stride/stripe_width.

Tested on RHEL 9.7 against live loop-backed md arrays: raid0/4/5/6,
raid10 near, far and offset layouts, a non-power-of-two row,
user-supplied values, dm-linear over one array and spanning two with
conflicting geometry, and aligned and misaligned partition starts.
Also tested against live raidkm (md level 71) arrays: plain m=2 and
m=3, parity-last, and declustered g=6/m=2 and g=8/m=3 over wider
pools.  In every case the computed row matches md's own
queue/optimal_io_size.

Test-Parameters: trivial
Signed-off-by: Hiroshi Nishida <hnishida@thelustrecollective.com>
Assisted-by: Claude:Fable-5 claude-code
Assisted-by: Claude:Opus-5 claude-code
Change-Id: I442528cea51a55c6b74f7f2282a1db566eab9783

LU-20542 gss: retry callback RPC when reverse ctx is stale

A server cannot renegotiate a reverse GSS context; one is only
installed from a peer's forward request. So when the peer drops its
contexts, the next callback is answered with GSS_S_NO_CONTEXT,
sptlrpc_req_replace_dead_ctx() returns -ECONNREFUSED, and
ldlm_handle_ast_error() evicts a client that was only mid-refresh.
This is the sanity-sec test_27d/27e failure.

Resend the callback a bounded number of times instead, marked with
rq_gss_rvs_resend so it skips the context refresh in
ptlrpc_check_set() and reaches the peer over the same dead context.
Every other resend still refreshes, so a reverse context that goes
stale on its own is still detected and replaced. Blocking and
completion ASTs stay bounded by rq_no_resend and the lock callback
timeout; for a glimpse AST, which sets neither, the resend count and
its backoff are the bound.

Fixes: 67acf6047e34 ("LU-17317 gss: do not continue using expired reverse context")
Test-Parameters: testgroup=review-dne-selinux-ssk-part-2
Signed-off-by: Hiroshi Nishida <hnishida@thelustrecollective.com>
Change-Id: I2a007b42656eb9aa0bed00291b67b6ac7db2d389

LU-20648 mdc: fix changelog mask composition

A zero cf_mask means two different things and chlg_ioctl() composed
them as if it meant one.  On the server it means "this user has no
mask of its own": mdd_chlg_usermask() returns 0 for a record that
carries none.  On the client it means "do not filter": a zero
crs_user_mask skips the record type test altogether.

So "lfs changelog --user NAME --mask creat" for a user who has no
mask computed in.cf_mask & 0, which is 0, which the record test then
read as unfiltered.  The caller got every record type back and no
error, which is a wrong answer rather than a failure.

Compose the two zeros as what they mean.  A user with no mask of its
own is not restricted to nothing, it is unrestricted, so a caller's
mask passes through unchanged; only two real masks intersect.

Registration writes a rec2 when either a mask or a name was given.
It fills cur_mask when a mask was given, and otherwise only while the
server's changelog_mask is still minimal, where a maskless user is
handed CHANGELOG_DEFMASK instead.  So the reachable case today is a
registration with a name and no mask made while the server mask is
set to anything wider.  A user registered with neither name nor mask
gets the older record type, whose mask reads as zero whatever was
written, and is affected too once LU-20647 makes such a user
reachable by lookup at all.

Two real masks that do not overlap intersect to zero as well, and
zero cannot carry "nothing matches" while it also means "do not
filter".  Two absolute masks cannot reach it: cfs_str2mask() seeds an
absolute mask from its minmask argument, and both the registration and
"lfs changelog --mask" pass CHANGELOG_MINMASK, so "-m creat" is really
MARK|CREAT and "--mask mkdir" is MARK|MKDIR -- always sharing
BIT(CL_MARK).  A relative one does.  llapi_convert_str2mask() seeds
from *oldmask when the first token carries an operator, and
lfs_changelog() seeds that with CHANGELOG_DEFMASK, so "--mask
=-mark,-creat" is DEFMASK without those two: non-zero, and sharing no
bit with a user registered "-m creat".  So the wrong answer is
reachable from the shell, and against such a user "lfs changelog
--user clN --mask=-mark,-creat" printed every record it had.

So crs_user_mask now names the record types to deliver: it starts as
all bits, an empty intersection is no bits, and the record test is a
plain bit test with no special case for zero.

test_160z asserts that the user carries no mask before going on,
rather than reasoning that it must: changelog_register() sets +hsm
before registering, and mdd_changelog_mask_seq_write() rebases a
relative mask onto CHANGELOG_DEFMASK while the proc mask is still
minimal, so the branch that hands a maskless user a default one
cannot fire there.  The test widens the mask so that CREAT and MKDIR
are recorded whatever an earlier subtest left it at, and only on
$SINGLEMDS, which is the facet its trap restores.  It asks for --mask
creat and requires the MKDIR record to be absent, then asks without
--mask and requires it to be present.  The second half is what keeps
the fix from turning "no mask" into "no records".

160z covers the disjoint case in its second half, with a user
registered "-m creat" read back through "--mask=-mark,-creat".  An
earlier attempt used two absolute masks and was removed after a
control run showed it passing against the unfixed module as well; this
one cannot, the composition there being zero and zero meaning "do not
filter".  The control beside it is the same user read without --mask,
so a run that selects nothing because the user has nothing cannot be
mistaken for the fix working.

Fixes: 41b55cf2309d ("LU-19296 changelog: Add user-specific changelog filtering")
Test-Parameters: testlist=sanity env=ONLY=160z
Signed-off-by: Hiroshi Nishida <hnishida@thelustrecollective.com>
Assisted-by: ClaudeCode:Opus-5 llm_code_and_review_tools
Change-Id: Ib178e4c15be40c415262bf06c08da8b061855cf9

LU-20647 mdd: look up a changelog user of either record type

mdd_changelog_user_register() writes CHANGELOG_USER_REC when the
caller gave neither a mask nor a name, deliberately, for
compatibility.  mdd_changelog_user_lookup_cb() matched
CHANGELOG_USER_REC2 only, so such a user was never found and
mdd_changelog_user_lookup() returned -ENOENT.  That is the shipped
"lfs changelog --user", which reaches the callback through
OBD_IOC_CHANGELOG_FILTER, not only the library call.

Every other MDD walker already takes both types and guards the fields
that exist in only one of them, so the callback was alone in
rejecting the record rather than the field.

The type test cannot simply be dropped.  struct
llog_changelog_user_rec is half the size of rec2 and carries neither
cur_mask nor cur_name, and the || short-circuited before strcmp()
reached cur_name; removing the first clause would read past the end
of the record into whatever the llog buffer holds next.  So a lookup
by ID matches on cur_id alone, a lookup by name skips an old record
instead of comparing against it, and the reply's username is filled
only from a rec2.  The reply's mask already came from
mdd_chlg_usermask(), which returns 0 for the old type, and a zero
mask is what the client reads as "do not filter".

The reply's username comes from the record's type and not from how the
lookup was made.  mdd_iocontrol() passes one buffer as both request
and reply, so cf_user_id holds the record's ID by the time the old
"was this a lookup by ID" test read it -- true on every lookup, and
harmless only because a name lookup had matched that same name.

The name comparison is bounded as well.  cur_name comes off disk with
no guarantee of a NUL inside its 16 bytes, and
mdd_changelog_name_check_cb() already allows for that on the same
field.

test_160w covers this command but registers both of its users with a
name and a mask, so both records are rec2 and the plain path was
never walked.  test_160y registers one user plainly, one with a mask
and one with a name, and asks for the records of each; the masked user
is the control that has always worked.  The named user is coverage
rather than a regression case: the old callback rejected a
CHANGELOG_USER_REC in the first clause of its ||, before strcmp() ran,
so a lookup by name walked past a plain record before this patch as it
does after it.  Only the cf_user_id != 0 path changes behaviour, and
the client parses "cl<N>" into an ID, so that is the path lfs takes.

This wants LU-20648 beside it.  Making a plain user reachable makes
cf_mask == 0 the ordinary answer, and a client without that fix
composes in.cf_mask & 0 and reads the zero as "do not filter": so
"lfs changelog --user cl1 --mask creat" on a plain user would print
every record type where it used to stop at -ENOENT.  Landing this one
alone trades a lookup failure for a wrong answer.

Fixes: 5b85a4eb7510 ("LU-19296 changelog: retrive changelog user info from MDT")
Test-Parameters: testlist=sanity env=ONLY=160y
Signed-off-by: Hiroshi Nishida <hnishida@thelustrecollective.com>
Assisted-by: ClaudeCode:Opus-5 llm_code_and_review_tools
Change-Id: I12ef76c773ce31e4548db9022f381cabfe7f5ddc

LU-20643 utils: clear stale lmd fields on reuse

get_lmd_info_fd() reuses one struct lov_user_mds_data for every object
and convert_lmd_statx() writes only the fields lstat(2) supplies, so
stx_attributes, stx_attributes_mask, stx_btime and the mask bits above
STATX_BASIC_STATS keep whatever was there before.

Two ways that is not this object's data.  On the V1 ioctl path the
lstat_t is copied over the front of the record, so stx_attributes_mask
ends up holding st_atim.tv_sec -- never 0 in practice -- and stx_mask
picks up bits from st_nlink, which reads as STATX_BTIME on a directory
whose link count has 0x800 set.  It does not reach as far as it looks:
lmd_st is 144 bytes against 272 for lmd_fid and lmd_stx, so it stops
at the end of stx_mtime and stx_rdev_major onwards is still the
previous conversion's.  On the lstat fallback the buffer holds the
previous object's data outright, or the file name written in for the
ioctl.

So a walk that crosses onto a non-Lustre subtree can answer -btime and
the attribute predicates from a file name.

Clear what is about to be reused on both paths, with the same line:
everything up to lmd_lmm, which is what the record's own fields amount
to.  That takes lmd_fid, lmd_flags, lmd_lmmsize and lmd_padding with
it -- nothing in lustre/utils reads lmd_fid today, but a short name
there passes fid_is_sane() as an IGIF, and llapi_get_lum_file_fd()
copies by lmd_lmmsize.  On the V1 path it goes after the memmove, so
the layout it has just moved up is not touched.  The glimpse path is
left alone: there the MDT's answer is still wanted.

Fixes: 11aa7f8704c4 ("LU-11367 som: integrate LSOM with lfs find")
Signed-off-by: Hiroshi Nishida <hnishida@thelustrecollective.com>
Assisted-by: ClaudeCode:Opus-5 llm_code_and_review_tools
Change-Id: I6126e73bb120bd6a12a904563e13303d63e75c1c

LU-20050 ptlrpc: fix import ref leak on sepol failure

ptlrpc_connect_import_locked() calls ptlrpc_request_alloc(), which
takes a reference on the import via class_import_get(). When
sptlrpc_sepol_get() fails, the IS_ERR() branch calls
ptlrpc_request_free(). That only returns the request to its slab
and does not drop the import reference.

The leaked reference keeps client_obd_cleanup() from running, so
the LDLM namespace kobject is left in sysfs and the next mount
fails with -EEXIST.

Use ptlrpc_req_put() on this path instead. It drops rq_refcount to
zero and calls __ptlrpc_free_req(), which releases the import.

The second failure path, after ptlrpc_request_bufs_pack() returns
an error, still uses ptlrpc_request_free(). That is correct: the
out_free label inside ptlrpc_request_bufs_pack() has already
dropped the import reference, so only the slab is left to free.
Both paths now carry a comment saying which case they are.

This bug is only reachable after 8a6788b508db, which stopped a
positive errno from reaching ERR_PTR() and so made the IS_ERR()
branch above actually trigger.

Fixes: dd200e5530fd ("LU-8955 ptlrpc: manage SELinux policy info at connect time")
Assisted-by: Claude:Opus-5 claude-code
Signed-off-by: Hiroshi Nishida <hnishida@thelustrecollective.com>
Change-Id: Ia4267f87219e2a7e4fbcb996978543174bcb7197

LU-20546 ptlrpc: fix import ref on request setup failures

ptlrpc_request_alloc() takes a reference on the import. Two
setup paths get its release wrong, in opposite directions.

ptlrpc_request_alloc_internal() open-codes atomic_dec() on
imp_reqs and then calls ptlrpc_request_free() when an idle
reconnect fails, so class_import_put() is never reached and the
reference leaks. Call ptlrpc_req_put() instead, which does both.

ptlrpc_request_bufs_pack() has the reverse problem: out_free
drops the reference but leaves rq_import set, so a caller that
disposes of the failed request with ptlrpc_req_put() releases it
a second time. gss_cli_ctx_fini_rpc() and osp_sync_new_job() both
do that on a pack error. Clear rq_import once it has been
dropped, which also lets callers use either free or put.

Fixes: 93d20d171c20 ("LU-11128 ptlrpc: new request vs disconnect race")
Fixes: 8e86156c34f3 ("LU-10486 osp: fix request leak on error in osp_sync")
Assisted-by: Claude:Opus-5 claude-code
Signed-off-by: Hiroshi Nishida <hnishida@thelustrecollective.com>
Change-Id: Iec07b093e380f262b452ff4fc5f5fac64f1c9fb0

LU-20050 ptlrpc: fix sepol_helper status decode

sepol_helper() declared ret as signed short while
call_usermodehelper() returns int, and decoded it with a single
"rc = ret>>8". That shift is only correct for a normal exit. A
kernel launch failure returns a negative errno, not a wait
status, and shifting collapses it (-ENOMEM becomes -EPERM). A
helper killed by a signal leaves the signal in the low 7 bits,
so ret>>8 is 0 and the failure is reported as success.

Widen ret to int and decode the three cases separately, with raw
bit tests since WIFEXITED/WEXITSTATUS are userspace macros:

  ret < 0            ->  kernel failed to launch the helper
  (ret & 0x7f) == 0  ->  normal exit (WIFEXITED)
  (ret >> 8) & 0xff  ->  exit code   (WEXITSTATUS)

The exit code must be sign-extended, not negated: l_getsepol
returns -errno whenever stdin is not a tty, always the case
under call_usermodehelper(), and exit() keeps only the low byte.
So -ENODEV arrives as exit byte 237, and (signed char)237 is
-19. sptlrpc_sepol_get() special-cases -ENODEV as "SELinux is
disabled, report nothing", so negating it made that branch
unreachable and turned those mounts into "Unknown error 237".
This extends 8a6788b508db.

A positive exit status is not an errno either, so negating it
invented plausible but wrong errors: 127 became EKEYEXPIRED and
126 ENOKEY, pointing at key management rather than a broken
helper. Report -EPROTO and log the raw status. sepol_helper()
can then no longer return a positive value, so drop the
unreachable rc > 0 case in sptlrpc_sepol_get().

The eight sptlrpc_sepol_get() error paths in mdc_locks.c,
mdc_reint.c and mdc_request.c leaked the request's import
reference and the LDLM locks gathered for early cancel: the
sepol check runs before the prep call, so the request still owns
both, but the error tail only called ptlrpc_request_free(). Keep
one exit per function and make it correct on both sides of the
prep call, through the new mdc_elc_req_put() helper, clearing
@count once the prep call owns the locks. This needs LU-20546,
which clears rq_import when packing fails so the reference is
dropped exactly once.

Add sanity-selinux test_21d: stub l_getsepol, mount with
send_sepol=-1, and check that exit 237 still mounts while 234
and a missing helper fail with the expected errno text. It
creates no nodemap on purpose -- with nodemap.<nm>.sepol set,
req_check_sepol() would refuse the -ENODEV connect for an
unrelated reason.

Suggested-by: Sebastien Buisson <sbuisson@ddn.com>

Test-Parameters: trivial testgroup=review-dne-selinux-ssk-part-1
Test-Parameters: trivial testgroup=review-dne-selinux-ssk-part-2
Fixes: c61168239eff ("LU-8955 sec: create new function sptlrpc_get_sepol()")
Fixes: 0a773f04b288 ("LU-8955 ptlrpc: manage SELinux policy info for metadata ops")
Assisted-by: Claude:Opus-5 claude-code
Signed-off-by: Hiroshi Nishida <hnishida@thelustrecollective.com>
Change-Id: If9312fce842fca133de21c4898cafd949ace6aa6

LU-20119 dbg: test lustre_helper labeling fix for test_21a

fortestonly patch validating the fix for the sanity-selinux
test_21a failure on the DNE + RHEL 8.10 server + RHEL 10.0 client
config, where the client mount fails because l_getsepol returns
-EACCES (-25 before the errno fix below) from the sepol connect
upcall.

Root cause: the l_getsepol/lctl/l_getidentity/l_getauth helpers
were left labeled bin_t instead of lustre_helper_exec_t, so the
kernel execs them into the unprivileged kernel_generic_helper_t
domain (via the base "kernel_t bin_t:process
kernel_generic_helper_t" transition) rather than lustre_helper_t.
That domain cannot read selinuxfs or the on-disk SELinux policy,
so l_getsepol fails with EACCES.  Two reasons the label was never
applied:

1. lustre.spec.in %post loaded the policy module with "semodule
   -i" but never ran restorecon.  semodule -i does not relabel
   already-installed files, and the binaries were labeled bin_t at
   install time (before the module existed).  Add a restorecon of
   the four helpers to %post (the make-install path already does
   this, but it is skipped for staged/RPM builds where DESTDIR is
   set).

2. lustre_helpers.fc labeled only /usr/bin/*.  On RHEL/Rocky 10.2+
   the /usr/sbin=/usr/bin file-context equivalency makes that
   work, but on el8/el9 and el10.0/10.1 /usr/sbin is a real path
   with no such equivalency, so restorecon of the /usr/sbin binary
   finds no rule.  List both /usr/bin and /usr/sbin.

The sepol_helper() CERROR and the errlog() errno fix from the
prior diagnostic patch are retained so that, if the mount still
fails, the kernel log shows l_getsepol's real exit code.

Not intended to land; validates the fix before it goes on the
LU-20119 review.

Test-Parameters: fortestonly
Test-Parameters: livedebug mdscount=2 mdtcount=4 osscount=1 ostcount=8 clientcount=2 testlist=sanity-selinux clientdistro=el10.0 serverdistro=el8.10 env=ONLY="21a",SHARED_KEY=true clientselinux
Signed-off-by: Hiroshi Nishida <hnishida@thelustrecollective.com>
Change-Id: Ifd207da02862849c90500a40b4ff71dff5bc711b

LU-20238 ec: branch-free byte-indexed scalar base path

Replace the per-byte gf_mul() in the scalar *_base hot paths with
a byte-indexed per-coefficient region table:

  tbl[b] = c * b in GF(2^8), built once per coefficient
  gf_region_mul(tbl, x) = tbl[x]   (single load, inner loop)

ec_encode_data_base() at k=6/p=2, gcc -O2: 0.79x at 256 B, 1.08x
at 1 KiB, 1.32x at 64 KiB. gf_mul() alone is ~20% slower than the
64 KiB direct lookup it replaces, so regions below
GF_REGION_TBL_MIN_LEN (1 KiB) keep a scalar fallback.

Byte-indexed lookup (gf-nishida-16 "FullByte"-style) is also
patent-clean: no PSHUFB, no nibble split, no 16-byte gftbl, so it
is structurally distinct from the ISA-L SIMD pattern that maps
onto US 8,683,296 claim 21.

ec_base.c carried an unconditional "#define GF_LARGE_TABLES", so
both ec.ko and libec.a compiled the 64 KiB gf_mul_table_base and
the small-table branch was dead code. With that define and the
inert ec_base_o_CPPFLAGS in ec/Makefile both gone nothing could
set it, so the block is deleted outright: .rodata 65792 -> 1536,
.text 2217 -> 3591 for the fallbacks.

gf_vect_mul_init is renamed gf_vect_mul_init_base, with a wrapper
in ec_base_aliases.c; its 64-bit path now selects on BITS_PER_LONG
for kernel builds, since __WORDSIZE is glibc-only. No existing
entry point changes behaviour.

Validated on RHEL 10.1 (AVX2+GFNI): exhaustive gf_mul + gf_inv,
k=6/p=2 round-trip KAT, and gfni_ec_test byte-identical parity.

Test-Parameters: testlist=sanity-ec
Test-Parameters: testlist=sanity env=ONLY=910
Signed-off-by: Hiroshi Nishida <hnishida@thelustrecollective.com>
Change-Id: I905e365d145d0018cbcd2bbc919eb4e516fafe8e

LU-20147 ptlrpc: module-level cache for SELinux policy upcall

Replace the per-import call_usermodehelper model with a single
module-level upcall_cache (hashsize=1, key=0) shared by all imports.

The original LU-8955 design called l_getsepol via
call_usermodehelper() on every RPC from every import independently.
On RHEL/Rocky 10.1 this fails: the kernel_generic_helper_t SELinux
domain is blocked from reading /sys/fs/selinux/policyvers and
writing to debugfs, causing a 30-second timeout on every first RPC.

Cache lifecycle and freshness:
- upcall_cache.h: add struct sepol_cache_data and union member in
  upcall_cache_entry.
- lustre_sec.h: remove ps_sepol and ps_sepol_checknext from
  struct ptlrpc_sec.
- sec.c: add sptlrpc_sepol_cache (module-level upcall_cache),
  sptlrpc_sepol_get(), sptlrpc_sepol_do_upcall() (fires l_getsepol),
  sptlrpc_sepol_parse_downcall(), and sptlrpc_sepol_cache_update()
  for unsolicited push updates.  send_sepol changed from static to
  extern.
- The cache is initialised with entry_expire = S32_MAX so the
  framework never reaps entries by time; sptlrpc_sepol_get() drives
  freshness by stat()ing the policy directory on every call and
  calling upcall_cache_flush_one() if its mtime differs from the
  value carried by the last downcall.  l_getsepol therefore runs
  only on mount or after an actual policy change.
- send_sepol == 0 disables the feature entirely (no policy info on
  any RPC).  Any non-zero value behaves identically; the legacy
  "send_sepol = N seconds" semantic is preserved as parameter-level
  backward compatibility but no longer affects runtime behaviour
  (with the cache in place, the LU-8955 reason for an N-second TTL
  doesn't apply).

Backward-compatible wire format and invocation:
- lustre_user.h: keep the pre-LU-20147 layout as
  struct sepol_downcall_data_v2 / SEPOL_DOWNCALL_MAGIC_V2 = 0x8b8bb843
  alongside the new struct sepol_downcall_data /
  SEPOL_DOWNCALL_MAGIC = 0x8b8bb844.
- sec_lproc.c sptlrpc_sepol_do_downcall() and sec.c
  sptlrpc_sepol_parse_downcall() accept both magics; only the new
  layout populates the policy-dir mtime tracking used by the
  freshness check.
- l_getsepol gained -2 / --new-format: with -2 it writes the new
  layout, without -2 it writes the v2 layout.  Default is v2 so a
  new userspace stays compatible with an old kernel.
- sec.c sptlrpc_sepol_do_upcall() keeps the pre-LU-20147 argv shape
  "--obd_type X --obd_name Y" (so old l_getsepol still recognises
  it) and just appends "-2" as a new flag.  X/Y are the type and
  name of any active OBD on the client (MGC preferred); the per-OBD
  srpc_sepol write handler funnels the result back through
  sptlrpc_sepol_do_downcall() into the module-level cache, so it
  doesn't matter which OBD is used.  We invoke under UMH_WAIT_PROC
  and on non-zero exit (eg. old userspace that doesn't recognise
  "-2") retry without "-2".
- sec_lproc.c: add module-level srpc_sepol write-only debugfs file
  at sptlrpc/srpc_sepol (LDEBUGFS_FOPS_WR_ONLY) so admin tools can
  feed the cache directly; per-OBD srpc_sepol kept for backward
  compatibility and used by the kernel upcall above.

l_getsepol changes:
- Replace file-scope static globals with per-call lgetsepol_ctx.
- Add -k/--kernel flag: writes downcall to module-level path
  sptlrpc/srpc_sepol; includes policy dir path and mtime.
- Add -2/--new-format flag (see above).
- Add -w/--watch flag: inotify daemon mode for policy change
  detection (retained as alternative; community feedback welcome).
- Add find_policy_with_highest_ver() helper used by the existing
  RHEL/Rocky 10.1 fix in LU-20045's stack; the helper itself is
  duplicated locally so this patch is self-contained against an
  older LU-20045 base.  When LU-20045 lands on master this duplicate
  can be dropped.
- Use fstatat(dirfd(dp), ...) in the policy version scan loop.

Signed-off-by: Hiroshi Nishida <hnishida@thelustrecollective.com>
Change-Id: Icf813c252b7359c3dc17d80405579b50b57ab025

LU-20084 osp: fix LDLM ns kobject leak in lwp_device_fini()

The LWP teardown path does not call ldlm_namespace_free_prior()
before client_obd_cleanup() -> ldlm_namespace_free_post().
This affects two paths:

Normal teardown (LCFG_CLEANUP -> lwp_process_config ->
lwp_disconnect -> lwp_device_fini): lwp_disconnect() calls
ptlrpc_disconnect_import() and ptlrpc_invalidate_import() but
never calls client_disconnect_export(), so
ldlm_namespace_free_prior() is never reached.  lwp_device_fini()
then calls class_disconnect() and client_obd_cleanup() directly,
skipping free_prior entirely.

Failed-connect path (lwp_obd_connect failure -> out_dis:
class_disconnect -> lpd_exp=NULL -> lwp_device_fini): the same
lwp_device_fini() is reached with lpd_exp NULL, so
class_disconnect is skipped and client_obd_cleanup() is again
called without free_prior.

Without ldlm_namespace_free_prior(), LDLM_NS_STOPPING is never
set.  If the pool recalc task runs concurrently with
ldlm_namespace_free_post(), the race can corrupt the pool kobject
cleanup path and cause ldlm_namespace_sysfs_unregister() to hang,
stranding the namespace kobject under
/sys/fs/lustre/ldlm/namespaces/.

Fix by calling ldlm_namespace_free_prior() in lwp_device_fini()
before client_obd_cleanup().  This is symmetric with lwp_setup()
-> client_obd_setup() where the namespace is created, and covers
all LWP teardown paths.  The import is still valid at this point
since obd_cleanup_client_import() runs inside
client_obd_cleanup() afterwards.

Signed-off-by: Hiroshi Nishida <hnishida@thelustrecollective.com>
Change-Id: Ie9a290ca529d18b0f837669a53c0c06cda4f767f

LU-20606 llapi: scan an ldiskfs target directly

Add llapi_scan_device(), which reads an MDT's or OST's objects
straight off the device with libext2fs and delivers the same records
llapi_scan_namespace() does.  No mount, no MDS, no kernel change: it
runs against a stopped target, a snapshot or a failover partner's LUN,
read-only and with no write path.

libext2fs stays off liblustreapi.  The backend builds as
scan_osd_ldiskfs.so and is dlopen'ed on first use, named and found the
way mount.lustre's own mount_osd_ldiskfs.so is, so a client build has
no plugin and the call answers ENOTSUP.  Three refusals get three
answers, because a caller keys off them: ENOTSUP where there is no
backend to ask, ENOPKG where the backend is there and refused the
target -- stock e2fsprogs will not open a filesystem with dirdata set,
and the package to install is the answer -- and EINVAL for a device
that is not a Lustre target at all.  Giving the first two the same
errno would have conf-sanity test_300 skip itself on a node carrying a
real MDT and a build that can scan it.

The record grows what a device answers for and a walk does not: the
object id, the inode generation, the LMA flags, the raw linkea and a
class.  The label is read the four ways mkfs.lustre writes it -- '-',
':', '=' and the '+' that tunefs.lustre --nolocallogs leaves on disk
while the target is stopped, which is exactly when this is what there
is to read it with.

Two more pieces of public API, so the diff holds no surprises. struct
llapi_scan_stats is what the new lfsp_stats points at: the caller sets
ss_size and owns the storage, the scan fills the rest on return, and
the man page describes it as part of the answer rather than as a
diagnostic.  And LLAPI_SCAN_F_INTERNAL asks for the objects a target
holds that its namespace never shows -- the OSD's own, the DNE agent
inodes, the ones with no LMA -- which a scan otherwise counts in
ss_class and keeps back.  Without it "every object on this device" and
"every object a walk would find" would be the same question, and only
one of them is what a target scan is for.

The counters are one accounting rather than several: ss_seen is
incremented once per object, in the pre-filter, so a skip the backend
raises after that point adds to ss_skipped alone.  ss_seen equals
ss_filtered plus ss_skipped plus the sum of ss_class on any completed
scan, and llapi_scan_device_test asserts it -- an inequality would not
have caught counting the same object on both sides of the pre-filter.

An object with no LMA is answered with its IGIF rather than with the
parts to build one from.  The inode number and generation of such an
object are its FID -- osd_scrub.c builds the same one, and it is not a
reconstruction, which is LFSCK's job -- so lfsr_fid carries it and
LLAPI_SCAN_FID is set.  That leaves the generation with no consumer of
its own and it is not a field: what remains is lfsr_ino, which every
object has and which names the ones no IGIF can, a ZFS object id being
wider than an IGIF holds.  There is no LLAPI_SCAN_GEN bit for it.

llapi_scan_namespace.3 gains the pointer to this page here rather than
in the patch that wrote it, where it would have referred to a page the
tree did not yet have.

A worker that cannot be initialised degrades the scan rather than
failing it, once one worker exists, as the pthread_create() arm beside
it does: a backend opens the device again per worker,
lfsp_thread_count is a __u8, and the cursor feeds whatever workers
there are, so running out of them is not a reason to turn a complete
result into an error.

The page says what stx_attributes carries across scanners.  A device
scan narrows the inode's own flags to the set llite also declares, so
one file answers "lfs find -attrs d" the same way whichever scanner
ran, and so_attrs_mask declares that set in the record's
stx_attributes_mask -- which is what lets a consumer tell "not
immutable" from "this scanner cannot say".  ERRORS also spells out the
two EINVAL cases a caller is most likely to trip, a short lfsp_size
and a short ss_size.

A directory's own stripe takes LMV_MAGIC_V1.  LMV_USER_MAGIC is how
the tree marks a *default* LMV: cb_get_dirstripe() sets it exactly
when fp_get_default_lmv is asked for and LMV_MAGIC_V1 otherwise, and a
default lives in trusted.dmv, which is never read here -- trusted.lmv
is the directory's actual stripe.  llite fills LMV_MAGIC_V1 for
LL_IOC_LMV_GETSTRIPE, so LMV_USER_MAGIC here would have the two
scanners answer one striped directory with two different values in the
same field -- the opposite of what lfsr_lmv promises, which is that
the field means one thing whichever scanner filled it.

No in-tree consumer reaches that difference from a scan record yet, so
this is a contract broken rather than a wrong answer observed.
lmv_dump_user_lmm() is what it would break: the magic decides its
"(Default)" prefix and which fields a bare -v shows, and
llapi_lov_dump_user_lmm() accepts both values, so a caller pointed at
a scan record instead of an ioctl reply would print a device scan's
directories as defaults.  The size is unchanged either way,
lmv_user_md_size() growing only for LMV_USER_MAGIC_SPECIFIC.

A parent FID the scan will not vouch for is left cleared rather than
written: LLAPI_SCAN_PARENT is set only where fid_is_sane() accepts
what trusted.link held, and the field follows the bit, so a caller
reading it without testing lfsr_valid does not get the raw xattr.

llapi_scan_device.3 gives lfsr_parent_fid an entry of its own, since
LLAPI_SCAN_PARENT is both what a caller puts in lfsp_want and what it
has to test in lfsr_valid, and a field named in neither place is a
field a caller cannot use.

run_tests() in lustre/tests/llapi_test_utils.{c,h} is split so that
run_test_tbl() takes the table: the new test is handed a device where
the others take a mountpoint.  llapi_scan_test, llapi_root_test and
llapi_pool_test share that helper, which is why the split is worth
naming here rather than left to be read out of the diff.

so_flags is narrowed to the attributes a namespace scan can also
report.  ll_dir_ioctl() puts only IMMUTABLE, APPEND and, under
HAVE_LUSTRE_CRYPTO, ENCRYPTED into stx_attributes_mask, and
llapi_scan_namespace() takes stx_attributes from that mask -- so
reporting COMPR and NODUMP here would make one file answer "lfs find
--attrs d" differently depending on which scanner ran, where
stx_attributes is meant to mean one thing whichever filled it.  The
bit is on the MDT inode either way: making it answerable is llite's to
do, for both scanners at once.

so_attrs_mask carries that same set into the record's
stx_attributes_mask, rather than the scanner assuming one set for
every backend: a bit clear there is "cannot say", which is a different
answer from the attribute itself being clear.  It is carved from
so_padding, so every offset above it is unchanged, and a backend that
leaves it zero has its records leave LLAPI_SCAN_ATTRS clear rather
than claim attributes nothing vouches for.

scan_lmv_to_user() clears the shard area, not just the header.  One
buffer serves every object a worker meets, and a directory's converted
LMV carries the real lum_stripe_count while the size answered is
header-only -- the shard FIDs being left out on purpose, an MDT index
needing an FLD lookup this has no client to make.  A consumer sizing
lum_objects[] by the count rather than by lfsr_lmvsize, which
lmv_dump_user_lmm() does, then read whatever the last object left
there: a preceding foreign directory's opaque value, in full.  The
clear now reaches as far as such a read can, bounded by the room
before a count off a device measures anything, so what it finds is
zeroes.  lfsr_lmvsize is the measure either way, and the header says
so.

The buffer is a union of the two structures it holds rather than a
bare char[], which carries neither one's alignment while the call site
casts to both.

An unreadable inode is counted by the chunk that owns it and no
other.  The skip arm continued without asking either question the
readable path asks below it, so an inode just past end_ino was
consumed by this chunk and read again by the chunk starting there,
and ss_seen and ss_skipped each took it twice; a reserved inode that
could not be read was counted as an object, where a readable one is
not.  ext2fs_get_next_inode_full() assigns *ino before returning any
of the three errors this arm catches -- unlike the errors above it,
which leave it untouched -- so both questions can be asked here.
Nothing was ever delivered twice: it was the accounting that
overstated, which is what the counters exist to be trusted for.

fid_is_root() moves to the UAPI header and the scan calls it.  It was
in lustre_fid.h by history rather than by need: LU_ROOT_FID and
lu_fid_eq() are both already public, so the helper is a one-line
predicate over two things userspace has, and userspace now has the
same question to ask -- a scan of a target has to tell the root from
the rest of FID_SEQ_ROOT.  unlikely() is dropped, that header being
compiled in userspace too, and the definition is removed from
lustre_fid.h, which includes the UAPI one, so every existing caller in
llite, lmv, lod, mdd and mdt is unaffected.

That also settles a test that had drifted.  lu_fid_eq() is a
whole-struct compare, so a hand-written test of f_seq and f_oid called
a FID in FID_SEQ_ROOT with FID_OID_ROOT and a non-zero f_ver the root,
where fid_is_namespace_visible() on the MDT would not.  The narrower
point the spelling exists for is unchanged: the echo client's root
shares the sequence and is still not visible.

Both scanners fill lfsp_stats.  The structure arrives with this patch,
which is why the walk did not fill it before -- the field came in with
the code that needed it, not with the walk -- and there is no reason
for it to stay that way: a walk counts an object seen once, then
filtered, skipped, or delivered and namespace-visible, which is every
class a walk can meet, and the same seen == filtered + skipped +
sum(class) holds.  It counts atomically rather than per-worker,
llapi_scan_namespace() running its callback on every thread at once
where the device scan merges per-worker counters at the end.
lfsp_stats is checked against LLAPI_SCAN_STATS_MIN_SIZE there too, and
after the copy-in for the same reason: a short lfsp_size may not reach
the field.

The struct comment says what the field is rather than which scanners
use it.  llapi_scan_device.3 also says the stats structure is
extensible -- ss_size negotiates it exactly as lfsp_size does, so
counters appended later leave an older caller reading its own fields
-- which was true and written down nowhere.

The struct comments describe the fields rather than which scanner uses
them.  What each scanner offers lfsp_filter, and what each takes 0 in
lfsp_want to mean, were spelled out in the header and again on both
pages, and the header's copy is the one that goes stale: it claimed a
device scan "offers no path and no name", which LU-20637 makes untrue
two patches later by naming an OST object from the parent FID in
trusted.fid.  What is left is the part that does not change -- the
filter sees only what the scan already had, and 0 means a default the
two scanners do not share -- with each page saying which fields that
is for itself.  llapi_scan_device.3 gains the one sentence it was
missing, what 0 means there.

stack_trap is called without EXIT, which is its default, and the
setupall after stopall carries no justification -- it is the ordinary
shape and needs none.  The base tree passes an explicit EXIT eighteen
times of ninety-two; this series had added thirteen more, so dropping
ours leaves the file at the eighteen that were already there.

The EXAMPLES fragment escapes its newline with a single backslash.
troff reads that as a register reference, so the page rendered
printf("%llu files; %llu seen, %llu skipped0, with the escape and the
closing quote swallowed.  It needs the doubled form that
llapi_fid_parse.3 and llapi_scan_rec_path.3 already use.

Signed-off-by: Hiroshi Nishida <hnishida@thelustrecollective.com>
Assisted-by: ClaudeCode:Opus-5 llm_code_and_review_tools
Change-Id: I8a28a8173b0b8f03ba8f9dc5a58a821118ac6296