Re: [PATCH v2 4/4] selftests/resctrl: Add non-contiguous CBMs CAT test

[Maciej Wieczór-Retman]

Hello!

On 2024-01-18 at 09:15:46 -0800, Reinette Chatre wrote:
>Hi Maciej,
>
>On 1/18/2024 4:02 AM, Maciej Wieczór-Retman wrote:
>> On 2024-01-17 at 10:49:06 -0800, Reinette Chatre wrote:
>>> On 1/17/2024 12:26 AM, Maciej Wieczór-Retman wrote:
>>>> On 2024-01-08 at 14:42:11 -0800, Reinette Chatre wrote:
>>>>> On 12/12/2023 6:52 AM, Maciej Wieczor-Retman wrote:
>
>>>>>> +	bit_center = count_bits(full_cache_mask) / 2;
>>>>>> +	cont_mask = full_cache_mask >> bit_center;
>>>>>> +
>>>>>> +	/* Contiguous mask write check. */
>>>>>> +	snprintf(schemata, sizeof(schemata), "%lx", cont_mask);
>>>>>> +	ret = write_schemata("", schemata, uparams->cpu, test->resource);
>>>>>> +	if (ret)
>>>>>> +		return ret;
>>>>>
>>>>> How will user know what failed? I am seeing this single test exercise a few scenarios
>>>>> and it is not obvious to me if the issue will be clear if this test,
>>>>> noncont_cat_run_test(), fails.
>>>>
>>>> write_schemata() either succeeds with '0' or errors out with a negative value. If
>>>> the contiguous mask write fails, write_schemata should print out what was wrong
>>>> and I believe that the test will report an error rather than failure.
>>>
>>> Right. I am trying to understand whether the user will be able to decipher what failed
>>> in case there is an error. Seems like in this case the user is expected to look at the
>>> source code of the test to understand what the test was trying to do at the time it
>>> encountered the failure. In this case user may be "lucky" that this test only has
>>> one write_schemata() call _not_ followed by a ksft_print_msg() so user can use that
>>> reasoning to figure out which write_schemata() failed to further dig what test was
>>> trying to do. 
>> 
>> When a write_schemata() is executed the string that is being written gets
>> printed. If there are multiple calls in a single tests and one fails I'd imagine
>> it would be easy for the user to figure out which one failed.
>
>It would be easy for the user the figure out if (a) it is obvious to the user
>what schema a particular write_schema() call attempted to write and (b) all the
>write_schema() calls attempt to write different schema.

Okay, your comment made me wonder if on error the schemata still is printed. I
double checked in the code and whether write_schemata() fails or not it has a
goto path where before returning it will print out the schema. So I believe that
satisfies your (a) condition.

As for (b) depends on what you meant. Other tests that run more than one
write_schemata() use different ones every time (CAT, MBM, MBA). Do you suggest
that the non-contiguous test should attempt more schematas? For example shift
the bit hole from one side to the other? I assumed one CBM with a centered bit
hole would be enough to check if non-contiguous CBM feature works properly and
more CBMs would be redundant.

>
>> On a side note I'm not sure if that's true but I'm getting a feeling that the
>> harder errors (not just test failures) are more of a clue for developers working
>> on the tests. Would you agree that it seems like users probably won't see
>> write_schemata() fail here (if the test execution managed to get to this point
>> without erroring out due to bad parameters or kernel compiled without required
>> options)?
>
>I do agree that users probably won't see such failures. I do not think these
>errors are clues to developers working on the tests though, but instead clues
>to resctrl developers or kernel development CI systems.

Right, I agree, the target group I mentioned was too narrow.

>
>Reinette
>

-- 
Kind regards
Maciej Wieczór-Retman