New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add support of level of testing in testcase.yaml #51718
Comments
this is related to #51288 |
Adding hierarchial level of testing makes a lot of sense. We should be able to define the level of testing in such a way where we each level includes all lower levels or have some schema for inclusion defined somewhere else, so just as an example: so if we take those levels: (borrowed from here)
I am sure we can have more levels defined.... We could have tests that have:
and some other test having for example
which base on some hierarchy definition would include unit? Some tests might belong to two level, we need to keep things flexible. So instead of how we do things right now by pointing to some test directory directly or use tags, we just specify the level of testing at the command line of twister and do the filtering based on that. |
@nashif What you proposed with levels is very useful and will help with understanding the role of a given test and having a better classification. It also helps with communication with PMs and auditors and can improve QA in general. E.g. when a PM (auditor) ask: do you (what kind of) unit/system/etc tests have, is this area covered? It is a practice to evaluate a master test plan for a given component by the criticality level of it. A certain criticality can have by default a defined requirements for test levels to be covered. |
e.g.
add a meta like 'target' which we recommend to run
|
levels are alreadcy suppported in testcase.yaml |
current testcase.yml meta keywords only defined as
string
but it is used as list. e.g.tags
we shall move it to list structure for more filterable and proporely controlled
testcases attributes shall be added such as what test plan it can be
The text was updated successfully, but these errors were encountered: