Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support of level of testing in testcase.yaml #51718

Closed
hakehuang opened this issue Oct 27, 2022 · 5 comments
Closed

Add support of level of testing in testcase.yaml #51718

hakehuang opened this issue Oct 27, 2022 · 5 comments
Assignees
Labels
area: Test Framework Issues related not to a particular test, but to the framework instead area: Twister Twister Enhancement Changes/Updates/Additions to existing features

Comments

@hakehuang
Copy link
Collaborator

hakehuang commented Oct 27, 2022

  1. current testcase.yml meta keywords only defined as string but it is used as list. e.g.
    tags
    we shall move it to list structure for more filterable and proporely controlled

  2. testcases attributes shall be added such as what test plan it can be

test_plan:
    -  Acceptance
    -  ...
  1. more
@hakehuang hakehuang added the Enhancement Changes/Updates/Additions to existing features label Oct 27, 2022
@hakehuang
Copy link
Collaborator Author

this is related to #51288

@henrikbrixandersen henrikbrixandersen added area: Test Framework Issues related not to a particular test, but to the framework instead area: Twister Twister labels Oct 31, 2022
@nashif nashif changed the title refacting testcase.yml meta to be more filterable and meaningful structure Add support of level of testing in testcase.yaml Nov 17, 2022
@nashif
Copy link
Member

nashif commented Nov 17, 2022

Adding hierarchial level of testing makes a lot of sense. We should be able to define the level of testing in such a way where we each level includes all lower levels or have some schema for inclusion defined somewhere else, so just as an example:

so if we take those levels: (borrowed from here)

  • Unit Testing : checks if software components are fulfilling functionalities or not.
  • Integration Testing : checks the data flow from one module to other modules.
  • System Testing : evaluates both functional and non-functional needs for the testing.
  • Acceptance Testing : checks the requirements of a specification or contract are met as per its delivery.

I am sure we can have more levels defined....

We could have tests that have:

 level: 
   - unit

and some other test having for example

level: 
  - system

which base on some hierarchy definition would include unit?

Some tests might belong to two level, we need to keep things flexible.

So instead of how we do things right now by pointing to some test directory directly or use tags, we just specify the level of testing at the command line of twister and do the filtering based on that.

@PerMac
Copy link
Member

PerMac commented Nov 24, 2022

@nashif What you proposed with levels is very useful and will help with understanding the role of a given test and having a better classification. It also helps with communication with PMs and auditors and can improve QA in general. E.g. when a PM (auditor) ask: do you (what kind of) unit/system/etc tests have, is this area covered? It is a practice to evaluate a master test plan for a given component by the criticality level of it. A certain criticality can have by default a defined requirements for test levels to be covered.

@hakehuang
Copy link
Collaborator Author

  • about level
level: (can be one or more below)
     -	unit
     - integration
     - acceptance
     - system
     - smoke

e.g.
if filter with 'unit' then only cases with 'unit' tag in level will be run.
if it is blank, then it will run when there is not level filter.
a case can have multiply tags

  • about target platform

add a meta like 'target' which we recommend to run

target:
     -	simulation/hardware/host

@nashif
Copy link
Member

nashif commented Jan 26, 2024

levels are alreadcy suppported in testcase.yaml

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area: Test Framework Issues related not to a particular test, but to the framework instead area: Twister Twister Enhancement Changes/Updates/Additions to existing features
Projects
Development

No branches or pull requests

8 participants