Welcome to The QA Manual, compiled 2005-10-03, for distribution with QA 0.9.
QA is a system that helps the developer manage specification files, test cases, test run results, and test run reports (html output). Hopefully, by making QA easy to understand and handle, more people will do it!
This program and the accompanying documentation is released under the
GNU General Public License, version 2. See file COPYING in the
distribution for a copy of this license. In Emacs, you can type
C-h C-c.
The QA home page is
<http://www.glug.org/people/ttn/software/qa/>.
And what is good, Phædrus, And what is not good... Need we ask anyone to tell us these things?--Robert M. Pirsig, Zen and the Art of Motorcycle Maintenance
In the pursuit of quality, one can go insane not knowing where the boundaries of the problem lie. If we are to assure quality, we need to make a choice as to how to approach the question, and perhaps even more importantly, figure out: what is the question? Ultimately, we find that how we ask the question is a large part of the answer.
This section describes the QA methodology upon which QA is built.
The QA process is intertwined with that of product development, although the bulk of the work is visible only after a drop. Before the drop, however QA is involved with the following areas:
The documentation aspects of QA arise because at the heart of QA's methods is a comparison between what is "right" and what is "actual". Defining what is correct is the responsibility of everyone on the team (both development and QA), but it falls on QA's shoulders the burden of documenting the product, both at the feature (user, black-box) level and at the component (white-box) level, because such specification directly leads to the production of test cases.
After the drop, QA does test runs to determine the build confidence level and publishes the result when done. This includes the build label, the overall confidence level, and a breakdown of what tests passed and what failed.
When bugs are found they are entered into the bug-tracking system, and new bugs are also converted into test cases for regression testing.
The actual front end used to administer test cases and test runs is QA, the subject of the following chapters in this manual.
For black box test cases, there are basically three areas covered:
The first area must be done manually, and can be done by anyone, not just QA. The test cases here are mostly the result of issues raised over time, rather than prepared beforehand.
The second area is amenable to automatic tools, such as a dead-link checker. The test cases will be to run the tool over the build expecting zero errors and only a reasonably small number of warnings.
The third area must be done manually. High-level specification is used to formulate the test cases.
For white box test cases, the component specification is used to formulate the test cases. There is a test harness usually written in the language of the software implementation that facilitates the test run process. Test cases include code that fits within this harness.
A particular test case can be automated by attaching to it the
automatic property. In this case, the action to be performed
should be a single Emacs lisp form that properly drives the (possibly
external) test harness. The form is eval'ed with the default directory
set to that of the test case database. The result should be a string
that can be compared with the expected-result.
It's good to agree on what we're talking about.
The specification is some written, version-controlled document describing the product at various levels of abstraction. It includes a feature list targeted at the end user and also lower-level descriptions of the product components, their interfaces, and their behavior. Although too much detail is discouraged (a pain to write, and hard to maintain), enough information must be available for QA to produce at least one test case per feature or component.
In the absence of a proper engineering specification, a marketing spec (just the feature list) should be used as the basis for fleshing out a full spec. In no cases should QA write a test case without referencing some part of this spec.
The test case is a description of how to go about doing a test run (see below). It must include the following information:
The main parts of the test case are the action and the expected result, both of which should be as specific as possible. Avoid ranges of behaviors, using instead simple directives and descriptions. For example, the action "type in host name" is still considerably vague; the host name might be a single word, or a partially-qualified name, or a fully-qualified name. Furthermore, should the host be a valid host or should it a new name? In this case, we would want something along the lines of "type in FQDN of existing host and press Enter".
Similarly, the expected result should be as specific as possible within the defined scope of the test (see below).
The test platform includes information such as browser, OS, and associated version numbers.
The specification reference is section or paragraph number in the specification (see above) that this test case is attempting to cover. This field should never be blank.
The scope of the test is colloquially divided into "black box" and "white box". These indicate the rough division between end user and programmer points of views.
Black box testing is to ensure that the end user experience is, in order of priority:
The functional aspect is important but note that it is listed last. From the user's perspective, consistency and predictability are stressed, following from the principle of least-surprise.
From the end user's point of view, functionality is definable by the product and so this area of testing looks at the connection between what the product says it will do and what it actually does. Thus, we consider high-level explanation (i.e., narrative) to be in the scope of black box testing.
White box testing is any testing that does not directly address the end user experience, and often makes use of methodologies that are unavailable to the end user. Here, the primary goal is to ensure proper functionality of the product as given in the spec (see above).
White box testing involves testing the components that make up the product, using an environment that has visibility into the component interfaces.
The scope of a test case need not be limited by "black box" and "white box". Especially for white box testing, it is more useful to consider the functional areas that are being tested, for example "database interface" or "session management".
A test run is the application of a test case to a product instance. Thus, a test run includes the following information:
The timestamp and test case id together uniquely identify a test run. Similarly, the product build id uniquely identifies an instance of the product, i.e., a particular build that is being tested. These forms of identification are important in order to avoid duplication in testing.
The actual result of the test follows the form of the expected result (which is influenced by the scope of the test). As the field name implies, the contents are based on fact and thus should be stated in the past tense. For example, if the expected result is "widget changes color to green", and such behavior was indeed observed, the actual result would be "widget changed color to green". Although these grammar rules may seem nit-picky, they do serve to emphasize that actual results means empirical observation.
Finally, the status is one of "pass", "fail" or "inconclusive".
This term is used when the development team hands to the QA team a product build deemed featureful and stable enough for testing using QA methods (as opposed to any pre-integration testing that developers may do themselves). The build must be tagged with a unique label (perhaps an increasing integer or dotted-integer, or the date) before drop. It may be necessary to include in the drop development tools that are changing over time, as well, to ensure reproducibility in the build. These tools should be labeled with the same name.
The mechanics of a drop are simple: take a snapshot of the build, label it, and place it somewhere where QA can wrap black-box or white-box environments around it. Ideally, the drop is relocatable by QA, although sometimes this may not be possible.
The confidence level describes the overall result of all the test runs. Many complicated formulas are possible, but for now we use:
confidence_level = percent_tests_possible * percent_tests_passed
This formula takes into account both QA and developer responsibilities. Because the right-hand-side elements are percentages, the result is also expressed as a percentage or as a decimal number between 0.0 and 1.0.
Interaction with QA is based on a session concept. In some ways, this is like "logging in" in that it establishes context for subsequent actions. To begin a session, use M-x qa-begin-session. Likewise, when you are finished, use M-x qa-finish-session.
| qa-begin-session spec-file | Command |
| Find spec-file and synchronize internal data structures. |
| qa-finish-session | Command |
Calls qa-show-session one last time and then resets all internal
variables.
|
As a convenience, there is a session log available that QA updates on
certain occasions. You are free to add notes and review the log at any
time with the commands qa-log and qa-show-session.
| qa-log msg &rest args | Command |
Log timestamped msg, which may contain formatting sequences.
args are used to consume formatting sequences. (This uses
format.) When called interactively, msg must be a simple
string (without formatting sequences).
|
| qa-display-session-log | Command |
| Display the QA session log in a buffer called "*QA Session*". |
| qa-edit-tcdb | Command |
|
Edit a test case database.
This is basically a wrapper around Each test case entry is shown in the display buffer by itself. Also, in another window, the text of each case's referenced spec-item is shown in a buffer named "*QA Spec Relevance Chain*", as well as parent nodes of the spec item, as far up the chain as possible. This is known as viewing mode, and
To edit the current record, type When done editing, type |
| tcdb-add-case | Command |
|
This command adds a test case to the database.
The spec-item reference is taken from the current test case, and the sequence number is determined by adding 1 to the largest sequence number for the current spec-item. A new test test is created, with the action and expected-result fields cleared, ready for input. |
| tcdb-delete-case | Command |
Currently, this command prints a warning message only.
If you really want to delete a test case, use M-x
db-delete-record. [TODO: Figure out synch schema.]
|
| tcdb-toggle-automatic | Command |
Toggle the automatic property of a test case. If the test case
has no text in its action field, this command (when toggling to
t) also sets up a LISP comment and an empty call to
qatr-run. See Automation.
|
| tcdb-force-redisplay | Command |
| This command redisplays the current test case, including the spec-item text from the spec buffer. |
| tcdb-jump | Command |
This command asks you for a spec item reference and then jumps to the
test case that uses that reference. When jumping, the index of the test
case you left is saved. This is useful for tcdb-other.
|
| tcdb-other | Command |
| Jump back to the the previous test case where you jumped from. This repeated invocations of this command bounce back and forth between two test cases. This can be useful when the test cases are similar, or refer to each other in some way. |
| tcdb-exit | Command |
| Offer to save the database if needed, then quit. This command also deletes the relevance chain buffer (and window). |
All the preparation of the spec file and test cases is for the sole purpose of doing a test run. A build is specified and then each test case is applied to the build. The actual application can be manual (done by a human being) or automatic (done by a script). This is all specified on a per-test-case basis.
Aside from the application of the test case to the build, the rest of the test run is basically an exercise in accounting. This is where QA comes in - to help you keep track of the test run results in as painless a way as possible.
To start a test run, use qa-run-tests.
| qa-run-tests | Command |
|
For each test case, prompt for application and record the results.
If a test case is has the Sometimes the actual result matches the expected result but you don't
want to type in the whole string (which can be error-prone, as well).
In this case, use You may also skip the test case by typing After all the results are recorded, the test run is scored as an aggregate. [There will be hooks for bug reporting in a future version...] |
[Interruptible test runs are not supported.]
[TODO]
[TODO]
Here is documentation for internal concepts, variables and functions. You should not rely on these to be stable in their name or methods. They are included for completeness (and to help the developers keep track of things).
| qa-session | Variable |
Non-nil if in the midst of a QA session.
Has useful properties hanging off of it.
|
| qa-get prop | Function |
| qa-put prop val | Function |
These wrap get and put of qa-session properties.
prop and val are treated as usual.
|
| while-qa-session | Macro |
| A macro to enforce qa-session relevance. |
qa-begin-session: Beginning a QA Session
qa-display-session-log: Beginning a QA Session
qa-edit-tcdb: Editing Test Cases
qa-finish-session: Beginning a QA Session
qa-get: Internals
qa-log: Beginning a QA Session
qa-put: Internals
qa-run-tests: Starting a Test Run
tcdb-add-case: Editing Test Cases
tcdb-delete-case: Editing Test Cases
tcdb-exit: Editing Test Cases
tcdb-force-redisplay: Editing Test Cases
tcdb-jump: Editing Test Cases
tcdb-other: Editing Test Cases
tcdb-toggle-automatic: Editing Test Cases
while-qa-session: Internals
qa-session: Internals