COMP 337
Fall 2006
Assignment 10
November 18, 2006
GROUP THREE MEMBERS
Martha Hogan
Ryan Garcia
Shawn Bochat
Bryan Adler
Bryant Belarmino
Allison Henderson
David Diano
TEST PLAN DOCUMENT
Copyright
(c) 2006 Madler Media
SECTION
I: QA TEAM (and areas of responsibility)
SECTION
II: TESTING PROCEDURES
SECTION
III: HOW TESTING REQUIREMENTS ARE GENERATED
SECTION
IV: BUG TRACKING SOFTWARE
SECTION
V: BUG CLASSIFICATIONS
SECTION
VI: BUG TRACKING
SECTION
VII: SCHEDULING AND LOADING
SECTION
VIII: EQUIPMENT BUDGET AND COSTS
SECTION I: QA TEAM (and areas of responsibility) |
i. Office
phone: (800) 555-2839
ii. Home
phone: (805) 555-2099
iii. Cell phone: (805) 555-1093
iv. E-mail: david.diano751@dolphin.csuci.edu
The internal testers will quickly go through the
game making sure there are no major bugs that cause the game to crash
throughout the development. These bugs are documented and given to the
lead programmer who will fix them as soon as possible. They will also be
in charge of going through the bugs that the external testers find and verify
or fix them depending on the severity of them. The following internal
testers will be used (at least one person must have programming skills):
External
testers are the ones that will go through the game with a fine-toothed comb and
pick out all the minor bugs. Any detail that they think violates the
consistency of the game or breaks the sense of immersion will be
documented. They are also responsible for the major bugs that the
internal testers did not catch. The team is:
SECTION II: TESTING PROCEDURES |
1. Bugs (e.g.
critical, severe, moderate, minor)
a. Detect them as soon as
possible after they enter the build.
Our configuration management/source control system will automatically
generate web-based reports when changes are checked into the code
base. The test team will monitor these reports on a periodic basis to
determine areas where substantial change has occurred, and will focus more
attention to these areas. Also, bugs that are marked
"resolved" by the development team will typically
be verified by the test team within one week. This process,
along with our daily regression tests, will facilitate quick detection of
other bugs that may have been introduced inadvertently when fixing a
bug.
b. Research them.
A bug is analyzed by the development team when it
is found. During this bug analysis, the cause of the
bug is determined, it is assigned to someone to fix, and a preliminary
solution may be proposed. The cause of the bug is categorized as
either a result of incomplete or incorrect requirements, faulty
design, a coding error, documentation error, or other reason.
c. Communicate them to the dev team. When a new bug is logged into the system,
email notification is automatically sent to relevant team members. The
bug component is specified when the bug is created, and this component often
dictates which team will be assigned the bug. For example, if the
bug component is "UI" then email notification is automatically sent
to the UI development team; if the bug component is "Art" then the
email notification is automatically sent to the Art team, etc.
d. Help get them resolved. The development team will request help
from other teams as required. Depending on the nature of the bug, the
development team may require assistance from the test team to reproduce the bug
or help in unit testing a fix.
e. Track Them. Bugs
can be easily tracked via the weekly bug report that is available on
the project intranet, where it can be viewed by all team members. This report will contain a summary of new
bugs, resolved bugs, and verified bugs in the past week, and a detailed
listing of all outstanding bugs which can be sorted by various criteria (name,
component, severity, priority, assigned to, etc).
2. Maintain
the Daily Build
A build of the
entire project will be run nightly at 2:00am, and an automated report will
be generated and posted on the project intranet. If the build
fails, email notification will be automatically sent to the team. A
broken build is given the highest priority and must be fixed at
once. Automatic regression tests will also be run on the nightly build,
and new bugs will be logged for test cases that fail.
3. Levels of Communication
Team members communicate in the following different ways:
a. Conversation:
Informal conversation between two or more team members. Notes on
conversations may be kept, as necessary
b. ICQ:
Instant messaging is often how team members informally communicate
throughout the work day, especially if they are not physically located
near each other. Instant messaging conversations may be logged or
printed, as necessary.
c. E-Mail
to Individual: Email is typically used for day-to-day communication
when the subject is more than a quick question or comment suitable for instant
messaging.
d. E-Mail
to Group: Email is automatically sent to appropriate email groups by
the automated build system, configuration management system, and bug
tracking system. Group email aliases are also available to the
project team for easily sending email to a selected group of team
members.
e. Daily
Top Bugs List: The producer, QA Lead, and Tech Lead generate a
daily top bug list when urgent bugs or tasks are identified. This list is
emailed to the relevant team members.
f. Stats/Info
Dump Area on DevSite: The project web site
contains all pertinent project documentation, including statistics on bug
trends, bug summaries, and detailed bug reports.
g. Formal
Entry into Bug Tracking System: Bugs
are formally entered into the bug tracking system when a bug is found in the
controlled source code.
b. Daily Activities
1. Generate a
nightly build. "Ant"
will be used as our build tool, both by the development team and as the nightly
build system. The test team is responsible for installation
and configuration of the nightly build.
2. Run the nightly regression
tests, as described in “Daily Tests” below.
After each nightly build is complete, automatic regression tests
will be run. The test team will assign new bugs for test cases
that fail.
3. If everything is OK, post
the build so everyone can get it. Completed nightly builds are saved in shared
location accessible to the project team. The build web page that contains
status on the nightly build will also contain a link to the completed
build.
4. If there’s a problem, send
an e-mail message to the entire dev team that the new build cannot be copied,
and contact whichever developers can fix the problem. If the
automatic nightly build fails, email notification will be automatically
sent to the entire team. A broken build is given the highest priority
and must be fixed at once.
5. Decide whether a new build needs to be run that
day. Since a broken build is
given the highest priority, it is expected that broken nightly builds will
be fixed ASAP, but no later than 11:00am that day. If a broken nightly
build is fixed before 7:00pm, generally the build will be re-run as
soon as it is fixed. Otherwise, if the
build is not fixed before 7:00pm, generally the next build to run will be
the scheduled nightly build at 2:00am. This practice may be
altered on a case-by-case basis if requested by the producer or any
team lead. For example, if it's 7:45pm and the Tech Lead says we must
re-run the build because some developers are waiting for it,
then normally it will be done.
1. Run through a predetermined set of
single-player levels, performing a specified set of activities
a. Run through of Lumic
Mission
i. Test weapons and check inventory functioinality
ii. Test map
functionality and navigation
iii.
Test effectiveness of diplomacy on A.I.
iv.
Run an automated script that reports the results of the various tests
and post them in the QA portion of the STAR SIFTER internal Web site
b. Run through of Oren Mission
i. Test weapons and check inventory functioinality
ii. Test map functionality
and navigation
iii. Test
effectiveness of diplomacy on A.I.
iv. Run
an automated script that reports the results of the various tests and post them
in the QA portion of the STAR SIFTER internal Web site
c. Run
through of Paphi Malip
i. Test
weapons and check inventory functioinality
ii. Test map
functionality and navigation
iii. Test
effectiveness of diplomacy on A.I.
iv. Run
an automated script that reports the results of the various tests and post them
in the QA portion of the STAR SIFTER internal Web site
2. Run through a predetermined set of
multiplayer levels, performing a specified set of activities
a. Run through multiplayer level "Who's your
friend"
i. Test weapons and check inventory
ii. Test map functionality and navigation
iii.
Check hide-outs and hidden item locations
iv. Each tester involved in the multiplayer
game to run an automated script that reports the results of the various test
and posts them in the QA portion of the STAR SIFTER internal Web site
b. Run through multiplayer level "Entropy and
Activity"
i. Test
weapons and check inventory
ii. Test map functionality and navigation
iii. Check
hide outs and hidden item locations
iv. Each tester involved in the multiplayer
game to run an automated script that reports the results of the various test
and posts them in the QA portion of the STAR SIFTER internal Web site
c. Run through multiplayer level "The small
bang"
i. Test weapons and check inventory
ii. Test map functionality and navigation
iii. Check
hide outs and hidden item locations
iv. Each tester involved in the multiplayer
game to run an automated script that reports the results of the various test
and posts them in the QA portion of the STAR SIFTER internal Web site
3. E-mail showstopper crashes or critical
errors to the entire team
4. Post
showstopper crashes or critical errors to the daily top bugs list
c. Daily Reports
The
automated report of the preceding daily test has been posted to the QA portion
of the STAR SIFTER project internal web site.
d. Weekly Activities
i. Weekly tests
1. Run through every level in the game performing
a specified set of activities and generating a predetermined set of tracking
statistics.
a.
Test Lumic Storyline
i)
(1) Test completion flag
(2) Check reward flag
(3) Check diplomacy card
update flag
ii)
(1) Test completion flag
(2) Test reward flag
(3) Test diplomacy card
update flag
iii)
Mission #3: I’m
an
(1) Test completion flag
(2) Test reward flag
(3) Test diplomacy card
update flag
iv)
(1) Test completion flag
(2) Test reward flag
(3) Test diplomacy card
update flag
v)
(1) Test completion flag
(2) Test reward flag
(3) Test diplomacy card
update flag
vi)
(1) Test completion flag
(2) Test reward flag
(3) Test diplomacy card
update flag
vii)
(1) Test completion flag
(2) Test reward flag
(3) Test diplomacy card
update flag
viii)
(1) Test completion flag
(2) Test reward flag
(3) Test diplomacy card
update flag
b. Test
Oren Storyline
i)
(1) Test completion flag
(2) Check reward flag
(3) Check diplomacy card
update flag
ii)
(1) Test completion flag
(2) Test reward flag
(3) Test diplomacy card
update flag
iii)
(1) Test completion flag
(2) Test reward flag
(3) Test diplomacy card
update flag
iv)
(1) Test completion flag
(2) Test reward flag
(3) Test diplomacy card
update flag
v)
(1) Test completion flag
(2) Test reward flag
(3) Test diplomacy card
update flag
vi)
(1) Test completion flag
(2) Test reward flag
(3) Test diplomacy card
update flag
vii)
(1) Test completion flag
(2) Test reward flag
(3) Test diplomacy card
update flag
viii)
(1) Test completion flag
(2) Test reward flag
(3) Test diplomacy card
update flag
ix)
(1) Test completion flag
(2) Test reward flag
(3) Test diplomacy card
update flag
x)
(1) Test completion flag
(2) Test reward flag
(3) Test diplomacy card
update flag
c. Test
Paphi Malip Storyline
i)
(1) Test completion flag
(2) Check reward flag
(3) Check diplomacy card
update flag
ii)
(1) Test completion flag
(2) Test reward flag
(3) Test diplomacy card
update flag
iii)
(1) Test completion flag
(2) Test reward flag
(3) Test diplomacy card
update flag
iv)
(1) Test completion flag
(2) Test reward flag
(3) Test diplomacy card
update flag
v)
(1) Test completion flag
(2) Test reward flag
(3) Test diplomacy card
update flag
vi)
(1) Test completion flag
(2) Test reward flag
(3) Test diplomacy card
update flag
vii)
(1) Test completion flag
(2) Test reward flag
(3) Test diplomacy card
update flag
viii)
(1) Test completion flag
(2) Test reward flag
(3) Test diplomacy card
update flag
ix)
(1) Test completion flag
(2) Test reward flag
(3) Test diplomacy card
update flag
x)
(1) Test completion flag
(2) Test reward flag
(3) Test diplomacy card
update flag
d. Test Endgame Storyline
i)
(1) Test completion flag
(2) Check reward flag
(3) Check diplomacy card update flag
ii)
(1) Test completion flag
(2) Test reward flag
(3) Test diplomacy card update flag
iii)
(1) Test completion flag
(2) Test reward flag
(3) Test diplomacy card update flag
iv)
(1) Test completion flag
(2) Test reward flag
(3) Test diplomacy card update flag
v)
(1) Test completion flag
(2) Test reward flag
(3) Test diplomacy card update flag
vi)
(1) Test completion flag
(2) Test reward flag
(3) Test diplomacy card update flag
vii) Mission #7: This
(1) Test completion flag
(2) Test reward flag
(3) Test diplomacy card update flag
2. Weekly Review of Bugs in the Bug Tracking System
a. Verify that bugs marked “resolved” by the development team really are fixed. This is the responsibility of the test team. A designated tester is responsible for verifying each bug marked as “resolved”, at which time the bug state will be changed from “resolved” to “verified”. The QA Lead is responsible for reviewing and managing this process.
b. Check the
appropriateness of bug rankings relative to where the project is in
development. The bug rankings are marked
as follows:
Critical – Must be fixed before development can continue.
Important – Must be fixed before moving to next part.
Watch – Bug occurs because a dependant part of code has yet
to be developed.
c. Acquire a “feel” for the current
state of the game, which can be communicated in discussions to the producer and
department heads. The most important
questions are:
i.
Is
it fun?
ii.
How
deep is the immersion factor?
iii.
Is
there a sense of freedom and exploration?
iv.
Is
combat too:
Easy?
Hard?
Repetitious?
v.
Is
the frequency of random encounters too high?
vi.
Are
combat and diplomacy balanced?
vii.
Is
the trading system too easy or difficult?
viii.
Are
the mission objectives varied enough?
ix.
Do
the plot points conflict with each other when done in random order?
x.
Is
there too much or little time between cut-scenes and story advancement?
d. A bug report will be generated on a
weekly basis and available on the project intranet, where it can be viewed by
all team members. This report will
contain:
§
A
summary of new bugs, resolved bugs, and verified bugs in the past week,
§
A
detailed listing of all outstanding bugs which can be sorted by various
criteria (name, component, severity, priority, assigned to, etc).
ii. Weekly Reports
1. Tracking statistics, as generated in the weekly tests above
a. Testing by completing all finished missions in random orders
i)
ii) Completion Flag Report
iii) Reward Flag Report
iv) Diplomacy Card Flag Report
e. Ad Hoc Testing
i. Perform
specialized tests as requested by the producer, tech lead, or other development
team members
The
lead tester will be in charge of handling all tests requested by the
development members. They will be prioritized by how much it impacts the
game regardless of how long it takes to produce results. There will be
several requested tests throughout the development of the game so each one will
be documented so that team members can refer to them if they need to reference
why a change has been implemented.
ii. Determine the appropriate level
of communication to report the results of those tests
All
test results will be handed or emailed to the producer in the form of a report
with graphs and charts to aid in making a quick decision. If an important
decision needs to be made, a full presentation will be given in order to
present how all the data was collected and the meaning of that data.
f. Integration of Reports from External Test
Groups
i. If
at all possible, ensure that all test groups are using the same bug tracking
system
External
test groups will use either Debian
(http://www.debian.org/) or the Distributed Defect Tracking System (DDTS) for
bug reporting.
ii. Determine which group is
responsible for maintaining the master list
The
Producer will hold the master bug list and a final report each week will be
sent directly to him/her. Additional
copies can be accessed upon request for review.
iii. Determine how
frequently to reconcile bug lists against each other
Bug
lists will be collected and reviewed every Monday. The following Wednesday will
have another bug list created to show new bugs and those still pending resolution.
iv. Ensure
that only one consolidated set of bugs is reported to the development team
The
Producer has final say over the bug list. After the Wednesday report, no
alterations can be made to the document until the next week's bug report unless
approved by the Producer.
g. Focus Testing (if applicable)
Delivering
the game for mobile phones will probably be the toughest obstacle this team
will face. The screen is smaller and there are many constraints with memory,
processor speed, and graphics quality. Therefore, this team believes that
an important resource to test is cell phone users to see what they think will
make this a popular game that everyone will want. The marketing team can
then push cell phone carriers to include this game in all of their new phones
increasing the scope of our audience.
i. Recruitment methods
·
Advertising at mobile phone stores/kiosks
·
Bulletins on our website
·
Viral marketing
ii. Testing location
It
should be important to have the testing done close to the development team so
that changes can be implemented quickly and then tested again.
iii. Who
observes them?
At
one point or another, the whole team should take a minute or two to observe our
cell phone testers. Programmers should make sure that there are no major
crashes on the mobile platform and that the interface is intuitive for all
users. The art department
should definitely pay attention to how their art looks on the smaller screen.
Minor adjustments might have to be made if something looks completely different
on a cell phone. While these are quick fixes, it is still important that
everyone try to get feedback from the testers.
iv. How is
their feedback recorded?
Feedback
will be recorded with interviews and questionnaires. Each tester will be
interviewed before the testing process and then asked to fill out a
questionnaire on their experience playing the game on a mobile phone. All
the data recorded will be given to the producer who will decide any important
changes to the game.
h. Compatibility Testing
i. Selection of external vendor
DinoGamz, an independent third party tester, will run
the game against the following platforms: PC, Apple, Playstation
3, Xbox 360 and Nintendo's Wii. This will ensure that
the game is free of bugs that can conflict with each of the different systems.
ii. Evaluation of results
Results
will be collected weekly, and released on Monday to be evaluated by programmers
to ensure that progress is being made in debugging the software. Another group
will be formed to follow up the Wednesday after the report to ensure that each
item has been addressed before the next round of testing begins on Friday.
iii. Method of
integrating filtered results into bug tracking system
Debug.txt
will be released pending Wednesday's results. Fixed bugs and unresolved bugs
will be clearly identified and flagged in the system. Bugs that have been
flagged repeatedly for twp consecutive weeks will be given top priority on
Thursday before testing resumes again on Friday.
SECTION III: HOW TESTING REQUIREMENTS ARE GENERATED |
a. Some requirements are
generated by this plan
Testing will be integral
to the development process. Initial testing for a component or
function will be done once it is checked into source control. After
code freeze, formal integration testing will be done on the entire
system. During this test phase, problem areas will be identified,
and these areas will receive more focus during subsequent
tests. Rigorous testing will be performed on the “problem” sections
with only moderate testing on the remaining portions of the game.
However, if it is discovered that the game possesses relatively few
critical/severe bugs or mainly those of the moderate to minor category, a
moderate amount of testing will be performed on all sections.
b. Requirements can also be
generated during project meetings, or other formal meetings held to review
current priorities
Although
testing of the game usually follows the requirements which have been generated
through initial testing, this is often modified. Team members can decide
that “new” portions of the game which have been recently completed should be
given the highest priority for testing. In addition, a priority may be
placed on determining the specific “in-game” action which caused an error/bug
to occur. Furthermore, team members may agree to give certain areas a
higher priority due to upcoming deadlines.
c. Requirements can also
result from changes in a bug’s status within the bug tracking system
When
a new bug is added or an existing bug's status changed in the bug tracking
system, testing requirements are also updated. For example, further
testing is often necessary for bugs with a high priority. If bugs which
exist in a certain area of the game are changed to high priority, the testing
requirements must be altered to reflect this. Conversely, if the priority
is lowered for the bugs present in a portion of the game, testing is no longer
immediately necessary. Additionally, changes to a bug's status such as “Need
More info” and/or “Can’t Duplicate” which can be present in the description of
a bug may require the testers to provide verification that it is actually
“fixed” or determine the cause.
d. Some requirements are
generated when a developer wants QA to check a certain portion of the game
The
testers may need to “re-test” a certain portion of the game which has been
“fixed”. The change in testing requirements will depend on the severity
of the bug which has been “fixed” (moderate testing will be needed in most
cases). Portions which have been added after the initial test of the game
will also require more testing before it can be released.
SECTION IV: BUG TRACKING SOFTWARE |
a. Package name
The
Distributed Defect Tracking System (DDTS) from IBM (formerly Rational Software)
will be used for bug tracking.
b. How many seats will be needed for the project
15
seats will be purchased initially, with the option to purchase another 15
next year.
c. Access Instructions
DDTS
has a web-based interface. The URL will be provided to the team once
it's installed at our site. DDTS also has a command-line
interface which will also be available to all team members.
d. “How to report a bug” instructions for using the system
SECTION V: BUG CLASSIFICATIONS |
Bugs will be classified according to their
severity and priority. Often, these two fields are related, but not
necessarily so.
Severity states are:
Priority states are:
SECTION VI: BUG TRACKING |
a. Who classifies the bug?
An attempt to classify the bug severity and priority is made by
the person who created the bug. Once a bug is created, email is sent to
the manager and team working on the specific component. The manager
and team members then have the option of reclassifying the severity or
priority.
Anyone on the team has the ability to modify any bug field,
including the severity and priority. If a team member wants to down-grade
a bug (say from moderate to minor), then a courtesy email asking for
endorsement of the change should be sent to the person who initially
classified the bug. Generally, disagreements regarding bug
severity or priority can be resolved this way. If the two people are
unable to reach agreement, then the matter should be brought up at the next
weekly bug-scrub meeting attended by the producer and representatives from
each component team.
b. Who assigns the bug?
As
mentioned earlier, anyone on the team has the ability to modify any bug
field, including the "assigned to" field. If a team member has
not already assigned the bug to himself or herself, the manager can assign the
bug to an appropriate team member.
c. What happens when the bug is fixed?
When
the "assigned to" person resolves a bug, he/she changes
the bug state to "R" (resolved). At this time the
test team is notified via email to verify the fix.
d. What happens when the fix is verified?
When
a tester verifies the fix, he/she changes the bug state to "V"
(verified). At this point the bug is considered completely resolved.
SECTION VII: SCHEDULING AND LOADING |
a. Rotation Plan
There
will be three categories of testers necessary for the project (e.g. “initial
testers”, “main testers”, and “new testers”). The “initial testers”,
which will only be used in this phase of development, will begin the testing by
isolating all problem areas which are present throughout the game. The
“main testers” will remain as testers for the duration of game-testing phase
and thus become familiar with the game and its problems. “New testers”
will be brought on and off every two weeks during the main and re-testing
phases of development. It is hoped that these tester will give different
perspectives on existing problems and expose other game flaws which may have
been missed.
b. Loading Plan
The
“initial testers” will consist of two initial game testers and all of the
“main” game testers. Five “main testers” will be necessary for this
project. A group of five “new testers” should be added after the after
completion of the initial testing phase. The final-testing phase of the
game can be completed entirely by the “main” game testers since most bugs should
be resolved at this point in testing. If new bugs are discovered, they
must be fixed and the game returned to the re-testing phase.
Phases and number of testers needed:
SECTION VIII: EQUIPMENT BUDGET AND COSTS |
a. QA Team Personnel with Hardware and Software Toolset
i. Team member #1 - David Diano
1. Hardware
a. Testing
PC – Dell XPS 700
1.
Specs:
Intel Core 2 Extreme processor x6800
(2.93GHz)
Microsoft Windows XP
2GB Dual Channel DDR2 SDRAM at
667MHz
640GB Performance RAID 0 (2 x 320GB
SATA 3Gb/s 7200 RPM HDDs)
16x
20” UltraSharp
2007FPW Widescreen Digital Flat Panel
Dual 256MB nVidia
GeForce 7900 GS
Sound Blaster X-Fi
XtremeMusic (D) Sound Card
b. Console
Debug Kit (Shared) – Xbox 360
1. 2 Wireless Controllers
2. USB Keyboard & Mouse
c. Console
Debug Kit (Shared) – PlayStation 3
1. 2 Wireless
Sixaxis Controllers
2. USB
Keyboard & Mouse
d. Console Debug Kit (Shared)
– Nintendo Wii
1. 2 Wii-motes
with nunchuck attachments
2. Software Tools Needed
a. Bug
tracking software – Debian and DDTS
b. Communication
software for email, IM, voice/video conferencing
ii. Team member #2 -
Wayne Chu
1. Hardware
a. Testing
PC – Dell XPS 700
1.
Specs:
Intel Core 2 Extreme processor x6800
(2.93GHz)
Microsoft Windows XP
2GB Dual Channel DDR2 SDRAM at
667MHz
640GB Performance RAID 0 (2 x 320GB
SATA 3Gb/s 7200 RPM HDDs)
16x
20” UltraSharp
2007FPW Widescreen Digital Flat Panel
Dual 256MB nVidia
GeForce 7900 GS
Sound Blaster X-Fi
XtremeMusic (D) Sound Card
b. Console
Debug Kit (Shared) – Xbox 360
1. 2 Wireless Controllers
2. USB Keyboard & Mouse
c. Console
Debug Kit (Shared) – PlayStation 3
1. 2 Wireless
Sixaxis Controllers
2. USB Keyboard & Mouse
d. Console Debug Kit (Shared) –
Nintendo Wii
1. 2 Wii-motes
with nunchuck attachments
2. Software Tools Needed
a. Bug
tracking software - Debian
b. Communication
software for email, IM, voice/video conferencing
b. Equipment Acquisition Schedule and Costs
Acquisition timeframe: Q406
Cost: $3500
Acquisition timeframe: Q406
Cost: $10,000
Acquisition timeframe: Q406
Cost: $30,000
to $50,000
Acquisition timeframe: Q406
Cost: $1700