Design of Experiment – Don W. Lail, Jr. & Chris Roe
Data Collection – Don W. Lail, Jr. (Based on on-campus availability.)
Data Analysis – Don W. Lail, Jr. (Quantitative), Chris Roe (Qualitative)
Presentation & Review – Don W. Lail, Jr. & Chris Roe
Out of 25 respondents to the original survey, 15 made appointments to set up a
trial and actually play the game. Subjects were all given the same introduction and
were allowed the same five minutes of free-play time before their ten-minute
The context of the trial included an area that encompassed a village and hill that
were both guarded by enemy forces. In order to complete the objectives, the
subject had to move through the environment and deal with/hide from enemies.
(Subjects were given 5 minutes to free-play in this environment. They were
allowed to move through the environment in any way and experiment with the
controls/actions of the game. Should they reach the end of the test area, the
practice time would end.)
Diagnostic Objectives (achieved during practice time):
Lie down and change camouflage.
Equip the Solid Eye and change to NV and then back to normal.
Change between Mk2 pistol and stun knife (and back).
(During this time, subjects were asked to complete 9 objectives within 10 minutes.
They were given the opportunity to see someone else play through once before
doing it themselves.)
Test Objectives (achieved during the play-test):
1. Shoot an enemy soldier at range.
2. Take down an enemy soldier in close-quarters-combat (fist or knife).
3. Purchase a special bullet for the Mk2 in the shop and shoot someone with it.
4. Reach a difficult to get item by grabbing a ledge while rolling (two choices).
5. Move a dead, sleeping, or knocked-out body to an inconspicuous location.
6. Throw an empty magazine to make a guard look the other way.
7. Sneak around/past an active enemy.
8. Move to the next area without causing an alert.
9. Make it to the end of the level or time limit without dying.
All subjects were required to use the same hardware/software to complete the
trial. A Playstation 3 with a Dualshock 3 wireless controller were used in
conjunction with a 50-inch HDTV which the subjects used to play the introduction
to the second chapter of the game Metal Gear Solid 4: Guns of the Patriots.
Subjects were screened during the survey process to ensure that they were
currently a Purdue Calumet student.
• There are more actions available in Metal Gear Solid than buttons/inputs
available on the controller (input device).
• Metal Gear Solid 4 utilizes context and button/input combinations to provide
70+ actions on a 17-button controller.
• Is this effective?
• Can players be reasonably expected to adapt to and understand this solution?
• Does this solution apply well to the target audience?
• Who is the target audience?
Genre: Metal Gear Solid 4 is commonly considered an action/adventure game.
*Metal Gear Solid defines itself as “Stealth Action”, or more specifically “Tactical
So, we are comparing usability between: (Source: MobyGames.com)
Genre: Action Games
Perspective: 1st-Person Perspective, 3rd-Person Perspective
Themes: Stealth, Shooter, Spy/Espionage
So, First-and-Third-Person, Non-Shooter Action Games and First-Person, Shooter
First/Third Person, Non-Shooter Action Games
First-Person, Shooter Action Games
All subjects must identify themselves by their experience with action and FPS
• Subjects affiliated strongly with FPS (more than two-thirds of Category 1, less
than one third of Category 2) will be grouped into the FPS trials.
• Subjects affiliated strongly with Action (more than two-thirds of Category 2,
less than one third of Category 1) will be grouped into the Action trials.
• Subjects that do not affiliate with either category (less than one third of both
categories) will be grouped into the Inexperienced trials.
Subjects were surveyed before the trial to determine their past experiences.
The full trial consisted of:
An outline of trial objectives for the subject’s review
A live demonstration by another player
A five-minute free-play time before their ten-minute trial
The definitions of action-game-player and FPS-player changed due to the fact that
none of the 25 survey respondents or the 15 participants in the trial fit in the
action or FPS categories as defined. When looking at the data in the review of the
experiment, we looked at these factors as ratios/percentages instead of distinct
In order to prove the effectiveness of Metal Gear Solid 4’s solution to the
action/input problem, we will make the following hypotheses:
HO : Previous game experience does not affect performance on MGS4 trials.
HA1 : Previous game experience negatively affects performance on MGS4 trials.
HA2 : Previous game experience positively affects performance on MGS4 trials.
More specifically, we could define variables as:
SP : Shooter Performance (Average number of objectives accomplished per FPS trial)
AP : Action Performance (Average number of objectives accomplished per action trial)
UP : Un-Experienced Performance (AVG # of objectives accomplished per other trial)
Unfortunately, it is impossible to directly compare these values in our hypotheses.
Therefore, it is necessary to compare these to a pre-established, arbitrary number.
It would be preferred to use the same value across hypotheses in order to make
comparisons easier in recommending further study, or deducing implications of
HO1 : UP ≤ 60% (Un-experienced players will accomplish less than 60% of the
HA1 : UP > 60% (It would be interesting to find that un-experienced players can
accomplish over 60% of the objectives.)
HO2 : SP ≥ 60% (FPS players will accomplish more than 60% of the objectives.)
HA2 : SP < 60% (It would be interesting to find that FPS players can accomplish under
60% of the objectives.)
HO3 : AP ≥ 60% (Action game players will accomplish more than 60% of the objectives.)
HA3 : AP < 60% (It would be interesting to find that action game players can accomplish
under 60% of the objectives.)
The look and placement afford behaviors other than those intended.
Does Metal Gear Solid 4 afford behaviors other than those intended?
This is ultimately the point of the game.
Let’s look at an example:
The look and placement afford behaviors other than those intended.
You are approaching a guard from behind. If he spots you, he will raise an alarm.
He hasn’t noticed you yet, but as you get closer your footsteps have a higher
chance of giving you away. He stands about five feet in front of you and acts as if
he is going to turn around. Which of these actions do you perform?
The look and placement afford behaviors other than those intended.
Accessing an item menu
Accessing a weapon menu
Calling for advice over a radio
Climbing over/onto objects
Crawling into/through small spaces in first-person view
A "punch, punch, kick" style melee system
Grabbing a guard to throw them
Grabbing a guard from behind to choke them
Looking around in first person view
Shooting while kneeling, crawling, or running
Knocking on walls (used to just be punching toward a wall)
Leaning on walls (while standing or crouching)
Moving along walls (while standing or crouching)
Seeing around corners while pressed against a wall
Accessing elevators, computers, or other devices
Shooting while riding in a vehicle
Hanging over a ledge
Doing a chin-up while hanging on a ledge
Shooting in first person view
Leaning in first person view
Leaning out from a corner while pressed against a wall
Jumping out from a corner to shoot
Executing a forward roll
Holding up a guard
Grabbing a guard from behind and to drag them
Grabbing a guard from the front
Grabbing a guard and disarming them
Grabbing a guard and using them as a shield
Grabbing a guard and interrogating them
Grabbing a guard and cutting their throat with a knife
Throwing a guard on the ground and holding them up as they lay down
Shooting while hanging from a ledge (in first or third person mode)
A system for selecting up to 8 weapons at a time to use in a sub-menu
A system for selecting up to 8 items at a time to use in a sub-menu
A camouflage system that requires the use of face paint and clothing to
blend in to the environment
Slow "stealth" walking and crawling
Using a knife
Moving while aiming in first-person view
Rolling in a drum/barrel
Tapping someone laying down on the back to retrieve items or wake
Controlling a small robot that can move and attack
Crawling on your stomach or back (at multiple speeds)
Shooting while laying on your back
Stabbing an enemy with a sensor to track their movements
Walking while crouched
Does not fit to what was expected. (Example: Proximity to similar functions.)
Players consistently found it surprising that they could not climb over waist-high
Players had difficulty understanding that an additional button needed to be
pressed to grab onto a ledge while falling.
Players had difficulty understanding why equipping an item required three
separate actions. (Much confusion over which menu did what.)
What you do and what you get look/feel similar.
Everyone achieved the objective to shoot a soldier at range. The L1+R1 button
combination felt appropriate enough for everyone to figure it out easily.
Because of the low difficulty level, a soldier would fall asleep after being hit by a
sleeping dart at any spot on the body. Most subjects felt satisfied and that the
response was appropriate, but some high-level players found it odd.
All subjects were able to tell immediately when they had been spotted, though
most of the time they didn’t seem to be sure why.
Planning and decision-making under stress.
Subjects found the looming time limit to be difficult, and were consistently
forgetting objectives altogether because of the pressure.
After attempting an objective once or twice, players would sometimes skip to the
next objective on the list and never return (even if they had time left or had much
better opportunity than before).
Ability to perceive color, detail, distance.
Subjects were shown how to use a night-vision item to see in low-light areas.
Several locations in the game were so dark that no detail could be seen. Players
often made mistakes in these locations because they couldn’t clearly tell the
distance between the player character and an enemy, or the direction either was
A single color scheme was used for the whole environment, making everything
appear to be a shade of blue/green. Some subjects had a hard time spotting
enemies moving at a distance because they blended in to the background.
Alarms, false alarms, occupational noise.
The alert noise did an excellent job of letting subjects know when they had been
False alarms: Subjects often heard enemies saying things like “enemy spotted” in
reaction to non-player characters on the map. Sometimes subjects took this as a
sign that they had been spotted.
Gunshots from an ongoing battle between enemies and other non-player
characters often caused inexperienced subjects to immediately take cover and
wait for the fighting to end, even though it did not have anything to do with them.
Top-down processing vs. bottom-up processing. Depth cues.
Top-down processing – Subjects familiar with first-person shooter games felt
accomplished and closer to completing objectives when they shot everyone in the
area. Only after doing so did they realize that some of the objectives could no
longer be completed.
Bottom-up processing – It was easy for most subjects to see the relation between
weapon and item menus because the item menu (on the left) was accessed with
the left trigger, and the weapon menu (on the right) was accessed with the right
trigger. Once one was learned, both were known and remembered.
Pop-out effect. Gestalt organizing principles: proximity, closure, continuity,
Pop-out effect – Soldiers were sometimes difficult to differentiate from the
background. Subjects who preferred shooting often left night-vision active to help
distinguish soldiers from the environment.
Symmetry – The weapon/item menus were easy to understand.
Proximity – The process of equipping an item was difficult because the buttons
and menus were disconnected/discontinuous.
Similarity – The pause menu offered an item menu, weapon menu, and shop
which all looked remarkably similar. This confused subjects to the point where
many gave up using them to achieve an objective. Also, enemy and friendly
soldiers were nearly impossible for subjects to differentiate due to their similar
Storage of information; short (30-90 seconds) and long (nearly permanent) term.
Many subjects forgot objectives along the way even after having reviewed them
on paper and seen them performed. Some even forgot after having attempted an
objective and failing.
Many subjects forgot how to use the shop and weapon selection inside the pause
menu, while they did not experience difficulty using the on-screen item/weapon
In the second half of the level, subjects could never remember the appropriate
path to scale the hill at the end. Some subjects experimented with the same
incorrect route multiple times.
Selecting one choice from a number of choices involving some level of uncertainty.
Most subjects attempted the objectives in order.
Almost all subjects attempted to grab a ledge in the same location, even though
there were two choices.
Subjects had the opportunity to select from four different special bullets to
achieve the 3rd objective. The bullets accomplished generally the same thing,
though in different ways.
Shortcuts that aren’t guaranteed to lead to the best answer, but are more efficient.
Subjects often chose to start shooting all the enemy soldiers they could find
because they felt it would make their objectives easier (despite the fact that some
objectives required the soldiers to be alive).
None of the subjects attempted to use any of the other items or weapons
Skilled behavior which can extract, integrate, assess, and act upon stimuli.
One piece of equipment—the Solid Eye—which was shown to subjects during
their free-play time showed a motion-radar in the top corner of the screen which
helps players identify the locations of nearby enemies. No one showed that they
noticed it at all.
During each trial in which the subject was spotted and caused an alert, there was
an instance where an enemy that the subject saw before had now moved to a
location that the subject did not expect. No one seemed to be keeping track of
Many subjects did not notice when enemy soldiers woke each other up (after
being incapacitated), even when there was dialogue heard and displayed onscreen.
The following is the most relevant data collected during our project.