State transition testing involves designing tests to trigger valid and invalid state transitions of a system. It is used when a system's features are represented as a finite number of states that transform based on rules. Tests are created for each initial state by identifying the input, current state, and expected output. An example is testing an ATM that allows a withdrawal if funds are available but refuses if the account is empty due to a prior withdrawal.