Instruction selection in LLVM maps the compiler intermediate representation (IR) to target instructions by matching nodes in a selection DAG. It performs this mapping greedily by choosing instructions with higher complexity that can cover more operations. The selection DAG is legalized to ensure operations are supported by the target before instruction matching using patterns defined in tablegen files.