The open system call opens a file and returns a file descriptor. It takes a file path, access flags, and optional mode as arguments. The library function open() wraps the system call. It returns the file descriptor on success or -1 on error and sets errno. The program then uses the descriptor for read, write, or other file operations.