The visual cortex is organized into a primary visual cortex and secondary visual areas. The primary visual cortex is located in the occipital lobe and receives direct input from the retina via the lateral geniculate nucleus. It is composed of six layers and contains vertical columns that process visual information like color, orientation, and motion. The secondary visual areas surround the primary cortex and further analyze and interpret visual information through two pathways - a fast pathway for position and motion, and an accurate pathway for color and detail. Removing the primary visual cortex causes blindness while removing secondary areas causes difficulties recognizing objects and reading words.