The pineal gland is reddish-gray and about the size of a grain of rice (5–8 mm) in humans, located just rostro-dorsal to the superior colliculus and behind and beneath the stria medullaris, between the laterally positioned thalamic bodies. It is part of the epithalamus.

The pineal gland is a midline structure, and is often seen in plain skull X-rays, as it is often calcified.

The pineal body consists in humans of a lobular parenchyma of pinealocytes surrounded by connective tissue spaces. The gland's surface is covered by a pial capsule.

The pineal gland consists mainly of pinealocytes, but four other cell types have been identified.

The pineal gland receives a sympathetic innervation from the superior cervical ganglion. However, a parasympathetic innervation from the sphenopalatine and otic ganglia is also present. Further, some nerve fibers penetrate into the pineal gland via the pineal stalk (central innervation). Finally, neurons in the trigeminal ganglion innervate the gland with nerve fibers containing the neuropeptide, PACAP. Human follicles contain a variable quantity of gritty material, called corpora arenacea (or "acervuli", or "brain sand"). Chemical analysis shows that they are composed of calcium phosphate, calcium carbonate, magnesium phosphate, and ammonium phosphate. In 2002, deposits of the calcite form of calcium carbonate were described. Calcium, phosphorus and fluoride deposits in the pineal gland have been linked with aging.

Pinealocytes in many non-mammalian vertebrates have a strong resemblance to the photoreceptor cells of the eye. Some evolutionary biologists believe that the vertebrate pineal cells share a common evolutionary ancestor with retinal cells.

In some vertebrates, exposure to light can set off a chain reaction of enzymatic events within the pineal gland that regulate circadian rhythms. Some early vertebrate fossil skulls have a pineal foramen (opening). This correlates with the physiology of the modern "living fossils", the lamprey and the tuatara, and some other vertebrates that have a parietal organ or "third eye," which, in some of them, is photosensitive. The third eye represents evolution’s earlier approach to photoreception. The structures of the third eye in the tuatara are homologous to the cornea, lens and retina, though the latter resembles that of an octopus rather than a vertebrate retina. The asymmetrical whole consists of the "eye" to the left and the pineal sac to the right. "In animals that have lost the parietal eye, including mammals, the pineal sac is retained and condensed into the form of the pineal gland."

Unlike much of the rest of the mammalian brain, the pineal gland is not isolated from the body by the blood-brain barrier system; indeed it has profuse blood flow, second only to the kidney.

Fossils seldom preserve soft anatomy. The brain of the Russian Melovatka bird, about 90 million years old, is an exception, and it shows a larger-than-expected parietal eye and pineal gland.

In humans and other mammals, the light signals necessary to set circadian rhythms are sent from the eye through the retinohypothalamic system to the suprachiasmatic nuclei (SCN) and the pineal.