Temporal lobe epilepsy fits the bill perfectly.
Instead of the seizures we typically think of where something moves, or maybe the whole person convulses, partial seizures confined to the temporal lobe can manifest with visual or auditory or olfactory hallucinations or even more bizarre experiences. I met a man who suddenly smelled burning wires, and so of course checked his whole house for a fire, found nothing, and the smell abruptly vanished. The second time it happened he googled it and turned up "temporal lobe seizures" and so got an MRI, which found his brain tumor. These people are not insane, not stupid and in between seizures are completely normal. They recognize the experiences they have during seizures as things that may not be real.
Oliver Sacks wrote several accounts of persons with temporal lobe epilepsy.
from The Man Who Mistook His Wife For a Hat
One night, in January 1979, she dreamed vividly, nostalgically, of her
childhood in Ireland, and especially of the songs they danced to and
sang. When she woke up, the music was still going, very loud and
clear. "I must still be dreaming", she thought, but this was not so.
She got up, roused and puzzled. It was the middle of the night.
Someone, she assumed, must have left a radio playing. But why was she
the only person to be disturbed by it? She checked every radio she
could find - they were all turned off. Then she had another idea: she
had heard that dental fillings could sometimes act like a crystal
radio, picking up stray broadcasts with unusual intensity. "That's
it", she thought. "One of my fillings is playing up. It won't last
long. I'll get it fixed in the morning." She complained to the night
nurse, who said her fillings looked fine. At this point another notion
occurred to Mrs. O'c.: "What sort of radio-station", she reasoned to
herself, "would play Irish songs, deafeningly, in the middle of the
night? Songs, just songs, without introduction or comment? And only
songs that I know. What radio station would play my songs, and nothing
else?" At this point she asked herself: "Is the radio in my head?"
From Wikipedia article linked above.
Simple partial seizures (SPS) involve small areas of the temporal lobe
such as the amygdala and hippocampus. The term "simple" means that the
level of consciousness of the patient is not altered during the
seizure. In temporal lobe epilepsy, a simple partial seizure usually
causes abnormal sensations only.
These may be:
- amnestic sensations such as déjà vu (a feeling of
familiarity), jamais vu (a feeling of unfamiliarity) amnesia; or a
single memory or set of memories.
- A sudden sense of unprovoked fear and anxiety
- Nausea Auditory, visual, olfactory, gustatory, or tactile
- Visual distortions such as macropsia and micropsia
- Dissociation or derealisation
- Synesthesia (stimulation of one sense experienced in a second sense) may transpire.
- Dysphoric or euphoric feelings, fear, anger, and other emotions may also > occur. Often, the patient cannot describe the sensations.
-  Olfactory hallucinations often seem indescribable to patients
beyond "pleasant" or "unpleasant".
There are accounts of rapture: amazingly overwhelming experiences of the divine. Accounts of things so terrifying there are not words for them.
Also, seizures can definitely cause amnesia, either for events happening during the seizure, or before or just after. Memory loss with a seizure can be inconsistent, and its occurence might depend on other things going on or the severity of the seizure.
I hope this is for a TV show. I picture everything moving slightly slower when the hallucination shows up. A clip of music - maybe Led Zeppelin plays over and over, a little too slow. You can see the character smells something horrible. Then the visual hallucination comes into view.