For the majority of time (hundreds of thousands of years), it orbits the Sun-star, but occasionally, it gets "snatched up" by the neutron star. During these periods, the planet's orbit is erratic and unpredictable - sometimes it's quite far from the star and everything freezes over; at other times it gets so close, the gravitational forces and radiation wreak havoc down on the surface.
Well, that's your first problem. If the system is chaotic enough to occasionally let the planet switch primaries, you are not going to have nice, well-behaved habitable orbits when it switches back to the regular Sun-like star.
Could intelligent life have evolved during one of these "stable" periods (when orbiting the main sequence star + "erratic" orbits around the neutron star that happen to be livable), only to be later wiped out when the planet got too close to the neutron star?
In merely a few hundred thousand years? No.
What precisely to change depends on what your purpose for having this cataclysm is. In any case, though, I would suggest simply letting your planet always orbit one star or the other, stably. It doesn't really matter which--or, you could have the stars in a very tight binary orbit, with the planet orbiting both of them relatively far out.
The important bit is that the neutron star be close enough to the Sun-like star to slowly accumulate a hydrogen atmosphere by accretion from the solar wind. This will result in periodic novas when the hydrogen atmosphere becomes thick enough to flash-fuse, which should do nicely for sterilizing the surfaces of any planets in the system.
How frequently that occurs will be a function of the distance between the two stars. If you have them in a close binary orbit with the planet orbiting the pair, novas will occur fairly frequently. The only complex surface life that could persist on that planet would be life that can persist itself in highly radiation and temperature resistant spores, probably underwater or underground, which would resprout after the disaster (cf. the alternating Hot Life and Cold Life from Hal Clement's Cycle of Fire).
The farther apart the stars are, the more infrequent novas will be. If the planet permanently orbits the neutron star, that puts a cap on how much the stars can be separated, as the planet+neutron star system must remain in the habitable zone of the Sun-like star. I do not know if that will give you enough separation to make the disasters sufficiently separated for a "normal" civilization to arise or not. If the cyclic nature of the disasters is not actually important, just that at least one happens after the rise of civilization, then I recommend simply letting the planet orbit the Sun-like star, with the neutron star being a distant companion, which accretes new material only very slowly. This gives you the freedom to put the planet exactly where you need it to be habitable over the long term, while not having to specify the exact location of the neutron star; figuring out precisely how far out it needs to be to explode at just the right time in your planet's history can be left as an exercise for the reader.