EDIT: (Important) M. A. Golding commented that the regalecid method of locomotion would be unfit for a constricting predator, so I have completely redone the taxonomy of the sea serpent. They are now eels, closely related to moray eels. The sea serpent taxon has been upgraded to family Serpentimaridae, as opposed to the old subfamily Serpentimaria. I don't have time to draw up a dendogram right now, but they're in the suborder Muraenoidei, in a clade with moray eels.
I think that the current lack of answers is because I have not alluded much information as to the physiology of these animals. Here is a rough sketch I made in Paint (Originally a preparatory draft for a real illustration of sea serpent anatomy):
Note: the reproductive and excretory systems are not shown here, but do not affect their potential size anyway so it shouldn't matter.
So, if the heart needs enlarging, or I should have a bundle of ganglia towards the back as an extension of the CNS, or any other amendment, please let me know. Hopefully, I'll get some answers now that I have clarified a bit more.
This is currently the sixth question about my ongoing worldbuilding project on viable fantasy creatures. I'm too lazy to link all of them here, but the last two were: What kind of animals are my trolls? and How can I explain the evolution of my giants?. You can find links for the rest at the latter.
Now, from giants and trolls we progress to the largest creature in my project, the sea serpent (Subfamily Serpentimaridae). These beasts are not serpents but fish, in the suborder Muraenoidei.
I want these serpents to be massive, at least the length of a blue whale - and upwards from there. Seriously, if the answer is "600 metres", then I'll take that. But, I rather doubt that a fish could grow to a length of 600 metres (Feel free to prove me wrong), so I'll have to find out what's rational and what's not.
Sea serpents are constricting predators, like boas, and use their musculature to crush whales - and, on occasion - ships. They also have pharyngeal jaws, like their close relatives the muraenids, which allow them to swallow a rowboat whole.
I haven't figured out what oceanic zone they'll inhabit, but any is fine for me really. Deeper zones would be convenient as an explanation for why they've remained undiscovered, except for reports by sailors throughout history. They need to be large enough to be able to create waves with lashes of their tail, as well.
To summarise: How big could Muraenoideian sea serpents be?