The direct answer to your question (in my opinion) is summed up in the famous quote from Arthur C Clarke - Any sufficiently advanced technology is indistinguishable from magic.
What it comes down to in the end is that the difference between science fiction and fantasy is the nature of the audience; science fiction fans actually want you to explain what's happening to at least some degree, but they'll settle for plausibility where their suspension of disbelief is required.
Fantasy readers on the other hand like you to push the boundaries of the credible, and tell a good rollicking tale that doesn't let science get in the way of a good plot device.
In other words; the difference is largely in the degree of how much of your 'magic' system you're willing to explain.
In science fiction, we talk about FTL drives that are warp drives, the existence of hyperdimensional space (hyperspace) and teleporters.
In fantasy, we talk about portals, pixie dust and flue powder.
Both achieve the same effect in terms of story and plot, but only one tries to explain why it exists in the universe being described without resorting to hand-waving.
There is another quote from a hard science fiction writer (at least, hard classical physics writher) named Larry Niven, who once said in the foreword of one of his books (I forget which; I read it 40 years ago now) that time travel should always be considered fantasy by definition, because science does not allow it.
While from a hard scifi perspective this is true and conforms to the description I give above, there have been many books written in scifi form that deal with time travel, and attempt to give plausible explanations for it scientifically. Again I would say that the difference is the willingness of the author to provide that plausibility in a scientific context, and the readiness of the reader to accept it.
As such, I actually see this as a spectrum ranging from hard science fiction through to medieval swords and magic. Every science fiction story will contain at least some fantasy, and vice versa. Where you want to place your story in this specific continuum is really up to you.