I will start by saying the Star Trek idea of parceling up space is unrealistic. The short answer is that there is not a very good reason for closing off empty space. In reality, "open space" probably would work more like "international waters," in that a certain distance from someone's home planet is considered territorial, but beyond that it's essentially the law of the high seas (or in this case, space?). Space is so vast this would likely even happen on a planetary level, rather than a star system level (i.e. space between Earth and Mars is international space).
However, that wasn't your question so I will work within its bounds.
The main reason you would close off space is to control lines of approach. This works especially well in the linear FTL world of Star Trek and Star Wars. You can't "cross my space" because it allows you to get close to something of value, like my home world. In this case, space acts more like a buffer. This way, it makes it much more difficult for someone to explain why they're in "your space," as they would have no reason to be there unless they were traveling towards your home world or a world you've claimed. However, even this doesn't quite fit, as countries aren't allowed to close off large portions of international waters "just because" it provides a straight line to their shores, and in some cases there are bodies of water recognized as "important" where territorial restrictions don't apply. Generally this is in cases where the body of water is the only access to an area (such as the straights of Gibraltar, the Bosphorus, Straights of Hormuz, etc...).
Given this, it becomes difficult to justify parceling off space, so the next remaining possibility is one of resources.
Empty space isn't empty at all. Even interstellar space is likely to contain asteroid and planet-sized rogue bodies. You could, in this case, say "all the stuff in this volume of space is mine, whether I know about it or not." In this case, it becomes something akin to a country's Exclusive Economic Zone, which is much larger than territorial waters (200 miles vs 12 miles). But even in the case of an EEZ, international law permits any and all innocent passage through its waters, including that of military ships, so once again it becomes impossible to justify whole blocks of space being off-limits to ships.
With FTL travel though, it becomes possible for a ship to arrive at a destination before anyone has the opportunity to detect it. In Star Trek, this isn't the case because subspace sensors can detect FTL ships, but in reality it would be very hard to detect a ship at warp. With this, you can again make the buffer case, and say that no ships should be allowed to cross into my predefined box because they can get to my planets before I have a chance to respond. This is the one difference between seagoing navies and spacegoing FTL navies. If I send a flotilla of ships to your country through international waters, you have days, weeks even to act and prepare yourself. If I send an FTL ship to your planet, it could theoretically arrive there in minutes, making it more akin to an aircraft problem than a sea vessel problem.
In this case, rather than being like shorelines and international waters, your space territory becomes one of airspace, and airspaces must be relatively well-controlled. This allows you to make the case that any starship entering your space territory must be under the control of the local space traffic authorities, and a warship refusing to follow these protocols is one that is acting in a dangerous manner, which could be considered an act of war.
So to answer your question: The best way for this to work is to treat your space territory as an airspace, requiring that FTL ships be in constant contact with control authorities to ensure safe passage for all ships. After all, two ships colliding at FTL speeds would release an immense amount of energy that would likely create a black hole (Ek = 0.5mv2 after all, and velocities greater than the speed of light imply incredible amounts of energy involved). Any ships refusing to operate under local controls are thus a navigation hazard and should be treated as a hostile actor.
Your space-nation would likely need to organize these FTL airspaces into controllable lanes, ensuring proper spacing between ships in an FTL lane and safe distances between overlapping lanes traveling in different directions or at different speeds. In Earth airspace, there is a concept of uncontrolled airspaces, but in your FTL lanes, any FTL speed is likely to require control, so uncontrolled airspaces might be relegated to sublight speeds. Earth airspaces require 1000 feet of separation in altitude, and usually 30 miles of separation for aircraft at the same altitude. You'd probably want to increase your boundaries based on maximum FTL speeds allowed in a given lane, and have them set so that a human-equivalent-alien can react in time to avoid a collision at the speed given. Let's set this at no more than 30 seconds. Traveling Warp 2 in the Star Trek universe is 10x the speed of light, so at this speed your separation distance would have to be about 90 MILLION kilometers, or about half the distance between the Earth and the Sun. 0.5 AU sounds big, and it is, but in interstellar space you could have thousands of FTL lanes easily. You would probably want a central set of high-speed corridors with no-go zone buffers in between, then you can surround that with slower speed corridors, etc...
Given the dangers involved, there would likely be FTL no-go zones near planets to prevent accidents, and given that space gets less empty the closer you are to a star, it's likely the FTL/sublight boundary would be at the far edge of a solar system. You sometimes see this plot-hole come up in Star Trek, where actors will say things like "you want to go to warp inside a solar system?!" or otherwise imply that going to warp near large bodies is unsafe and undesirable.
Truly deep, interstellar space would likely be nothing but FTL lanes, as there's almost no reason someone would want to be going at sublight speeds there, but you would probably want "step down" zones the closer you got to stars, allowing ships to slow to lower warp levels and finally to sublight speeds as they get close to planets.
This is a little off in the woods from your original question, but all of this regulation takes up a lot of space, time, and money. It goes to reason then that each of the nations undertaking the maintenance and control of the FTL lanes in their space has a vested reason to claim sovereignty over that space, and that any ship operating outside of the rules established by the local sovereign is a threat to the local order, thus you have a reason for the boundary lines specifying ownership and governance.