You can't measure it directly at a distance with current technology.
Tech however at a given time is fairly well constrained. E.g. If you are building battleships, then knowing the dimensions of the battleship give you a pretty good idea of the mass. Space allows extreme magnification of images, so getting an approximate mass at a distance of several AU's is reasonable.
Another factor is the power/mass ratio. This will affect acceleration, and possibly also weaponry. A destroyer has a much higher power/weight ratio than a battleship. Thinner armour, smaller guns. Destroyers run rings around fleets and convoys to intercept subs.
In space you also look at the waste heat. The volume of a ship goes up with the cube of linear dimensions, while the surface area goes up only as the square. A big ship has a bigger problem getting rid of waste heat. It's radiators will run at higher temps.
You would also monitor communications. Difficult. Comm channels are likely to be tightly beamed.
All of these are subject to a certain amount of spoofing:
- A small ship can inflate a large metallic balloon and appear to be much larger.
- A small ship, by manouvering at well below max acceleration may be thought to be larger.
- A small ship, by directing it's cooling heat at you can appear to be disposing of more heat -- signature of a larger ship.
- A large ship, by directing waste heat away from you appears smaller
- A large ship, if shaped like a disk or a brick can maintain an orientation edge on or end on and appear smaller.
- A large ship, with the outline of a small ship in white and the rest of the ship black, may be classed as a small ship at long range. Of course black doesn't help the heat disposal problem.
Detectors: Visual gives you the most information about the detail. Far infrared gives you more information about the skin temperature, and hence the power use. The outline technique above wouldn't fool an IR scan.
Having scouts spread throughout the space of interest makes deception harder. A small detector, disguised or part of an asteroid would be nearly impossible to spot. If space also had several radar illuminator ships, the passive scouts would get radar distances -- You get both the reflection of the radar off the enemy, as well as the time signal from the radar ship. That and geometry give you a solution of the distance.
Note that as soon as you change technology, all this goes out the window. E.g. at present one of the ways we would measure is the amount of rocket exhaust and the resulting acceleration. Replace that with ion drives, and we may not even see the exhaust.
Analogues:
Submarines are identified by their sound signatures. And some of the same forms of deception are used, pretending to be something they are not. (Usually however a sub's goal is to be undetected.
Radar can determine the difference between a Cessna and a Boeing by strength of reflection, as well as the way it manoevers. Adding a corner reflector increases the radar return substantially making the Cessna look bigger. Stealth coatings on fighters make the radar return vanishingly small.
In the days of sail, a ship of the line could be spotted further away -- taller masts. At closer range, you could look at the rigging and get a good idea of the size of the ship. Counting gun decks was another way.
The uncertainties add to the fun in your universe. Allows for possible ruses des guerre