This is a cheater method.
Tessellate. Cover the map with hexagons or triangles or squares of roughly equal size. Call this the "grid" and the shapes "cells".
Ensure the size of the cells is small compared to the thing you are measuring. Like, 80%+ or more of the cells that touch your shape are fully within the shape, and 20% or less are partly within the shape.
Count. Treat any cell that is partly in or out of the shape you want to measure as being a half cell.
So long as your tessellation is aligned random relative to the geography you are measuring, this will get really close. You can increase precision by subdividing (all of hexagons/triangles/squares can be subdivided into smaller regions of sufficiently similar shape).
The projection of your geography onto the regular grid can warp the size of the cells, which can cause errors or add to the complexity of the math.
You can look at the "edge" (the ones partly covered) to determine if it is likely to over or under estimate the shape. A smattering of dots in a void will be over estimated, and a smattering of voids in a region will under estimate the area of the region. You can bound the amount of error, however, buy the total area of the "edge" region.
I own some transparent grid and hex sheets. You can place this above your map and use that to tessellate.
You can even just sample on a regular pattern. Unless your pattern is somehow correlated with your shape and the pattern is fine enough, counting how many spots on a regular pattern are "inside" will get you a good measure of the area.
Pay attention to the size of your "dots" you are measuring intersection with; the larger they are, the more it will overestimate the "boundary"s contribution to the area of the shape.
A less mechanical and more mathematical method is to start off by drawing two polygons. One fully within the region you want to measure, and one fully without. The area between is your "complex coastline".
The trick is the area of what you want to measure is between those two region's areas. As you add more detail the gap shrinks. Or you can look at the gap, and fudge a percentage of it being within the region.
A very mathematical method is to use Green's Theorem. The area can be viewed as a double-integral; this can be converted into a path integral over the boundary of the region.
Using the above "inside" and "outside" curves or polygons, you just integrate over the paths and get the upper and lower bounds of area.
But this is probably not what you want to do.
An amusing way to do it is to use Monte Carlo methods. Pick a random spot and see if it is inside or outside.
After a few 1000s of samples you'll have a pretty good estimate of the area.
This technique is actually useful when describing the region you want to measure is difficult.
Take a picture. Flood fill the region you want to measure.
Take a picture. Flood fill the region you want to measure as white, and everything else as black.
Blur the entire image uniformly. If the resulting pixel is 65 white, the shape was 65/255 or 24.5% of the original image.
I mean, how accurate do you need to get?
Look for big bulky "filled" parts of your shape. Toss some rectangles in and find out how big they are.
Then measure the fuzzy perimeter. How thick is it? (Ie, from the point where you already measured, to the definitely outside) How dense is it? Is it like half-ish inside and about 50 km thick and about 1000 km long? Well, that is about 25,000 km^2 give or take. Add that to your 100,000 km^2 interior and get a decent estimate.
Or even worse, just measure total width and height of your shape. That is your upper bound. Then guess what fraction of that area looks to be inside. Multiply. If your shape is complex? Do it in chunks that are easier to estimate.
So lets say 3 pieces to estimate this. One is the 45 to 60 degree band, the other the NWT/Yukon 60 degree to 70 degree band, and the last if the northern islands.
The southern band is from 60 to 135 degrees, or 75 degrees, or about 20% of the Earth. The northern band is from 90 to 140, about 50 degrees or 14% of the Earth. The northern islands look to be about 1/3 of the area of the northern band total.
Both "solid" bands look to be about 80% land ish.
At 52 degrees latitude on the Earth, 75 degrees is about 5000 km.
At 65 degrees latitude on the Earth, 50 degrees is about 2500 km.
Each degree latitude is about 110 km, so 15 degrees is 1650 km, and 10 degrees is 1100 km.
So my estimated square km using this method is (5000 * 1650 + 2500 * 1100 * 4/3)*.8 = 9,533,333 km^2
The actual area of Canada is 9.985 million km^2.
Considering how fudgey most of my guesses where, that is really close - closer than I expected, and way closer than you'll need for any practical world building reason.
Map of Canada Image source: Wikipedia