There are perhaps 2 methods that can achieve what you are looking for depending on the scale of your jagged edges.
Sedimentary layer shifting, collapse and erosion
It is conceivable that a sedimentary area has over time become eroded and the layer angle shifted to produce straight edges over dozens of metres. This is evident at Mistaken Point, Canada. As the strata layers have different properties and formed over different eras of time, the layers are straight. As time progresses, they shift and erode, exposing the layers to water forming straight edges.
Earthquake faults
Like at the Pupatea Fault, New Zealand, where a magnitude 7.8 quake caused a seismic shift and exposed along the shore a roughly km long straight edge. This is rare and relies on a geologically consistent layer to form, but nonetheless creates a permanent face in the rock which could be, over time, a jagged straight shoreline.