Yes, you can! You could imagine a universe where physics is based on Galilean relativity instead of special relativity. I'll skip over the mathematical details (unless you're interested), but basically Galilean relativity describes a spacetime that has one universal axis of time, which never mixes with the dimensions of space as it would in special relativity.
In Galilean relativity, there would not be any invariant speed that is the same for all observers. Specifically, the speed of light (by which I mean the actual speed that light travels, not the constant $c$) would be different depending on the conditions under which it was measured.
There are some different ways you could make this work. Probably the simplest is filling your universe with some medium that light travels through, and it always travels at a fixed speed relative to the medium. (This was a real scientific theory in the late 1800s; the medium was called the luminiferous aether.) Light in this model would behave a lot like sound, for which air is the medium. Objects like planets and solar systems could affect the motion of the medium, just like mountains and buildings affect wind, and you might even have aether-weather phenomena if the dynamics of the medium are complicated enough.
An alternative would be that light simply moves at a certain speed relative to whatever emitted it. In this model light would behave like a projectile, e.g. a bullet from a gun, so it is a natural match to a particle model of light (though you could do it with a wave-particle dual model of light too). The speed at which light is emitted might depend on the energy of the photons, and/or on the mechanism by which they are produced. It would probably be quite natural to have higher-energy photons, corresponding to higher frequencies of light, travel at higher speeds. If you do this, you would have an interesting effect where you'd see a faraway object at different times in different frequencies - for example, a brief flash of white light would be seen from afar first as purple, then transitioning through blue, green, yellow, orange, and red.
Galilean relativity would not require forces to propagate at infinite speed, but they could. In other words, you would be able to have two kinds of forces in this universe. One kind is the ones which are carried by fields, the way forces work in our universe. The object that exerts the force actually triggers some kind of propagating disturbance in the field, and then the object that feels the force reacts to the disturbance. These disturbances would propagate at some speed characteristic to the field - but note that all the discussion from earlier paragraphs still applies, concerning how the speed would change depending on the conditions under which it was observed. Depending on how you want it to work, you can achieve a wide variety of effects, including the "temporal chromatic aberration" from the last paragraph, except now with forces: if gravity worked this way, then a sudden change in a mass distribution (assuming such a thing were possible) would have a prolonged effect because gravitational waves at different frequencies would arrive at different times.
The other kind of force is that which instantaneously affects the entire universe. Actually you could think of this as a subset of the previous kind of force, where the natural speed of the force is infinite. The existence of this kind of force would allow for instantaneous communication between widely separated locations, although if the distances involved are large enough, the people in your universe might have technological problems detecting a signal because it would simply be too weak. The phenomenon where a signal weakens in proportion to $1/r^2$ would still apply in Galilean relativity, or at least it could, though I think it would be possible to have a force that does not weaken over distance, if it's not carried by a field.
You could still have a big bang, which would represent a definite beginning of time. Everything in the universe would start out moving away from everything else, though you wouldn't have a good answer to the question of why it started doing that in the first place. Anyway, afterwards, the evolution of the universe could proceed much the same way it did in the real world; you'd still get galaxies, stars, and planets, for example, though not until some hundreds of millions (or billions) of years after the beginning. Special and general relativity actually aren't that important for most of the universe's history.
Your universe could have a finite size, in the sense that there is only a finite region of space filled with stuff, although with a big bang-like event at the beginning, that region would change size over time. You could set it up so that characters in your universe could travel beyond the edge of this region (assuming they have the technological means to get to the edge), into an endless void; or the edge could be a hard wall that they run into; or anything that hits the edge could just vanish. The latter two options are a bit tougher to reconcile with existing physics though. Depending on the details of how light and forces travel, and the size of the universe, characters in your universe may or may not realize that the universe is finite. Naturally the closer they are to the edge, the easier it is for them to tell.
If your universe is infinite, on the other hand, depending on how light behaves, it might be subject to Olbers' paradox, which basically points out that in an infinite universe filled with stars, the entire night sky would be lit up because there is no direction you would look that would not run into a star eventually. However, if you have a beginning to your universe and light travels at finite speed, this argument wouldn't hold. So you could still have a dark night sky.
If gravity (or some other attractive force) travels at infinite speed in your universe, the expansion will slow down over time, and will eventually stop and reverse. So your universe is doomed to collapse in on itself at some point in the possibly distant future. If gravity does not travel at infinite speed, then it may or may not collapse, depending on the details.
Lastly, none of this invalidates quantum mechanics. You wouldn't have quantum field theory, but there could still be nonrelativistic quantum effects and so a lot of quantum phenomena we are familiar with would still be possible.