Criteria
I've narrowed what your looking for down to a solution which meets a few criteria:
1) You are looking for a weapon which is based upon a physical mechanism (non-magical) that can is roughly analogous two modern day equivalents: the Main Battle Rifle (MBR) and Heavy Artillery.
1a) For the MBR I'll use the following restrictions: a small arms weapon capable of being wielded unaided by the average human, is capable of continuous fire, is re-loadable in a modular fashion (if applicable) and is capable of inflicting lethal damage to the average human up to ranges of at least 300 meters.
1b) For Heavy Artillery you mention the addition analogy of navel artillery, capable of being mounted on spacecraft and inflicting severe damage to the constructs on which the heavy artillery are mounted (presumably other spaceships, but not planets). I'll assume that the artillery are not hard guided (no missiles), while some sort of "soft" guidance is allowed in analogy with modern day artillery. For real world analogies, I'm presuming your thinking of 12 - 16 in. guns.
2) The physical mechanism by which these weapons operate is "mostly realistic". Since "mostly realistic" is very open to interpretation, I'll restrict my solutions to only those which do not violate known symmetry laws in physics (most of the Standard Model). Fictional physics I'm assuming can be used, but only as long as it does not violate already established laws.
3) The Physical mechanism by which the weaponry operates is not a common trophe of science fiction. Thus, I will rule out the following: Laser beams, Plasma projectiles or bolts, Gauss guns, "Ray" guns, as well as real life projectiles: metal and polymer bullets, depleted uranium and tungsten bullets and kinetic projectiles. The caveat here is some elements of the above may be used, but only if they are coupled with a mechanism\material which is not typically associated with the weapon system.
4) The weaponry should incorporate unique and unusual physics and/or materials in its' design. Thus mostly ruled out are chemical firing mechanisms and electric firing mechanisms.
Possible Avenues:
With these restrictions in mind I'll present some possible solutions. Unfortunately all of these criteria eliminate a lot of possibilities. I'll start with high levels of handwavium and proceed downwards.
A) A fifth force: You can do all sorts of fun stuff with a fifth force without explicitly breaking known physics. As long as the fifth force involves improbable scenarios and as long as that scenario doesn't explicitly violate known laws it can appear plausible:
A-1) For example, does this fifth force interact with matter? Perhaps as an interaction between masses but unlike gravity the interaction is strictly repulsive. How is this force triggered, perhaps by running a current through an exotic material. Now you can have a gun whose power is limited by the structural integrity of your material and electric current. Combine this with high-temperature super-conductors and you have one hell of a weapon. Of course the constants for such a field to work have to be just right and you still need to respect conservation of energy, but hey its a fifth force and can be tailored to how you like it.
A-2) Of course invoking a fifth force will come across as very contrived to anyone knowledgeable about physics, but it can provide the playground to explore really cool phenomena which can be used for all sorts of other things. Working through the physics of new fields and particles interacting can given rise to constructs well beyond weaponry. This approach is also very conducive to effects and weaponry which is not already (over?) explored in science fiction.
B) Less contrived than a fifth force would involve materials and phenomena, but interesting enough be be unique. Superconductivity is an obvious choice which will allow all sorts of electromagnetic weaponry, but this is quite common in science fiction. Other areas along this avenue are explosive propellants which are not commonly used in Sci-Fi and which have no real-life counterpart. For example, inducing nuclear emissions in some exotic material to produce a powerful explosive reaction which could result in more energetic projectiles. While no real cases of induced nuclear emission have been found suitable for this purpose, there have been some proposed cases which might serve for insight into what this phenomena might look like (see Hafnium controversy).
C) At the bottom of the rung are possibilities found IRL, unfortunately most of these are already found somewhere in fiction and science fiction. You can have caseless ammunition, Self-guiding (smart) munitions and miniature atomic weaponry. Each of these can be tweaked to be unique but will not appear very exotic.
Where to go:
I would imagine that it would be easier to work backwards than forwards. By this I mean that it would be easier to flesh out a weapon system and some characteristics that this weapon system has. For instance, what does the rifle look like? Does it have any characteristics which stand out? Perhaps it needs a very cylindrical metal chamber, a long barrel and fires what appears to be bright bolts of light. Perhaps it can fire rapidly, but has a serious over-heating problem and is loaded with cartridges containing bullet shaped rods, but no propellant or casing. The more descriptive the more you can contrive some physics by which this sort of weapon could be made.
Working forwards on the other hand is not easy if you want to be consistent with the physics. For example, if you want to incorporate an IRL exotic states of matter, degenerate electron matter for instance, this really narrows down what you can do since there already exist very good theories of electron degeneracy pressure and when it is applicable which is essentially only in the conduction band of super cooled metals, or in super dense, high gravity star systems such as white dwarfs. Unfortunately real life physics is very restrictive which I believe explains the tendency of all science fiction, especially the sort which tries very hard to not break existing physics laws, to sort of converge on the same ideas and concepts.