There is a sapient jellyfish species resembling the box jellyfish found to be lurking in the ocean today. A team of researchers took a sub and dived into the abyss to look for it. What kind of non-invasive tests (in-situ) can be performed on this jellyfish to see if it is sapient? Don't expect it to talk back or open a jar! However incomplete fossil records show that they could evolve some sort of neuron cells similar to those found in humans and current theory suggests they could be extremely intelligent.
While azerafati's test is the one they should start with, it is likely to fail or be inconclusive. Manipulating objects or showing curiosity in new things may not manifest in the ways we expect.
Your scientists must include experts in animal behavior, perhaps those who also have training in anthropology. Ideally they would set up a research station to observe the jellyfish at length. That may not be possible, due to the conditions, so multiple video cameras are a decent second choice.
Immersion into the jellyfish society is even more difficult to achieve logistically but is the very best way to discover their abilities. Think Jane Goodall and her work with wild chimpanzees.
The method that most scientists would use is the easiest for humans and the one that will yield the worst results (in many ways): capture a few jellyfish and bring them to an aquatic center where humans can easily interact with them. I don't recommend this for real-life wild animals and I certainly wouldn't recommend it for your fictional (or maybe not so fictional, how would we know?) jellyfish.
Emotionally traumatizing beings by ripping them away from their families and communities and putting them in a prison cell filled with unfamiliar things and different environmental conditions is pretty much the worst way to encourage their trust.
Observation in the communities the beings live in is your only hope. For animals we call it "in the wild." This takes weeks to get a sense of where to go next, months before you can start pointing to conclusions. We're talking about a species very different from our own. Chimpanzees are easier to study because physically they're a lot like us and there is a lot of overlap with their emotional states and expression, family relationships, and societal organization too. But jellyfish? This could take a team of experts months or years just to get a handle on it.
Yes, there are tests one might devise that, if the creatures pass them, could be evidence (probably not proof, not yet) that they are sapient. The problem is, not passing the tests means little. Even tests we use routinely on mammals like self-awareness of one's reflection, theory of mind (the classic test is hiding food in front of just one being and then seeing if that being knows if a naive being brought in later knows where the food is), and so forth may just not be relevant for jellyfish.
The tests that will work are to see how members of the community treat each other. Theory of mind will come into it, we just don't yet know how. What do familiar relationship look like? (they may not have anything to do with genetic relationships). What power hierarchies are there, if any? How does the community make group decisions? What happens if an individual makes a decision counter to the group? How does the community respond to danger? And so on.
The definition of sapience is often given as follows:
The ability to apply knowledge, experience, understanding, common sense and/or insight to a situation or circumstance.
As such, one would have to understand what consists of normal behavior for said species, successfully introduce one, and only one, variable, and then test for the ability to apply the learned behavior to future situations. Ideally also checking to see if the learned behavior is passed onto others or not.
Given that we are dealing with nonhuman species, understanding what consists of that definition from the creature's point of view is an additional wrinkle. After all, it may be perfect common sense from the creature's point of view to do or think something completely illogical from a human's point of view. Not to mention individual variance.
It would most likely take repeated tests across multiple population groups to collect enough data to indicate the potential likelihood of sapience. Ultimately, viable communication would need to be established as final proof, should the creature or creatures be willing.
They put an Object on the path of the fish which is totally amazing and extraordinary for the fish and monitor it's reaction from a cover or use some hidden cameras.
- If sapient, the fish would surely be curious and show some abnormal behavior toward the object indicating it's intelligence.
You've mentioned that the neuron cells are similar to those found in humans. If so, then it should require a significant size collection of them to achieve sentience. Just look for the jelly fish with the human brain sized growth.