###There is only so much you can do if you want to extract many specific, "stereotypical" features from real cultures as set pieces in your work. Using less "real" material and knowing more about the source of your inspiration will be helpful.

That's not the most charitable way of phrasing what you're looking to do, but it's correct in the essentials. There are whole cultures, thousands of years old and rich in detail and nuance beyond routine expression, which have produced the features that have captured your interest.

Extracting a handful of conspicuous features from those cultures without any context and without providing any cultural representation beyond those conspicuous features, just because you think they look cool or suit your setting, generally is what people complain about in situations like this. It's definitely worse if the chosen features denigrate the group, but many object to their cultures being "cheapened" just to suit aesthetics in a fictional setting by someone not at all engaged with their actual culture and history.

So my main answer to the main question is: don't wholesale-copy much from existing cultures, and know enough details to be "inspired" well. Inspired by is not the same as copied from. There are many things that aren't especially culture-specific-- mud houses have been used in many parts of the world at many points in history by totally disparate cultural groups. Mud houses, alone are probably fine.

Mud houses that look exactly like those used in a certain culture... that's a bit different. Architectural styles and practices are varied, and can be very culture-specific. Having mud houses is one thing, but having mud houses that look just like Puebloan mud houses is a bit closer to copying rather than being inspired. This becomes more severe as you add more features which are specific to a given culture.

And just as importantly, think about the features that you aren't carrying over from your inspiration. A big risk is not noticing distinctions meaningful to members of the source culture: "these groups are all the same" is very offensive to groups that view themselves as being very different, especially if you casually blend details together in a way which more familiarity with the cultures would prevent.

If you're using the architecture, clothing, and art styles from a culture but nothing else, it's fairly easy for someone to say that you are caricaturing the culture without bothering to know (or at least, express) anything meaningful about it. This is especially the case if any of those elements had broader cultural significance (like clothing styles that weren't just popular for some reason, but had deeper religious or cultural meaning)-- it may not be possible to respectfully imitate one without the other.

As an example, I have some Italian heritage. I don't especially care about Italian representation in media in the modern day (we don't see the same kinds of social attitudes that were common in the U.S. around the turn of the 20th century), but a cartoon of a mustachioed chef eating spaghetti and meatballs while speaking in a funny accent is a clearly "Italian" caricature, but is a poor representation of the culture that produced the Roman Empire, lead the Renaissance, and countless other items.

As for the "know enough details" piece, a person can't be much inspired by a culture they know little about. Doing some real research on a culture (reading published papers, well-regarded history books, etc.) can give you enough understanding that you aren't just copying superficial details. A few hours of casual googling simply isn't enough.

Broader notes:

The goal of not offending a "culture" is good, but fuzzy. Cultures often don't have representatives empowered to make blanket judgements on this sort of topic, as noted in other answers. And even if they do, that's no defense against individuals feeling offended and complaining. There is no bright line for the amount of respectfulness you can display (however that's defined) which guarantees that no one will feel offended, or that relieves you of any possible responsibility if someone takes offense.

There is no obvious standard for how to do this sort of thing "right", aside from not doing it at all. Talking to people who are a part of the source culture, especially if they are cultural experts (the exact definition of which will vary between groups) to get guidance on how appropriate your ideas seem to them can be helpful, but will never be definitive.

The shallower your inspiration and translation of cultural elements are, the more likely you are to give offense. The more knowledgeable, cautious, and substantive you are in representing cultural elements, the less likely. And if you can avoid copying enough superficial elements to obviously point to a real culture, you'll be on firmer ground still.