I don't think the characters were acting out of character. Amplified an aspect of a character? Sure, I can agree with that.
Take Sisko: we know he does have an obsessive streak in him. (Which most captains have had, honestly.) That trait is amplified here, but it's not out of character.
"Amplified"... perhaps. I might also include the descriptive word "surreal", if not phrase "fever dream" or "LSD trip" to describe the episode and characterizations. Never understood what's so great about this story, much less the characters in it. Is the holosuite that limited?
And the Vulcans? Vulcans have always had that arrogant, superiority streak about them. Solok just represents the Vulcan trait a bit more amplified.
By intent or by misaligned scripting that forgets they are emotionless? I still don't buy into Solok, who isn't as much amplified as he is caricaturized: https://tvtropes.org/pmwiki/pmwiki.php/Main/Flanderization (which pretty much says that a tv character originally given depth has superficial traits amplified and any depth removed. Characters become hollow pastiches of their original selves, sometimes with parody built-in as well. Not always deliberately so, but it still can happen.)