There have been non-sport events at the Olympics before. In 1900 and 1904 they had motorsports events. Between 1912 and 1928 they had architecture, sculpture, painting, music and literature competitions with medals.
Though personally, I think the modern Olympics has now become an event for physical skill games. Whether that's outright athletic skill (running, jumping, lifting, gymnastic) or a highly physical skill (archery, shooting and so on) I think that's fine. Personally I'd like to see an end to the horse events, I don't think they belong in either category.
Chess and Poker are both great games and I've spent many hours playing both, but they already have their own high profile worldwide competitions, I don't think they need to be at the Olympics.
The other problem with Poker, too, is that it's a betting game. I suppose you could play a tournament for chips, but encouraging kids to go to casinos is probably not the look the IOC would want.