meaning - What are the difference among "should open" and "should be opened" and "should be opening"?

The big difference here is active and passive voice. The original sentence is in active voice. In your suggested options, I think you're trying to use passive voice, but you need to change the order of the words.

When you say “it should be opened,” it is the object of the opening. But the object should be the window. So you want to say “One new popup window should be opened.” Note that in passive voice, you don't say who or what is doing the opening.

The same goes for the message: “the following message should be displayed.” Again, you don't say that the window is doing the displaying.

Putting it together, you could say:

One new popup window should be opened and the following message should be displayed to the user: “Thanks for your feedback.”

Note that you could use active voice for one part and passive for the other, though that is probably more awkward.

Generally, the choice of voice is determined by the style guide, or consistency with the rest of the document. So check whether someone has made a guide for these documents and if not, just pick either active or passive voice and use it throughout.