For DOS and ScummVM games, simply hit Alt+Enter to switch to and from windowed mode; this works for all games. If you want a game to launch in a window by default rather than fullscreen you will need to edit the game's .conf file (in the case of DOSBox) or change the settings (in the case of ScummVM).
For native Windows games, however, it varies. Some respond to Alt+Enter, while others only allow you to change in and out of fullscreen mode by choosing the appropriate option in the settings, and many don't support windowed play at all.
Some people have had success using
to force a windowed mode on games that don't have one. The original is in Japanese, although [url=http://www.fileplanet.com/198137/190000/fileinfo/D3DWindower-English-version]an English version also exists. I haven't used this myself so I don't know how well it works or what games it works with.