All above points are correct, but one is missing.
Your main reason for having own dedicated server is to run your server for your multiplayer online game. So... you have to check if there is version for Linux of your application. If not, if there is only your .exe version for Windows, you have to use Windows on your server instead of Linux (or Linux plus Wine, but is this possible and/or reliable - inportb will tell you).
And I suggest you - just like inportb - if you just starting with this, then before you spend some more money for this try to set up VPS on your existing comp (it's limited, yes) and check how it actually works, check if this will be as you expected. (You can save another 10$ with using free .co.cc, .uni.cc etc domains). Then when you learn how it actually works and when you will personally know if this worth it - you can build dedicated server for this. (just a suggestion, you'll do what you want).
