Typically just moving from one server to the next has no issues at all so long as you're not making any other changes to the site, such as changing URLs in a re-design process. Just make sure its a quality, respected server. If the server has issues then, yeah, you'll inherit them. But otherwise you'll be fine.
I don't know of any real data out there that says changing the IP address of a site will negatively affect you as far as Google is concerned. However, make sure that:
1. The new server is just as fast and well-optimized as the old server
2. The new host/server is still in the same country/location as your targeted audience (i.e. don't host a US-Based company in Zimbabwe).
It's highly unlikely that moving web hosting would impact your rankings. However there are a few situations where your visibility could decrease if not accounted for:
During the move the website isn't available for a sustained period - crawlers won't be able to reach the content on the website causing your rankings to decreases over time.
If your server response time increases - make sure your new hosting is to the same performance standard, slow load times DO impact rankings.
HTTPS encryption - if your current website has a SSL certificate you will need to ensure this is updated/retained for your new server to retain the (small) existing ranking benefit.