For the design? The downside is that you could end up with a very generic looking application/site. It's also not your (or your developer's) code so there will usually be a period of time required to get used to it in order to make any adjustments.
In the end, it may not really save you that much time...Depending on what it does or doesn't do.
Templates certainly aren't a bad place to start though. They can save time in many cases. Just not all cases.
Using a template limits your web application from being unique and performing the way you really want it to perform as envisioned.
In my opinion, a custom designed web application makes your branding unique and makes your business and application appear apart from the rest. You are best having a custom solution.
I also build templates and custom web applications, so if you have further questions, please let me know.
I've been a web developer for 10+ years and I have experience with custom building something versus using something that is templated. There are trade-offs to both and the right answer depends on what you need.
Using a template is really useful when a lot of pages are going to be the same. However, the downside is that it's a template. It will naturally have restrictions and there will always be something you will need on on off pages.
Having something custom made is more level of effort. It's harder to maintain consistency in an application and making something reusable across projects becomes more difficult. However, it does allow you more flexibility to change things on the whim without the confines of a template.
If you want to discuss a particular system in more detail, I'd be happy to have a call with you to answer your follow up questions.