15 years experience in early-stage startup growth. I consult on overall strategy, Paid Advertising, PR, SEO and more for young companies that are ready to start growing efficiently and effectively without wasting money and time! I have a long resume to prove it!
The easiest answer to this is something a mentor told me a few years ago when I was working on a niche app....she said "do you honestly feel like you could be passionate about this for the next 10 years"? That may not sound like it's relevant but it is...honestly that's the only thing that matters when starting a compay (besides the obvious of is this valuable)...That's the first thing to figure out. Then when I answered and said maybe, she said "do you think that this is the biggest thing you could be doing with your time? Why stop at helping a few thousand people, why not pour 10 years of your life into something that can help millions".....so reframe your question, is my product only helping a few people? Is there something else I could be doing that will help more people? If the answers are obvious then you know what you need to do...back to the drawing board.
I would really need to see the ads that you're selling to be able to suggest what method to scale them with... but you hit the nail on the head...You have two options really...building internal sales processes or going through an agency. Usually companies go through an agency first, to get professional handholding in "sales product" development...ie. they help you scale your pricing and ad units and offerings...and get your foot in the door with bigger clients...but you can do a short term contract with most agencies...say 6 months...so that after 6 months you are in a much better spot to handle it on your own and stop paying 30% to an agency...
Step one, spend a day researching competitors, keywords, onpage and offpage SEO...and then get yourself knowledgeable enough to find an expert who has experience in your industry and then work with them...try and implement their suggestions yourself if your team can...that will reduce cost.