Typically open-source projects that end up generating a profit have an active product that's already being used.

So step 1. Is to open source it and see if anyone uses it - then you can make money in a bunch of ways.

Checkout Wordpress, MySQL, Linux for examples - but typically it's providing paid devices for either hosting, managing or customizing the software.

The hardest part is getting others to use and contribute to it.

Fully agree with Dan.

The most important question is: could you build a community around your software?

Which, by the way, will no longer be your software.

Hope this helps

PS: I am currently selling Open Source Software (OSS) and the key is to be competing with proprietary software (not other OSS companies)

Steve