The court certainly will NOT see you as a gold digger. That is just ridiculous. Someone must be having their own child support issues and is attempting to join the ranks of the lazy parent brotherhood.
It is obvious to me that your baby's father is attempting to intimidate you and, thus far, he is doing an excellent job. Unfortunately for him, that is not how things work. He can use your past all that he wants to but, in a court of law, your past is irrelevant unless you have been convicted of child abuse, neglect or endangerment. Aside from those things, your past does not matter in a court of law. If you did drugs or drank ever weekend, it doesn't matter and any claim that he may have had has gone out the window because, for the past 2 years, he has felt that you were more than capable of caring for your son. The court will absolutely consider that in any decision it makes.
You can be found guilty of smoking marijuana if the court compels you to take a drug test so stop doing marijuana. Stop smoking, get clean for a month or two, and then file for child support. As long as you file first the court will absolutely recognize that this man has had no interest in his child, or caring for him, until you filed a motion for support. All the money in the world cannot change the fact that he has never seen his own son.
He may be able to provide the court with proof that he has the financial ability to care for his son but that does not mean that you will lose custody. It simply means that the court can confirm that he can afford to pay child support. The fact that he has never seen or met his son will have a much bigger impact on any decision the court makes. Do not let his lawyer bully you. Decide what you want, stick to it, remain unshakable and go for it.
However, if you are doing this just because you feel that you can get more money, the court will see right through you. If the court feels that you are not able to provide the best home, and if they feel that you are using your child to secure finances, the court may very likely favor any petition he files for custody and you may wind up being the one paying support. Children are not game pieces or bargaining chips, consider that. The reason that I point this out is because you stated that you know your son could be getting a lot more. If your son is cared for with basic needs, a home, food and medical expenses, he does not need anything more.