What is my problem, is it a childhood issue/daddy issue?

To this point you have been a willing participant in this individuals need to be in total control. You can end your participation at any time by totally ignoring him. You must do this if you want to be free.