To be honest, if you want to build muscle, you really should set your diet to accommodate that. I am not trying to tell you what you should do, but there is the easy way to do something, and then there is all the more difficult and most difficult ways to do it.
Sure there are some vegan doing it, or someone doing it intermittent fasting, or whatever new FAD. Just because they can do it, doesn't mean you can do it. Building muscle is already a tough en devour, and all you are doing now, is making it exponentially more difficult.
You should decide what your priorities are, and follow that.
/rant
Best of luck on your vegan muscle building journey, I know it won't be easy, and you will need all the support you can get.