It's a nice idea and someone I know also talked about this and would like this.
Just some things that I thought of (not shooting down the idea) having worked with sound, from recording to mixing and mastering. For the English dub, fans would likely have to travel to a studio in USA or Canada where a lot of voice work is done. Using different studios close to people could result in different sounding recordings due to the equipment being used, recording technique etc. This sounds weird in game and I've noticed this in a few titles. Even with communication, the sound engineers might not always be able to replicate results from other places.
Having the director at one place is also better to guide everyone.
For the other games Yu made people fly to Japan instead so he could follow the process closely. The cost of this, and the directing got criticized though: http://www.shenmuedojo.net/forum/viewtopic.php?t=40831
What happens though when they don't like the performance? Or a foreign accent which clashes with the native English voice cast? Quite difficult situation, especially when the fan has paid for it. Just using the voice as background murmur seems anticlimactic.