You'll be happy to know it's already being done. If you look on youtube, you can see dozens of personal manned drone projects, some of which are quite professional. One company, E-Volo, is actually performing manned flights in Dubai. Check it out:
I predict this tech will make its way into the mainstream within a decade.