It's very much so possible to login with a forums account, see this 2 year old branch (with 2 factor!)
The issue is that XF RESTful API isn't really meant for that type of integration, see it's docs on API keys that are made in the ACP. Trying to make an addon for this would difficult to...