It's possible although perhaps unlikely that your computer may be overheating. This explains the delay in the issue occuring, and why it happens during a relatively stressful thing for the computer to do (encoding), but perhaps not others.
Otherwise see if you can get a GPU stress tester (http://www.ozone3d.net/benchmarks/fur/) and examine if it occurs after running for a similar amount of time to xsplit. This can hopefully narrow down if it's a hardware problem or a software issue.
Obviously ensure all drivers etc are up to date, uninstall / reinstall.
I've had similar issues with my computer before in the past, drove me nuts for 2 months. It would completely freeze with no mouse movement, keyboard response, anything after a little while. Occasionally I'd get a recovery error on start up with regards to an nvidia dll. Turns out it was a faulty graphics card. Resulting from this I wouldn't ignore the AMD error, it may rightfully be pointing towards graphics.
If it's freezing up on furmark and your temps are staying in a reasonable range then the next step will be to try and borrow a graphics card off someone or use your onboard graphics card and try again. If it doesn't occur this time then you can correctly diagnose the graphics card as your most likely issue and RMA it.
Even the smallest donations help keep sc2sea running! All donations go towards helping our site run including our monthly server hosting fees and sc2sea sponsored community tournaments we host. Find out more here.