This is actually an error produced by Windows and typically happens on 32 bit servers when trying to backup large Exchange or SQL database files (e.g. 80 GB or more). You should be able to see the error in the Windows event log which should also be checked for further info, but this Microsoft KB article may help: http://support.microsoft.com/kb/304101
You may need to increase the page file size since the RAM is limited on the 32 bit server.