IMHO we should combine both approaches.

Yes, API for rapid application development and forward "cross-platform" compatiblity. But fully overridable and possible to be nuked from RAM, with memory mapped or otherwise accessible raw hardware ports which you can hack at will bypassing all the API and OS. Sure you can make a call to put a pixel at X,Y or draw a line from X1,Y1 to X2,Y2, but you can just as easily ride the raw video memory area and set up your network buffer in lower left of the display. Sounds acceptable?