app.open will encounter problems that you really have no control over so you get an annoying dialog that you can't do anything with. The one that caused me the most grief is when a script tries to open a corrupt jpg image. IMNSHO, app.open should throw an Exception if it can't open the file and DisplayDialogs.NO is set. That way you can log the filename and deal with it later.
Bottom line is that there is nothing you can do in JS. One thing I have done in the past was use a utility like AutoKey on Windows that looks for these dialogs and automatically responds to them so you can continue processing. I'm fairly certain there is something similar available for OS X.