David, I came across this with my middle-Mississippi model 4.1 to 5.0.3 conversion as well. In the Unsteady Flow Analysis window under Options -> Calculation Options and Tolerances… I discovered the Water surface calculation tolerance was set to 0.5ft. I lowered this below the max of 0.2 and RAS 5.0.3 executes just fine.
It seems like when our models were set up or converted from DWOPER/FLDWAV, the modelers had lower expectations on the accuracy of calculations. Or maybe to keep runtimes down, they deemed a half foot of instability as acceptable. Somehow RAS 4.1 was ok with it, but RAS 5 is not.