I am having the same issue. I resampled my terrain in GIS and still did not solve it. Exporting to a shapefile works in v4.1.0 but not v.5.0.3. When I exported in v.4.1.0, the WSE was somewhat pixelated. Also, using the Raster Domain tool created a very pixelated boundary as well. The shapefile in v4.1.0 was much better.
Any fix to getting this to work in the new version? RAS mapper makes it too easy to map the floodplain but yet impossible to export that out to GIS.