I have fixed swatches by adding the product_page_image_small, product_swatch_image_medium, and product_swatch_image_large entries to the etc/view.xml file of my design theme.
An example (set your own sizes for swatches):
https://github.com/magento/magento2/blob/2.3.0/app/design/frontend/Magento/luma/etc/view.xml#L84-L92
The problem is described here: