Look at the \Magento\Framework\Data\Form\Element\Fieldset::addField method:
https://github.com/magento/magento2/blob/2.0.0/lib/internal/Magento/Framework/Data/Form/Element/Fieldset.php#L209-L217
As you can see a fieldset tries to set a common renderer to an element:
https://github.com/magento/magento2/blob/2.0.0/lib/internal/Magento/Framework/Data/Form/Element/Fieldset.php#L212-L214
It is good, but I propose to do it better: additionally try to set a renderer, specific to the fieldset, if it is present, for example:
if ($renderer = $this->getElementRenderer()) {
$element->setRenderer($renderer);
}