django sum получите 0, если нет
from django.db.models.functions import Coalesce
answers = Answer.objects.filter(<something here>)
.annotate(score=Coalesce(Sum('vote__type'), 0))
.order_by('-score')
Repulsive Raven