Angular remote form not displaying error

I have created an angular form that displays input validation errors received from the server. My solution works fine except for one small issue. If I submit the form with no value, after the page loads, I am receiving the correct response from my server i.e. 422, but the validation error is not displayed. If I then start typing a value in the input the validation error flashes and disappears.

I am almost certain that it has something to do with my directive, but I'm not sure how to fix it. This is my current directive code:

var appServices = angular.module('webFrontendApp.directives', []);

appServices.directive('serverError', function(){
  return {
    restrict: 'A',
    require: '?ngModel',
    link: function(scope,element,attrs,ctrl){
      element.on('change keyup', function(){
        scope.$apply(function(){
          ctrl.$setValidity('server', true);
        });
      });
    }
  };
});

I think the issue is with the element.on('change keyup'.... section of this code. That's why the error message flashes when I start typing. Also when I change this to 'change' instead of 'change keyup', the error message is displayed permanently when I start typing.

Does anybody have an idea of how I can display the error message even if I did not type any value into the input before submitting it the first time?

UPDATE AS PER COMMENT

Here is my form:

<form ng-submit="create(memberData)" name="form" novalidate>
  <div class = "row form-group" ng-class = "{ 'has-error' : form.email.$dirty && form.email.$invalid }">
     <input type="text" ng-model="memberData.email" placeholder="janedoe@mail.com"  name="email" class="col-xs-12 form-control" server-error>

     <span class="errors" ng-show="form.email.$dirty && form.email.$invalid">
       <span class="glyphicon glyphicon-remove form-control-feedback"></span>
       <span ng-show="form.email.$error.server">{{errors.email}}</span>
     </span>
  </div>
  <div class="row">
    <button type="submit" class="btn btn-danger col-xs-12">Join Private Beta</button>
  </div>
</form>

And my controller:

$scope.memberData = {};
$scope.create = function() {
    var error, success;
    $scope.errors = {};
    success = function() {
      $scope.memberData = {};
    };    
    error = function(result) {
        angular.forEach(result.data.errors, function(errors, field) {
            $scope.form[field].$setValidity('server', false);
            $scope.errors[field] = errors.join(', ');
        });    
    };



    BetaMember.save({ beta_member: { email: $scope.memberData.email || "" }}).$promise.then(success, error);

};

It seems that there was an extremely simple workaround. Since I had the filter form.email.$dirty in my view, the error would not be displayed if the user clicked submit without clicking on the form first.

After removing form.email.$dirty and only having form.email.$invalid, it works perfectly. I think this is sufficient in my case, as this validation is dependant on a server response and will not be triggered before the form is submitted. The error object is also cleared up in my controller ensuring that the page does not load an error when it's first loaded.