I looked around and could not find the solution as to how to add a custom class to the form element/elements or the <form> tag for that matter. I have a custom form validation function which does some custom validation. form_set_error does set an error class on the elements but I wanted to add my custom class anywhere within the form tag.
Since the goal is just to customize the display of your field on error, a cleaner way is to create your own theme_form_element() in your theme and use the function form_get_error($element) to add the class you want if any error is returned on a field. Using this method you can also display the error message next to the field in error, instead of on top of the form.