There is an interesting notebook on https://github.com/Trusted-AI/adversarial-robustness-toolbox/blob/main/notebooks/expectation_over_transformation_classification_rotation.ipynb to perform adversarial attacks when the image is also rotated.
However, I would like to edit the transformation to include other transformations.
From the code, it is seen that the following class is imported for the rotation:
from art.preprocessing.expectation_over_transformation import EoTImageRotationTensorFlow
eot_rotation = EoTImageRotationTensorFlow(nb_samples=eot_samples,
clip_values=clip_values,
angles=eot_angle)
Then, the attack uses the information about the EOTImageRotationTensorflow class to attack the image
classifier_eot = TensorFlowV2Classifier(model=model,
nb_classes=nb_classes,
loss_object=loss,
preprocessing=preprocessing,
preprocessing_defences=[eot_rotation],
clip_values=clip_values,
input_shape=input_shape)
attack_eot = ProjectedGradientDescent(estimator=classifier_eot,
eps=eps,
max_iter=num_steps,
eps_step=eps_step,
targeted=True)
when I checked such a class in /usr/local/lib/python3.8/dist-packages/art/preprocessing/expectation_over_transformation/image_rotation/tensorflow.py, I saw that there is a function that performs the rotation.
def _transform(
self, x: "tf.Tensor", y: Optional["tf.Tensor"], **kwargs
) -> Tuple["tf.Tensor", Optional["tf.Tensor"]]:
"""
Transformation of an input image and its label by randomly sampled rotation.
:param x: Input samples.
:param y: Label of the samples `x`.
:return: Transformed samples and labels.
"""
import tensorflow as tf # lgtm [py/repeated-import]
import tensorflow_addons as tfa
# pylint: disable=E1120,E1123
angles = tf.random.uniform(shape=(), minval=self.angles_range[0], maxval=self.angles_range[1])
angles = angles / 360.0 * 2.0 * np.pi
x_preprocess = tfa.image.rotate(images=x, angles=angles, interpolation="NEAREST", name=None)
x_preprocess = tf.clip_by_value(
t=x_preprocess, clip_value_min=-self.clip_values[0], clip_value_max=self.clip_values[1], name=None
)
return x_preprocess, y
IN THEORY, I could simply change the _transform() function to make the transformation I want.
However, such a function is not used to rotate the images, I can even erase the function that nothing happens if I rerun the algorithm. I even created a python for loop to open all python files in the art library and search for a keyword rotate but even commenting these entries the algorithm runs without problems
Therefore, I believe the rotation happens somewhere I don't have the minimum idea where it is.. Is that any way of me doing reverse engineering and finding exactly where the rotation is performed?
One option is to add a deliberate error into the function, such as 0/0
(division by zero). That will tell you whether it's called, and where from...