tensorflowpytorchpre-trained-modelefficientnet

How to access the weights of a layer in pretrained efficientnet-b3 in torch?


It's not about loading the weights of a model. I am trying to see the weights of layers in loaded efficientnet-b3 model in the torch.

os.system('pip install efficientnet_pytorch')
from efficientnet_pytorch import EfficientNet
MODEL_NAME = 'efficientnet-b3'
effnet = EfficientNet.from_pretrained(MODEL_NAME) 
effnet.modules # this works, but only gives the module names
effnet.weights # doesn't work
effnet.layers # doesn't work
effnet.modules[1]  # doesn't work, second module is batch norm ._bc0

I would like some functionality to replicate the following TF code in torch, access weights of the first batch norm layer

from tensorflow.keras.applications import EfficientNetB3
base_model = EfficientNetB3(weights="imagenet")
base_model.trainable_variables[1].numpy() # indexing with 1 gives weights of conv layer

Output for the above code:

<tf.Variable 'stem_bn/gamma:0' shape=(40,) dtype=float32, numpy=
array([ 0.1913209 ,  2.7074034 ,  9.623442  ,  2.5562265 ,  3.127593  ,
        4.348222  ,  2.4381876 ,  3.4623973 ,  3.6115906 ,  4.1241236 ,
        2.18851   ,  8.9716835 ,  0.7232651 ,  0.6261555 ,  9.050293  ,
        7.9233327 ,  0.47725916,  3.4991856 ,  5.334402  ,  4.843143  ,
        1.4122163 ,  1.953061  ,  8.150878  ,  5.0044165 ,  2.3806598 ,
        4.2976685 ,  2.2239766 ,  0.551327  ,  7.799995  ,  3.3823645 ,
        1.8910869 ,  4.0793633 ,  0.73215246,  3.4526935 , 10.874565  ,
        2.0920732 ,  6.272054  ,  3.6823177 ,  4.2152214 ,  3.4319222 ],
      dtype=float32)>

Solution

  • You can access the weights of a model by calling model.named_parameters().

    In your case, for i in effnet.named_parameters(): print(i) returns a list of 2-tuples, where the first item in the tuple is the name of the parameter, and the second item is the actual parameter tensor.

    Here is a sample extract so you can see what I mean:

    [('_conv_head.weight', Parameter containing:
    tensor([[[[ 0.0074]],
    
             [[-0.0904]],
    
             [[-0.0091]],
    
             ...,
    
             [[-0.0504]],
    
             [[ 0.1140]],
    
             [[-0.0710]]],
    
    
            [[[ 0.0635]],
    
             [[ 0.0290]],
    
             [[-0.0583]],
    
             ...,
    
             [[-0.0813]],
    
             [[-0.0230]],
    
             [[-0.1120]]],
    
    
            [[[ 0.0873]],
    
             [[-0.0174]],
    
             [[-0.0170]],
    
             ...,
    
             [[-0.0014]],
    
             [[ 0.0323]],
    
             [[ 0.0569]]],
    
    
            ...,
    
    
            [[[-0.0218]],
    
             [[ 0.0292]],
    
             [[ 0.0267]],
    
             ...,
    
             [[-0.0044]],
    
             [[-0.0463]],
    
             [[-0.0257]]],
    
    
            [[[-0.0209]],
    
             [[ 0.0279]],
    
             [[ 0.0094]],
    
             ...,
    
             [[-0.1759]],
    
             [[-0.0702]],
    
             [[-0.0902]]],
    
    
            [[[-0.0488]],
    
             [[ 0.0276]],
    
             [[-0.0174]],
    
             ...,
    
             [[-0.0391]],
    
             [[-0.0268]],
    
             [[-0.0205]]]], requires_grad=True)), 
    ('_bn1.weight', Parameter containing:
    tensor([2.3115, 2.0343, 2.0015,  ..., 1.7868, 2.3552, 1.8885],
           requires_grad=True)), 
    ('_bn1.bias', Parameter containing:
    tensor([-1.8066, -1.4178, -1.3111,  ..., -1.0494, -1.8520, -1.1798],
           requires_grad=True)), 
    ('_fc.weight', Parameter containing:
    tensor([[-0.0157, -0.0483,  0.0139,  ..., -0.0201, -0.0092, -0.0752],
            [-0.0100, -0.0575,  0.0328,  ..., -0.0362, -0.0534, -0.0041],
            [-0.0083,  0.0142, -0.0006,  ...,  0.0151, -0.0033,  0.0500],
            ...,
            [-0.0840, -0.0217, -0.0354,  ..., -0.0620,  0.0143,  0.0786],
            [-0.0956, -0.0169,  0.0738,  ...,  0.1063, -0.0742,  0.0036],
            [ 0.0485,  0.0470,  0.1002,  ..., -0.0832,  0.1081,  0.0145]],
           requires_grad=True)), 
    ('_fc.bias', Parameter containing:
    tensor([-1.8788e-04, -2.1204e-02, -2.2974e-02, -3.5067e-02, -4.2469e-02,
            -4.0472e-02, -3.3574e-02,  7.5904e-03, -1.3298e-02, -1.3364e-02,
            -4.7947e-02, -6.8513e-02, -5.1592e-02, -4.0660e-02, -2.1086e-02,
            -5.7097e-02, -8.5144e-02, -4.0252e-02,  1.5397e-02, -2.6116e-02,
            -7.2120e-02, -4.8167e-02, -4.5482e-02, -5.7588e-02, -6.5176e-02,
            -4.3350e-02, -6.1431e-03, -3.3575e-02, -1.6232e-02,  4.4864e-03,
            -7.7549e-02, -6.1085e-02, -3.7735e-02, -4.0341e-02,  1.7911e-03,
            -7.5653e-02,  3.0368e-02, -3.7621e-02, -1.5108e-02, -1.8987e-02,
            -7.0831e-02, -4.8989e-02, -4.6129e-02, -3.8295e-02, -2.3450e-02,
            -7.5764e-02, -9.7884e-03, -2.0963e-02, -5.0398e-02, -3.1158e-02,
            -4.3633e-02,  2.1600e-02,  2.2160e-02,  9.6163e-03, -3.5956e-02,
             2.3973e-03, -3.3229e-02, -5.7656e-02, -1.0077e-02, -1.7506e-02,
             3.1533e-02,  4.6831e-02, -1.2585e-02, -5.1598e-03,  2.8223e-02,
            -8.0142e-03, -3.1187e-03, -5.5687e-02, -1.6465e-02, -6.9663e-02,
            -1.0505e-02, -1.2698e-02, -4.9928e-02,  2.3078e-02, -3.1174e-02,
            -9.6878e-03,  3.8667e-02,  1.0131e-02,  2.0751e-02,  1.1730e-03,
            -2.4952e-02, -6.8574e-02, -2.2635e-02, -8.7558e-02, -7.1596e-02,
            -9.8193e-04, -3.9943e-02,  1.3720e-02, -1.0468e-02, -8.8728e-03,
            -6.1661e-02, -6.7819e-02, -4.3548e-02, -6.7823e-02, -1.1364e-01,
            -8.0226e-02, -3.1601e-02, -4.1108e-02, -1.0578e-01, -2.4910e-02,
            -3.6718e-02, -6.0854e-03, -1.7552e-02,  4.8444e-02, -1.0772e-01,
            -8.1519e-02,  1.1975e-02, -5.9317e-02, -4.8866e-02, -1.0751e-01,
            -4.0855e-02, -2.0209e-02,  2.0903e-02, -4.4489e-02,  4.7728e-03,
            -1.0581e-01, -6.0772e-02, -5.2845e-02,  5.1716e-02, -6.6787e-02,
            -6.2687e-02,  8.3072e-03, -7.7149e-03, -8.9555e-02, -4.8636e-02,
            -7.7438e-02, -2.0582e-02, -3.4283e-02, -6.6013e-02, -8.3553e-02,
            -1.7237e-02,  1.0309e-02, -1.3888e-02, -8.5651e-02, -1.0593e-02,
            -8.5910e-02, -4.4669e-02, -4.7769e-02, -5.6502e-02, -3.7197e-02,
            -5.9685e-02, -6.1591e-02, -3.5832e-02, -3.9733e-02, -2.6279e-03,
            -7.5230e-03, -4.8190e-02, -5.2629e-02,  1.3794e-03, -6.2900e-02,
             1.2298e-02,  3.3339e-02, -3.2597e-02,  2.7527e-02, -1.2776e-02,
             4.9703e-02,  1.7361e-02,  2.0815e-02, -3.7160e-02,  3.7085e-02,
            -3.6573e-02,  6.2998e-02,  6.3216e-02,  3.9644e-02,  2.1697e-02,
            -7.6096e-02, -3.1489e-02, -4.0166e-02,  3.0928e-02,  1.8492e-02,
             3.6115e-03,  6.6122e-02,  3.6215e-02,  1.1731e-02,  6.3609e-02,
            -5.1814e-02, -2.1708e-02,  2.3418e-02,  1.1752e-01, -5.7509e-05,
            -9.9666e-03,  3.7792e-02,  5.2852e-02,  2.7368e-02,  8.6873e-02,
             5.4690e-03,  1.4777e-02,  1.8022e-02,  1.4611e-02,  8.3892e-02,
            -2.4939e-02,  9.1274e-02,  6.7359e-04,  1.2745e-02, -3.5555e-02,
             1.3300e-01,  4.2316e-02,  3.8856e-02, -1.3746e-02,  7.2174e-02,
            -3.3013e-02,  2.3095e-02,  4.3574e-02,  9.0962e-02, -2.4382e-03,
             1.9608e-02, -9.1173e-03,  5.3899e-02,  4.8285e-02,  8.5972e-02,
            -1.0665e-02,  3.8411e-02,  3.8558e-03,  5.2799e-03,  1.4861e-02,
             3.9330e-02,  7.3648e-03,  8.2185e-02,  5.4206e-02,  5.7331e-03,
             6.0913e-02, -2.7097e-02,  9.5242e-03,  5.7349e-02, -8.2799e-03,
             3.3900e-02, -4.3817e-02,  7.2375e-03, -7.7864e-03,  6.6191e-03,
            -1.6398e-03,  3.7898e-03,  2.0504e-02, -2.7386e-02,  3.3018e-02,
             2.3809e-02,  6.7705e-02,  7.4084e-02,  4.5319e-02,  4.4979e-02,
             2.6763e-02,  2.5106e-02,  5.7135e-02,  4.2436e-02, -5.5549e-02,
             8.4225e-02, -1.8866e-02,  5.4292e-02,  7.3890e-02,  4.6438e-02,
             2.8466e-02,  9.7005e-02, -1.9982e-02,  5.4305e-02,  5.3359e-02,
             4.3933e-03,  1.2966e-02, -1.3056e-02,  2.8389e-02,  6.9528e-03,
            -9.5808e-03,  1.3317e-02, -5.5829e-02,  4.7666e-03,  3.1541e-02,
             2.3011e-02,  4.1077e-02,  5.9078e-02, -5.6244e-02, -1.7187e-02,
             4.5811e-02,  3.0973e-02,  1.8721e-02,  8.7556e-03, -1.3825e-02,
             1.6460e-03, -6.1395e-02, -1.6215e-02, -9.6606e-03, -3.0871e-02,
             2.9623e-02,  4.9153e-02,  4.9580e-02, -1.5432e-02,  4.9673e-02,
             3.5004e-02, -3.5870e-02, -4.6010e-02, -3.7892e-02, -5.9807e-03,
             7.9569e-03, -8.6313e-02, -2.0522e-02, -2.7805e-02, -4.1220e-02,
            -4.9287e-02, -9.1146e-03, -2.2379e-02, -3.7943e-02, -2.0446e-02,
            -6.7551e-02, -2.6903e-02,  3.5022e-03, -4.7364e-02, -1.0330e-01,
            -6.1197e-02, -1.9515e-02, -6.4015e-02, -3.7570e-02, -4.0806e-02,
            -1.7590e-02, -6.0645e-02, -3.7628e-02,  2.2510e-02,  6.6436e-02,
            -3.2913e-02, -7.2844e-02, -8.4280e-02, -9.0194e-04, -3.3313e-02,
            -5.0213e-02, -6.0932e-02, -6.3614e-02, -6.2003e-02, -1.8194e-02,
            -7.3311e-02, -6.6968e-02, -1.9100e-02, -6.7069e-02, -9.9181e-02,
            -2.4951e-02, -2.4315e-02, -2.8859e-02,  2.5487e-02,  1.7767e-03,
            -4.7624e-02, -1.6528e-02, -2.6242e-02, -2.0435e-02, -4.6075e-03,
             1.4161e-02, -1.7872e-02, -2.7038e-02, -2.0696e-02, -3.1628e-02,
             3.1667e-02,  1.2643e-02,  3.3634e-02, -2.2169e-02, -1.3101e-02,
            -2.6296e-02, -2.9903e-02, -5.1367e-02, -2.5846e-02,  1.1791e-02,
            -2.0295e-02,  2.2274e-02,  3.4522e-02,  9.1384e-02,  5.7635e-02,
            -3.7340e-02,  4.5483e-02,  9.2709e-03, -3.4176e-03, -1.8723e-02,
             4.7653e-02, -4.3498e-02,  1.5783e-02, -2.6597e-03,  4.8437e-02,
             5.4055e-03,  5.2496e-02, -7.5699e-02, -7.2362e-03, -2.4031e-02,
            -3.8313e-02, -5.7455e-02, -3.2797e-02, -1.0619e-02, -3.5333e-02,
             6.8303e-02,  4.2202e-02,  1.2961e-02, -1.0357e-02, -7.2061e-02,
             3.5756e-02,  1.0100e-02, -6.8939e-03, -3.2524e-03,  1.9981e-02,
            -7.9577e-02, -9.9632e-03, -6.4924e-02, -7.7214e-02, -4.2186e-03,
             7.7079e-03, -6.8280e-02, -9.0313e-02,  5.5780e-02,  4.0597e-02,
             1.5803e-02,  7.6331e-02,  8.0219e-02, -4.2492e-02, -5.0285e-02,
             2.8535e-02, -5.1758e-02, -5.5653e-02, -2.6481e-02,  8.4919e-02,
             3.3701e-02, -4.7973e-02,  4.3328e-02,  5.4467e-02, -3.5105e-02,
            -2.8679e-02,  3.2789e-02, -3.0762e-02, -5.7570e-03,  6.3135e-02,
             2.6109e-02,  4.5541e-02,  1.5574e-02,  3.4254e-02,  2.1206e-02,
             8.7765e-03, -1.5536e-02,  8.7542e-02,  2.8484e-02,  7.4923e-02,
             2.7793e-03,  3.0007e-02,  4.5130e-02,  3.4480e-02,  1.4038e-02,
             1.0104e-01,  8.9144e-03, -6.6594e-03,  2.3476e-02,  2.1639e-02,
             8.8072e-02,  3.4813e-02, -6.1209e-02, -1.9577e-02, -1.3342e-02,
             8.3109e-02, -5.9933e-02,  4.5758e-02,  8.2524e-02, -3.3844e-02,
            -2.8108e-02, -1.2548e-02,  2.2878e-03, -3.2685e-03,  4.4044e-02,
             8.5652e-02,  3.7278e-03,  7.4127e-02, -4.7680e-02, -1.0711e-02,
            -1.9743e-03,  2.8187e-02,  3.1251e-02,  1.3193e-01,  1.3656e-02,
            -8.9823e-03, -8.9561e-03, -3.4255e-02,  2.8345e-02,  3.1666e-02,
             5.3442e-02,  2.4717e-02, -1.8101e-02,  2.4845e-02, -1.5233e-02,
             1.0711e-01, -2.0983e-02, -4.5709e-02, -8.3504e-04, -2.5989e-02,
             1.4240e-02,  1.3953e-02, -1.6013e-02,  9.2497e-04, -3.4750e-02,
            -3.9424e-02,  7.6613e-02, -1.2504e-02,  1.2788e-01,  7.7914e-02,
             6.1930e-02,  1.4219e-03, -2.5443e-02,  6.1517e-03,  6.1918e-02,
            -2.7085e-02,  7.3593e-03, -7.3484e-02,  2.8937e-02,  9.8165e-02,
             2.0922e-03,  4.2529e-02, -1.1309e-02, -4.9490e-02,  1.6751e-02,
             1.0809e-02,  2.3618e-02,  2.0515e-02,  3.3854e-02, -1.1171e-01,
            -9.9522e-03, -2.2146e-02,  1.3169e-02,  1.1341e-01,  6.5784e-02,
             1.1520e-01, -1.2588e-02, -7.9052e-02, -1.4347e-02,  4.7605e-02,
             3.4523e-02, -4.4671e-02,  1.1775e-02,  1.0015e-01,  2.6564e-02,
            -1.3457e-02, -4.4883e-02,  3.8352e-02,  1.2466e-02, -1.1996e-03,
             1.0156e-01, -9.3025e-02,  1.9178e-02, -3.7339e-02, -3.1499e-02,
             1.3383e-02, -2.9213e-02, -4.8970e-02, -4.5686e-02,  2.8906e-02,
            -5.9064e-02, -1.5512e-03,  1.9657e-02,  7.8270e-03, -4.0602e-02,
             1.9702e-02,  4.0297e-02, -1.9938e-02, -1.2144e-02, -2.3456e-02,
            -5.3594e-02, -4.2403e-02,  7.4747e-02,  1.2299e-02, -2.0902e-02,
            -5.5567e-02,  1.4648e-02,  1.4435e-03,  1.0079e-01,  9.2539e-02,
            -1.0607e-02, -1.4108e-02,  9.1721e-03,  1.1072e-02,  2.9608e-02,
            -7.6078e-03,  2.5653e-02, -1.1136e-02,  7.2409e-02, -5.1752e-02,
             8.0248e-02, -6.1112e-02,  6.1196e-02, -2.2555e-02,  8.3575e-02,
            -3.1096e-02, -4.8845e-03,  7.4904e-02, -6.5950e-03,  1.9850e-02,
            -3.0121e-02,  3.6329e-02, -4.4467e-03,  8.3675e-02, -7.4437e-02,
            -4.0771e-02, -2.8450e-02,  8.7870e-02, -4.0946e-02, -3.8329e-03,
            -6.9315e-02, -1.0016e-02, -2.0778e-02,  8.0915e-02,  3.9063e-02,
            -1.5094e-02,  3.3627e-03, -2.3838e-02,  3.1394e-02,  1.2513e-02,
             1.3142e-01,  3.0603e-02, -2.2055e-03,  2.0502e-02,  1.0135e-01,
             7.1754e-02,  9.7944e-03,  3.7959e-02,  1.3989e-02,  1.7989e-02,
            -3.4136e-03,  3.9650e-02, -3.4606e-02, -2.8792e-03, -5.2186e-03,
            -3.3676e-02,  5.8599e-02,  6.2828e-02,  4.3374e-02,  2.6202e-02,
             9.4092e-02, -3.8893e-03,  9.6722e-02, -6.6398e-02, -5.4730e-02,
            -4.2413e-02,  2.9454e-02, -1.1035e-02, -1.3873e-02,  3.5274e-02,
            -5.2795e-03,  7.2810e-03,  5.6811e-03,  2.5880e-02, -1.6221e-02,
            -4.2152e-02,  2.0760e-02,  1.4120e-02,  7.2473e-03,  6.9235e-03,
            -6.1877e-02,  8.1591e-03,  5.1506e-02,  4.0437e-02,  1.4943e-02,
            -2.9214e-02,  7.7097e-02, -3.1952e-02,  3.0927e-03, -6.7025e-04,
             6.2347e-02,  8.4001e-03,  3.2072e-02,  5.1373e-02, -2.0174e-02,
             1.1651e-02,  1.1074e-02,  1.7667e-02, -8.6863e-03,  3.0662e-02,
            -1.7535e-02, -5.4594e-02, -1.0278e-02, -6.7767e-02,  1.8709e-02,
             4.3460e-02,  2.7637e-02,  5.1533e-02, -3.5134e-02,  3.0937e-02,
             1.0212e-02,  3.0756e-02,  6.7154e-02,  3.9204e-02,  6.0210e-02,
            -6.5048e-02,  7.2162e-02,  4.5481e-02,  4.0012e-02, -2.6958e-02,
             2.3580e-03,  1.9551e-02,  1.4279e-02,  6.8494e-02,  2.1821e-02,
            -3.8355e-02, -5.8506e-02, -7.3684e-02, -7.3094e-03, -4.3607e-02,
            -2.3467e-03, -8.3147e-03,  6.3290e-02, -1.1461e-02, -1.4600e-03,
             6.5386e-02,  9.3397e-03, -4.8493e-02, -2.3290e-02,  4.6487e-02,
             1.0229e-01,  4.5283e-03,  4.7067e-02,  1.0678e-02,  5.5262e-02,
             7.6263e-03, -3.7516e-02,  1.8642e-02,  1.2551e-02, -7.5747e-02,
            -6.9164e-03,  9.2134e-03, -7.1654e-02, -3.2144e-02,  4.4780e-02,
             1.6665e-02,  5.4213e-02, -5.7801e-02, -4.3401e-05,  9.2516e-02,
             2.4602e-02, -2.6873e-02,  6.9050e-02, -1.1062e-02, -6.5441e-02,
             3.2812e-02,  1.6676e-02, -3.6145e-02,  6.2055e-02, -5.5214e-02,
            -2.6646e-02,  1.5051e-01, -1.4609e-02,  7.3359e-02, -1.1975e-02,
             2.5550e-03,  2.8461e-02,  6.1058e-02,  1.5934e-02,  1.1280e-02,
            -5.7301e-02, -8.1453e-02, -1.3112e-02,  1.4247e-02,  2.6493e-02,
            -3.2159e-02, -1.2888e-02,  3.9029e-02, -3.8758e-02,  1.1523e-01,
             5.7670e-02, -3.0118e-02, -1.3478e-02,  5.0629e-02, -1.2562e-02,
            -3.1457e-02, -7.3343e-03, -4.2227e-02, -5.6207e-02, -2.7644e-02,
            -2.3450e-02,  3.6726e-02, -1.7531e-02,  5.1430e-02,  8.6157e-02,
             3.9421e-02, -3.5995e-02, -1.9596e-03,  1.4217e-02,  8.6251e-02,
             9.4631e-04, -8.1190e-02,  9.1071e-02,  2.4760e-02, -1.1475e-02,
             2.2392e-02,  5.3320e-02,  5.3901e-04, -4.8530e-02, -4.3654e-03,
            -5.6498e-02, -1.8796e-02,  7.8893e-02,  4.2227e-02,  5.3552e-02,
             7.8111e-02,  2.7515e-02, -1.6096e-02,  1.2153e-02,  6.6272e-03,
            -4.8895e-02,  5.0869e-02,  6.7843e-02,  6.1599e-02, -1.1996e-02,
            -9.4545e-03,  1.0977e-01,  5.2409e-02, -4.7280e-02,  3.6631e-02,
            -2.9781e-02, -4.2014e-03, -4.2217e-02, -1.1914e-02, -2.9351e-02,
             1.4702e-01, -2.4856e-02, -5.4605e-02,  9.2256e-02, -5.8185e-02,
            -4.9112e-03, -4.8391e-02,  9.6750e-03,  9.3729e-02, -7.0812e-02,
             1.3358e-02, -3.0346e-02, -1.2634e-02,  5.4181e-02,  3.5779e-03,
            -1.3641e-02,  1.8990e-02,  4.1806e-02,  1.4102e-02, -3.5755e-02,
            -9.7328e-03, -6.9314e-02, -1.1238e-02,  4.4573e-02, -5.3725e-02,
             1.4389e-04,  3.9408e-02, -4.2209e-02, -3.2018e-02,  8.1474e-02,
             9.0362e-03,  8.6104e-02,  1.3965e-01, -1.3486e-02,  3.1769e-02,
             3.9424e-02, -2.7504e-02,  4.4009e-02,  4.5092e-02,  3.0902e-02,
             9.5173e-02,  6.0828e-02,  1.8025e-03, -8.7061e-03,  6.5157e-03,
             3.7598e-02,  1.0603e-01,  1.6571e-01,  5.5066e-02, -7.3351e-03,
            -5.2073e-03, -6.4366e-02, -6.7670e-02,  5.9512e-02,  6.4599e-02,
            -5.4156e-02,  3.0373e-02, -6.4649e-03,  6.1524e-02, -6.0863e-02,
            -5.4839e-02,  3.2778e-02, -5.2311e-02, -6.3305e-02, -1.4740e-02,
             8.6265e-03, -1.4593e-02,  7.0843e-02,  9.6598e-03, -4.1514e-02,
             1.0386e-01,  9.7102e-02,  2.7971e-02, -8.3213e-03,  8.9160e-02,
             6.4613e-02,  7.2493e-03,  2.0423e-02, -1.7861e-02, -1.0312e-02,
            -2.7538e-02, -4.7007e-02, -5.1747e-02,  5.8215e-02,  6.6977e-02,
            -1.7237e-02, -3.6396e-02,  5.7006e-02,  7.6054e-03, -4.7352e-02,
            -9.1480e-03,  4.0137e-02,  6.6371e-03,  7.7091e-02,  5.0351e-02,
             6.8603e-04, -5.7972e-02,  1.2907e-02,  3.9684e-02, -2.7078e-02,
             5.0169e-02,  3.0860e-02,  5.1556e-02, -4.8597e-02, -2.1098e-02,
             5.4887e-02, -4.1379e-02,  2.0082e-02,  1.8863e-02, -1.6039e-02,
             4.4650e-02, -2.7449e-02,  2.2729e-02,  3.3439e-02,  6.7221e-02,
             3.5019e-02,  1.2706e-02, -3.3244e-02, -2.0954e-02, -7.2121e-02,
            -2.5076e-02, -9.3874e-02, -1.4208e-02, -4.0450e-02,  1.0401e-01,
            -2.2949e-02, -2.6151e-02,  8.9983e-03, -7.2041e-02, -2.8341e-03,
            -4.9221e-02, -4.2876e-02,  6.4064e-03, -2.5436e-02,  1.6128e-02,
            -4.1882e-02, -3.8972e-02, -4.6335e-02, -5.4710e-02, -1.8748e-04,
            -5.4616e-02, -5.6258e-02, -3.8972e-02, -2.6863e-02,  1.6232e-02,
             1.6155e-02, -1.8721e-02, -2.4802e-02, -1.3311e-02, -1.7580e-02,
            -4.3439e-02, -4.1209e-02, -2.6509e-02,  2.1775e-02, -1.2127e-02,
            -1.7708e-02, -1.5500e-02, -7.4598e-02, -5.0580e-02, -6.6671e-02,
            -4.6614e-02,  4.4161e-02,  1.9872e-02,  1.7880e-02, -7.2944e-03,
             9.7257e-03,  1.2171e-01,  2.0895e-03, -7.2846e-02, -9.7906e-03,
            -5.5273e-02, -1.8346e-03,  3.9121e-02,  1.7541e-02,  1.6898e-02,
             2.0039e-02, -8.8893e-03, -2.4714e-02, -3.4171e-02,  1.0218e-02,
             1.2038e-02, -5.3019e-02,  3.3046e-02, -3.6093e-02, -5.2178e-02,
            -4.7642e-02, -6.6923e-02, -2.2623e-02, -7.2356e-02, -1.1350e-01,
            -2.2522e-02, -1.8415e-02, -5.9977e-02, -2.2627e-02,  4.6349e-02],
           requires_grad=True))]