fiwarefiware-cygnusfilabfiware-cosmos

OAuth2 in Cygnus


Since it was announced the access to Cosmos' WebHDFS in FIWARE Lab is protected with OAuth2, Cygnus (version 0.8.1) is not able to persist Orion's context data in Cosmos HDFS:

15/07/09 08:45:21 INFO handlers.OrionRestHandler: Starting transaction (1436424291-649-0000000000)
15/07/09 08:45:21 INFO handlers.OrionRestHandler: Received data ({  "subscriptionId" : "51c0ac9ed714fb3b37d7d5a8",  "originator" : "localhost",  "contextResponses" : [    {      "contextElement" : {        "attributes" : [          {            "name" : "temperature",            "type" : "centigrade",            "value" : "26.5"          }        ],        "type" : "Room",        "isPattern" : "false",        "id" : "Room1"      },      "statusCode" : {        "code" : "200",        "reasonPhrase" : "OK"      }    }  ]})
15/07/09 08:45:21 INFO handlers.OrionRestHandler: Event put in the channel (id=109760451, ttl=10)
15/07/09 08:45:21 INFO sinks.OrionSink: Event got from the channel (id=109760451, headers={timestamp=1436424321536, content-type=application/json, transactionId=1436424291-649-0000000000, fiware-service=def_serv, fiware-servicepath=def_serv_path, ttl=10, destination=room1_room}, bodyLength=460)
15/07/09 08:45:22 INFO sinks.OrionHDFSSink: [hdfs-sink] Persisting data at OrionHDFSSink. HDFS file (def_serv/def_serv_path/room1_room/room1_room.txt), Data ({"recvTime":"2015-07-09T06:45:21.536Z","temperature":"26.5", "temperature_md":[]})
15/07/09 08:45:22 ERROR sinks.OrionSink: Persistence error (The /user/frb/def_serv/def_serv_path/room1_room directory could not be created in HDFS. HttpFS response: 503 Service unavailable)
15/07/09 08:45:22 INFO sinks.OrionSink: An event was put again in the channel (id=109760451, ttl=9)
15/07/09 08:45:22 INFO sinks.OrionSink: Finishing transaction (1436424291-649-0000000000)

I figure out an OAuth2 token has to be configured somewhere, but how?


Solution

  • IMPORTANT: Before implementing any of the commands below, backup your Cygnus configurations since they will be still valid with minor changes.

    Sadly, available Cygnus versions have no support for OAuth2. Until now. You will have to upgrade to the latest 0.8.2, which allows to configure a token through this OrionHDFSSink parameter:

    <your_agent_nam>.sinks.<your_hdfs_sink_name>.oauth2_token = <token>
    

    Upgrading to Cygnus 0.8.2 is as easy as installing it from the FIWARE repo:

    $ (sudo) yum clean all      # just to clean the yum cache
    $ (sudo) yum list cygnus    # this will show you 0.82 is available
    $ (sudo) sudo rpm -e -vv --allmatches --nodeps --noscripts --notriggers cygnus # this is needed if you have installed a version < 0.8.0
    $ (sudo) yum install cygnus # this installs 0.8.2
    

    Remember the FIWARE repo is setup by editing /etc/yum.repos.d/fiware.repo (most probably you already have it):

    [Fiware]
    name=FIWARE repository
    baseurl=http://repositories.testbed.fi-ware.eu/repo/rpm/x86_64/
    gpgcheck=0
    enabled=1
    

    In the example below, we are moving from a 0.7.0 version to latest 0.8.2:

    $ sudo yum clean all
    Loaded plugins: fastestmirror, security
    Cleaning repos: ...
    Cleaning up Everything
    Cleaning up list of fastest mirrors
    $ sudo yum list cygnus
    Loaded plugins: fastestmirror, security
    Determining fastest mirrors
     * base: sunsite.rediris.es
     * epel: ftp.cica.es
     * extras: sunsite.rediris.es
     * updates: sunsite.rediris.es
    ...    
    Installed Packages
    cygnus.x86_64               0.7.0-g4cddd26            @Fiware
    Available Packages
    cygnus.x86_64               0.8.2-0.g7353160          Fiware 
    $ sudo rpm -e -vv --allmatches --nodeps --noscripts --notriggers cygnus
    D: loading keyring from pubkeys in /var/lib/rpm/pubkeys/*.key
    D: couldn't find any keys in /var/lib/rpm/pubkeys/*.key
    D: loading keyring from rpmdb
    ...
    D: closed   db index       /var/lib/rpm/Name
    D: closed   db index       /var/lib/rpm/Packages
    D: closed   db environment /var/lib/rpm
    $ sudo yum install cygnus
    Loaded plugins: fastestmirror, security
    Loading mirror speeds from cached hostfile
     * base: sunsite.rediris.es
     * epel: ftp.cica.es
     * extras: sunsite.rediris.es
     * updates: sunsite.rediris.es
    Setting up Install Process
    Resolving Dependencies
    --> Running transaction check
    ---> Package cygnus.x86_64 0:0.7.0-g4cddd26 will be updated
    ---> Package cygnus.x86_64 0:0.8.2-0.g7353160 will be an update
    --> Finished Dependency Resolution
    
    Dependencies Resolved
    
    ==============================================================================================================================================================
     Package                            Arch                               Version                                       Repository                          Size
    ==============================================================================================================================================================
    Updating:
     cygnus                             x86_64                             0.8.2-0.g7353160                              Fiware                              94 M
    
    Transaction Summary
    ==============================================================================================================================================================
    Upgrade       1 Package(s)
    
    Total download size: 94 M
    Is this ok [y/N]:
    Downloading Packages:
    cygnus-0.8.2-0.g7353160.x86_64.rpm                                                                                                     |  94 MB     00:03     
    Running rpm_check_debug
    Running Transaction Test
    Transaction Test Succeeded
    Running Transaction
      Updating   : cygnus-0.8.2-0.g7353160.x86_64                                                                                                             1/2 
    [INFO] Creating cygnus user
    [INFO] Creating log directory
    Done
      Cleanup    : cygnus-0.7.0-g4cddd26.x86_64                                                                                                               2/2 
    [INFO] Uninstall the cygnus
    Stopping Cygnus chatrooms...                      [  OK  ]
    [INFO] Deleting links
    rm: no se puede borrar «/etc/cygnus/flume.conf»: No existe el fichero o el directorio
    [INFO] Removing application log files
    [INFO] Deleting the cygnus folder
    [INFO] Deleting the cygnus user
    Done
    
    Updated:
      cygnus.x86_64 0:0.8.2-0.g7353160                                                                                                                            
    
    Complete!
    

    In order to check you have the right version, you can do several things. For instance, using the rpm command:

    $ rpm -qa | grep cygnus
    cygnus-0.8.2-0.g7353160.x86_64
    

    Or using the yum list:

    $ (sudo) yum list cygnus
    Loaded plugins: fastestmirror, security
    Loading mirror speeds from cached hostfile
     * base: sunsite.rediris.es
     * epel: ftp.cica.es
     * extras: sunsite.rediris.es
     * updates: sunsite.rediris.es
    Installed Packages
    cygnus.x86_64              0.8.2-0.g7353160                @Fiware
    

    Or simply check the oauth2_token parameter is there ;)

    $ cat /usr/cygnus/conf/agent.conf.template |grep oauth2_token
    cygnusagent.sinks.hdfs-sink.oauth2_token = xxxxxxxx
    

    IMPORTANT: Backup your Cygnus configurations since they will be still valid with minor changes: