Chef datadog wrapper cookbook won't run with "missing api key" error

I created a wrapper cookbook to retrieve my datadog api keys from an encrypted data bag but it looks like it is not running during the execution.

Here is my code:

attributes/default.rb

node.default['datadog']['encrypted_data_bag'] = 'datadog'
node.default['datadog']['encrypted_data_bag_item'] = 'datadog_keys'

recipes/set_key.rb:

node.default['datadog']['api_key'] = data_bag_item(node['datadog']['encrypted_data_bag'], node['datadog']['encrypted_data_bag_item'])['api_key']
node.default['datadog']['application_key'] = data_bag_item(node['datadog']['encrypted_data_bag'], node['datadog']['encrypted_data_bag_item'])['chef']

and del_key:

node.rm['datadog']['api_key']
node.rm['datadog']['application_key']

I created a role named datadog and run list of this role looks like:

datadog-wrapper-0.1.0::set_key
datadog::dd-agent
datadog::dd-handler
datadog-wrapper-0.1.0::del_key

I'm expecting this wrapper recipe load datadog keys, then datadog recipes to run and finally another wrapper recipe to remove keys. But when Chef is running, I receive an error message like:

ArgumentError
-------------
chef_handler[Chef::Handler::Datadog] (datadog::dd-handler line 52) had an error: ArgumentError: Missing Datadog Api Key

Since I'm new to Chef and data bags use, I'm a bit confused. Why my setter recipe is not running?

Thanks.

Solution

As I have mentioned in the comment, you are affected by two pass model. You should remove the keys in the resource added to the end of the chef run or triggered by the DD cookbook resources invoked as the last one in the run.

ruby_block "clean datadog api attributes" do
  block do
    node.rm("datadog", "api_key")
    ....
  end
  subscribes :create, "template[<some dd template using api keys>]", :immediately
end

However, it may not work with all versions of DD cookbook. From few DD cookbook versions, it is possible to store keys in node's run state which is not written to the Chef server.

node.run_state["datadog"] = {
  "api_key"         => datadog["api_key"],
  "application_key" => datadog["application_key"]
}

The above example is preferred solution to your issue.