How to read specific pattern in a file

I am very new to Terraform world and sorry for asking the basic question. I googled a lot but could not find how to proceed.

My task is to read an external file from a terraform module. And after reading the file, I'll have to check for specific pattern in each line. So for this I have written like this -

main.tf

 resource "null_resource" "demo_null_res" {
   provisioner "local-exec" {
       command = "./test.sh"
    }}

test.sh

  #!/bin/bash
 while read -r line; do
   if [[ $line == *"ERROR"* ]]; then
    rr=$(echo $line | cut -d ':' -f 4)
    echo $rr
    first=$(echo $rr | cut -d ' ' -f 1)
    sec=$(echo $rr | cut -d ' ' -f 2)
    th=$(echo $rr | cut -d ' ' -f 3)
    echo ${first/[a-z]*/''} >> output.txt
  fi
done < input.txt

But problem with this approach is that, any changes to input.txt file will not be captured by terraform because it thinks that terraform state has not changed.

Another way I am thinking is that, maybe I need to create a custom provider and resource type but I am not sure this is the right way.

Can you please suggest how to proceed further?

If this file is a static part of your configuration directory (that is: the file will already be there before you run any Terraform commands) then you can read it using the file function. Because that's a function you can use it anywhere that Terraform expects general expressions, but for the sake of example I'll show it as a local value:

locals {
  file_content = file("${path.module}/input.txt")
}

With the above local value definition, you can use local.file_content elsewhere in your module to obtain a string representing the file contents.

If the file is instead something generated by other work done in your Terraform configuration then you'll need to model it as a data block so that Terraform can understand when is the appropriate time to read the file in relation to other actions it'll be taking. The local_file data source from the hashicorp/local provider is a typical choice for reading a file from disk during runtime:

data "local_file" "example" {
  filename = "${path.module}/input.txt"
}

In this case, you'd use data.local_file.example.content to access the file contents.

Once you have the file contents as a string you can parse it using other Terraform functions and language operators.

In your case I think the building blocks you would need are:

• The split function to split the single string into a list of strings using "\n" as the delimiter.
• A for expression to filter out the lines that don't contain the substring "ERROR".
  • You can use regex or regexall to match strings.
• More calls to split to create a similar effect as your cut calls, to split the individual lines on particular delimiters.
• The replace function to replace substrings of a string with other substrings.
• String templates to reassemble your separated parts back into a single string in the format you need for your output.

The Terraform language is not really optimized for this sort of arbitrary parsing and computation and so implementing something like this is inevitably a little clunky, but I think Terraform has equivalent building blocks to all of the shell features you used in your example.

I'm not sure I fully follow what the goal is of your shell script but here's a partial example that will hopefully be a reasonable starting point that you can adapt:

locals {
  # NOTE: This is assuming the local.file_content declared
  # in one of my earlier examples. You could use
  # data.local_file.example.content instead if you used
  # the data block example.
  file_lines = split("\n", local.file_content)

  lines_with_error = tolist([
    for line in local.file_lines : line
    if length(regexall("ERROR", line)) > 0
  ])
  lines_split_colon = tolist([
    for line in local.lines_with_error : split(":", line)
  ])
  lines_rr = tolist([
    for line in local.lines_split_colon : line[3]
    if length(line) >= 4
  ])
  lines_rr_split_space = tolist([
    for line in local.lines_rr : split(" ", line)
  ])
  lines_letters_removed = tolist([
    for line in local.lines_rr_split_space :
    replace(line[0], "/[a-z]+/", "")
    if length(line) >= 1
  ])
}

Notice that one key difference in approach compared to shell scripting is that in Terraform we typically transform the entire list at once each time rather than visiting each row and working on that row in isolation across multiple steps. This is because the Terraform language is closer to a functional programming approach than imperative, and it's the data relationships which imply the order of operations, rather than instructions being evaluated in source-code order.