Transform

Transformation allows you to transform the scraped configs before they are saved to config db.

JavaScript

You can supply a JavaScript code to transform the scraped configuration. Your JS code will have access to the special config variable which will contain the scraped config. Your script is expected to return a stringified JSON object which will be the new configuration.

Example: The following Config DB configuration specifies a transformation that'll add a new field "hello" with the value "world" to all the scraped configurations.

file-scraper.yaml

apiVersion: configs.flanksource.com/v1
kind: ScrapeConfig
metadata:
  name: file-scraper
spec:
  file:
    - type: Config
      id: $[0].id
      name: $[0].name
      transform:
        script:
          javascript: |+
            for (var i = 0; i < config.length; i++) {
              config[i].hello = "world"
            }
            JSON.stringify(config)
      paths:
        - fixtures/data/multiple-configs.json

CEL

You can also supply a CEL expression to transform the scraped configuration instead of the above JS. Your expression will have access to the special config variable which will contain the scraped config. Your script is expected to return a stringified JSON object which will be the new configuration.

Example: The following Config DB configuration specifies a transformation that'll add a new field "hello" with the value "world" to all the scraped configurations.

file-scraper.yaml

apiVersion: configs.flanksource.com/v1
kind: ScrapeConfig
metadata:
  name: file-scraper
spec:
  file:
    - type: Config
      id: $[0].id
      name: $[0].name
      transform:
        script:
          expr:
            config.map(e,e.hello = "world").toJSON()
      paths:
        - fixtures/data/multiple-configs.json

Considering that the fixtures/data/multiple-configs.json file contains the following configuration

[
  {
    "name": "Config1",
    "id": 1,
    "password": "p1",
    "secret": "secret_1"
  },
  {
    "name": "Config2",
    "id": 2,
    "password": "p2",
    "secret": "secret_2"
  }
]

The JS or CEL transformation will result in two new config items

{"id": 1, "name": "Config1", "hello": "world", "secret": "secret_1", "password": "p1"}
{"id": 2, "name": "Config2", "hello": "world", "secret": "secret_2", "password": "p2"}

Go Templates

Go template is another powerful way to transform the scraped configuration. Just as you provide a javascript code, you can also provide a Go template. The Go template will have access to the special config variable which will contain the scraped config.

file-scraper.yaml

apiVersion: configs.flanksource.com/v1
kind: ScrapeConfig
metadata:
  name: file-scraper
spec:
  file:
    - type: Config
      id: '$.id'
      name: 'scraped'
      transform:
        script:
          gotemplate: |+
            [{
              {{range .config}}
                "name-{{.id}}": "hi {{.name}}",
              {{end}}
              "id": "what"
            }]
      paths:
        - fixtures/data/multiple-configs.json

The above transformation will result in the following config

{ "id": "what", "name-1": "hi Config1", "name-2": "hi Config2" }