Skip to main content

Automation

Maintaining a sound cloud infrastructure includes tasks that need to be performed regularly. Fix Inventory can help you automate many of these tasks by defining jobs.

Examples of these tasks include:

  • enforcing company policies (e.g., tagging),
  • sending notifications about critical infrastructure changes (e.g., to Slack, Discord, or PagerDuty), and
  • cleaning up unused resources in your development accounts.

Jobs

A job is a sequence of commands executed by Fix Inventory at a specific time interval or when a specific event is emitted.

Jobs in Fix Inventory are similar to Unix cron jobs. However, while cron jobs run only on a specific schedule, Fix Inventory jobs can also be triggered based by events.

Job Overview

The typical structure of a job has three parts: a trigger, an optional filter, and an action.

Job Trigger

The job trigger defines when a job is executed.

There are three types of job triggers: schedule triggers, event triggers, and combined triggers.

Schedule Trigger

A schedule trigger executes a job at a specific time interval described by a cron expression.

Examples
  • Every 5 minutes
    --schedule "*/5 * * * *"
  • Every day at 3:00am
    --schedule "0 3 * * *"
  • Every Monday at 4:00am
    --schedule "0 4 * * MON"
  • Every New Year's Eve at 11:59pm
    --schedule "59 23 31 12 *"
tip

See How to Create a Scheduled Job for step-by-step instructions describing how to create a job with a schedule trigger.

Event Trigger

An event trigger executes a job when a specific event is emitted by Fix Inventory.

Fix Inventory updates the state of resources in the four steps of the collect_and_cleanup workflow: collect, cleanup_plan, cleanup, and generate_metrics.

Each of these steps emits events that can be used to trigger jobs.

Examples
  • When resource collection is complete and the database reflects the latest state of your resources
    --wait-for-event collect_done
  • When the cleanup planning process has started to mark resources for cleanup
    --wait-for-event cleanup_plan
info

See Events for a list of events emitted by Fix Inventory.

tip

See How to Create an Event-Based Job for step-by-step instructions describing how to create a job with an event trigger.

Combined Trigger

A combined trigger combines a schedule trigger with an event trigger, and executes a job when a specific event is emitted after a schedule trigger is fired.

Combined triggers are useful if you want to perform an action on a specific schedule, but only after a specific event is fired.

Example

Let's say you want to clean up development accounts at the end of each week.

To do so, you could define a schedule trigger in addition to specifying the cleanup_plan event trigger:

--schedule "0 22 * * FRI" --wait-for-event cleanup_plan

The above combined trigger executes the job based on the latest state of resources after 10pm every Friday evening.

tip

See How to Create a Scheduled Event-Based Job for step-by-step instructions describing how to create a job with a combined trigger.

Job Filters & Actions

The scope and functionality of a job is defined by filters and actions, respectively.

To perform more complex automations, it is possible to define multiple sets of filters and actions delimited by a semicolon. Each set of filters and actions is executed sequentially when the job is triggered.

Job Filters

A job filter is an optional search that returns the resources of interest. Only resources matching the filter will be processed by the job.

Example
  • Instances without an owner tag in the dev or playground accounts
    search is(instance) and tag.owner==null and /ancestors.account.reported.name in ["dev", "playground"]

The result of a job filter is a list of resources that is piped as input to job actions.

Job Actions

A job action is an action command performed on resources matching the optional filter.

Examples
  • Update resource tags using the tag update command

    search ... | tag update owner=team-cumulus
  • Delete resource tags using the tag delete command

    search ... | tag delete costcenter
  • Mark resources for cleanup using the clean command

    search ... | clean "Reason for cleanup"
  • Create PagerDuty alerts using the pagerduty custom command

    search ... | pagerduty summary="Reason for the alert" dedup_key="xyz"
  • Send Prometheus Alertmanager alerts using the alertmanager custom command

    search ... | alertmanager name="Description of the alert"
  • Send Slack notifications the slack custom command

    search ... | slack title="Description of the alert"
  • Send Discord notifications the discord custom command

    search ... | discord title="Description of the alert"
  • Stop running AWS EC2 instances using the aws command

    search is(aws_ec2_instance) and instance_status=running and ... | aws ec2 stop-instances --instance-ids {id}
  • Start stopped AWS EC2 instances using the aws command

    search is(aws_ec2_instance) and instance_status=stopped and ... | aws ec2 start-instances --instance-ids {id}
  • Send chunked data to a webhook server using the http command

    search ... | chunk 50 | http POST my.node.org/handle
  • Protect resources from cleanup using the protect command

    search ... | protect
  • Edit resource metadata using the set_metadata command

    search ... | set_metadata owner=team-cumulus

Job Management

Jobs are persisted in the Fix Inventory database. The job command allows you to add, update, delete, and list jobs.

tip

See Related How-To Guides below for step-by-step instructions to create jobs.

Examples
  • Add a job that sends a Discord notification when compute instances without an owner tag are detected

    > jobs add --id no_owner_tag_in_dev --wait-for-event collect_done 'search is(instance) and tag.owner==null and /ancestors.account.reported.name in ["dev", "playground"] | discord --title "[DEV] Compute instances without owner tag" --message "There are compute instances in the dev accounts without owner tag:"'
  • Add a job that marks compute instances without an owner tag for cleanup every Friday evening

    > jobs add --id no_owner_mark_cleanup --schedule "0 22 * * FRI" --wait-for-event cleanup_plan 'search is(instance) and tag.owner==null and /ancestors.account.reported.name in ["dev", "playground"] | clean "Compute instances without owner tag"'
  • List all jobs

    > jobs list
    ​id: no_owner_tag_in_dev
    ​command: search is(instance) and tag.owner==null and /ancestors.account.reported.name
    ​ in ["dev", "playground"] | discord --title "[DEV] Compute instances without owner tag"
    ​ --message "There are compute instances in the dev accounts without owner tag:"
    ​active: true
    ​trigger:
    ​ message_type: collect_done

    ​---
    ​id: no_owner_mark_cleanup
    ​command: search is(instance) and tag.owner==null and /ancestors.account.reported.name
    ​ in ["dev", "playground"] | clean "Compute instances without owner tag"
    ​active: true
    ​trigger:
    ​ cron_expression: 0 22 * * FRI
    ​wait:
    ​ message_type: cleanup_plan

Further Reading