Skip to main content

Collector Plugin Development

Plugin source code lives in the plugins directory within the someengineering/fixinventory GitHub repository. Each plugin is maintained as separate project.

Contributions are made via pull requests to the GitHub repository. You will first need to fork the repository.

Pull requests should target a single plugin. Please refer to Contributing to Components for details on how to clone the repository, set up a virtual environment, start Fix Inventory, and submit your changes for review.

Collector Plugins​

Collector plugins allow for importing of arbitrary resources into Fix Inventory in graph form. The most common use case is to gather information about cloud accounts and/or resources. However, any data expressible is graph form can be collected—be it social media accounts, software dependency trees, network topology, steps for cooking your favorite food, etc.

Once the graph data is collected and sent to Fix Inventory Core, the power of Fix Inventory's search is at your fingertips.

Plugin Interface​

The collector plugin interface is defined as follows:

def collect(self) -> None:
  """Collects all the Cloud Resources"""
  account_graph = collect_account()
  self.graph.merge(account_graph)

The typical flow of the collect method is:

  1. An instance of a Graph class is created and filled with cloud resources.

  2. The graph is merged with the plugin's internal graph.

Resource Graph Structure​

Usually, the graph is structured as follows:

  1. The top-level node is the cloud itself. It is added by the collector plugin automatically.

  2. The next level is the account (e.g., an AWS account, Google Cloud project, DigitalOcean team, etc.).

  3. Where applicable, the following level is the region (e.g., US-East-1, US-West-1, etc.).

  4. The remaining levels are cloud-specific resources.

Merging Collected Resources​

Fix Inventory uses a thin wrapper on top of NetworkX to operate on graphs. The two most used methods in the wrapper are add_resource and add_edge.

Keep in mind that there are two types of edges, default and delete.

  • default edges represent how resources relate to each other (e.g., a resource may be represented by a child node of a region node).

  • delete edges define the order in which resources should be cleaned up (e.g., a specific instance needs to be deleted before a particular volume can be deleted).

Please refer to the example collector plugin for a model of how to implement this logic.

Testing​

To test a collector plugin, simply launch Fix Inventory and trigger the collect action manually by executing workflow run collect in Fix Inventory Shell.

Once resource collection is complete, you can execute search (<plugin_resource_type>) to see the newly collected resources.

Tips​

  • Resource properties types must be globally unique.

    For example, if there is a property status: str defined by some other plugin and you add a property status: int, the collection will fail. The reason for such behavior is that all resource properties are indexed for search.

  • Try to not introduce unique property names.

    Only do it in event that types are not compatible with existing properties. For example, use status: str instead of plugin_name_resource_name_status: int.

  • Don't forget to call sanitize(graph) before checking its properties in tests.

    Please refer to the DigitalOcean plugin as an example.