Microservice your IT monitoring toolset with Kafka

Apache Kafka is well known to be used in modern application architectures, it’s also being used to re-architect the toolset used by IT teams. I explain how…

IT monitoring teams have had to balance the risk and costs between taking enterprise software and using open source technologies. The fear with enterprise software is with being “locked-in” (especially if there are agents to install). The concerns with open source software are around complexity and development effort.

No alt text provided for this image

One size does not fit all. In an ideal world, operations teams should have a mixture of open source and enterprise tools to best meet requirements, provide best overall value and ultimately improve quality of services.

Doing this has become much easier over recent years for a few reasons

  • SaaS and “the download-model” has lowered the barriers to entry of enterprise software and offers shorter-term commitment
  • Cloud services and CI/CD pipelines have forced vendors to mature their API integrations making it much easier to get data in and get data out
  • As organisations “shift left” with DevOps we see a “you buy it, you run it” approach rather than expecting centralised teams to manage the tool

Following in the footsteps of modern application architectures, decoupling components of your monitoring tools between data collection and storage/analysis through a message broker such as Apache Kafka makes a lot of sense. Microservicing your toolset if you like.

No alt text provided for this image

The benefits include:

  • Easier to select tools that best meet specific discrete team requirements
  • Flexibility and speed in integrating or replace tools
  • Fewer agents to deploy and maintain
  • Different teams can choose their preferred different tool for same requirement
  • Drive actions from correlated data on-the-wire (such as an alert or an automated runbook)
  • ..more!

One good example is separating the tool that you use to monitor a service and the tool that you want to use to investigate (incident) since the best tools for each function aren’t necessarily the same product and much of the same data is needed in both solutions.

Are you designing in this way? We’d love to see your comments or contact us

Keep informed of our future blogs by following us on LinkedIn here.