Log4j Lesson Learned: Inventory Everything

Introduction

The Log4j vulnerability has taught us a number of lessons. In this series of Log4j related posts, I’m going to outline some lessons learned along with what we can do to be prepared for the next such vulnerability. The first lesson is that we need an inventory of java assets at our fingertips so that we can quickly find the next java vulnerability in our enterprise.

In my previous post on the topic, I went over some background information about what Log4j is and what happened. Please take a look.

What about everything else besides Java?

In my last article, I spoke about how we can inventory and index our java assets in Splunk or something like it.  Simply walk the file system looking for jars and other java related assets (war files, etc.), then record all the metadata we can about each file (name, size, checksum, etc).  But, wait a minute, if we are walking the file system, why only record the metadata about java, why not everything?  Certainly this will be more expensive computationally to take all of those checksums and from a storage perspective it will have some cost.  To solve the computational problem, we can take these inventories during non-peak hours for those end points.  Further, with some cooperation from our end-point protectors, we can pull the data from them, they are doing it anyway.   Busier, server resources can be managed by working from file system snapshots or use other standard techniques to alleviate the I/O and other resource burdens.

Then we can ask Splunk about all of our file system assets: 

“Which file systems/hosts have a filename/checksum/what-have-you“

endpoint protectors can’t answer this question at time zero + 30 seconds. 
Splunk can.

Not only would having this data in your index help you to know if the next java or other vulnerability exists in your enterprise, it can also help you answer other, important IT questions (like does a particular piece of software exist in my enterprise).

How to make this happen. 

Late last year I wrote a short shell script that demonstrates one way to inventory your java assets (Log4JFinder) the output of which can be fed to Splunk.   Once the information is in Splunk, we can use simple search techniques to locate nodes that have the vulnerability, then take appropriate action.  Shortly I will be extending this to inventory everything and reposting.  Stay tuned.

With information like that at your fingertips, decisions can be made and immediate action taken.

Walking file systems can be resource expensive, especially in a busy enterprise.  Of course we can mitigate these problems using the tools we have at our disposal.  Use cron to schedule the search during off hours.  Segment the file system walk and only perform a subset each night.  Make file system snapshots and walk those.  You know what to do.

Search your inventory with Splunk

Now we can, in a single search, identify all instances of almost any foe in our enterprise with a single search in Splunk, in seconds… 

Immediately Locate Your Java Assets 

Late last year I wrote Log4jFinder, a shell script that demonstrates one way to locate your java asset. Read below to understand how to use this tool in conjunction with Splunk to instantly locate all instances of a java jar file in your enterprise.