A beginner-friendly intro to the Correlator for effective cybersecurity detection

At TeskaLabs, we know that a cybersecurity system is only as effective as its ability to detect threats. That's why we developed a powerful tool that will prove essential in your arsenal: the Correlator.

We developed the Correlator as a module of our product TeskaLabs LogMan.io, but it can be adapted for other systems too.

Why do we need the Correlator?

In a system's sea of log data, some events are completely benign, but others are inevitably part of suspicious activity and cyber threats. However, a single event might not be enough to tell us that we're experiencing an attack or security breach. Often, it takes a series of events over a period of time to show a pattern or combination that indicates malicious activity. A single event, such as a failed login attempt, could be perfectly innocent — a user simply mistyped their password. However, many failed login attempts in quick succession could mean that someone is trying to log in to an account that's not theirs.

For effective cybersecurity, you need to know what patterns, anomalies, and threats to look for, and which individual events are relevant to those activities. Your system needs to be able to sort through the huge volume of information, thousands of logs every second, and spot those patterns. This process is the Correlator's job.

What problems does the Correlator solve?

The Correlator satisfies two crucial requirements: The ability to perform detections in real time, and the capacity to deal with huge volumes of data in a growing system. The Correlator achieves both of these things by operating in the data stream, primarily in your system's quick-access memory (RAM) rather than working with a database in your system's storage.

If you're not familiar with those concepts, here's the idea: Working in-stream versus using a database is like the difference between working with a huge whiteboard and analytical superpowers versus working with books in a library. Imagine if every system event was recorded instantly on a big whiteboard, and you could immediately see and analyze each event as it appeared, drawing conclusions quickly. What if instead, every event was written in a book, and that book was put away on a shelf after every entry? You'd have to find and read through the book every time you wanted to gather information. This would take so much longer that by the time you could analyze anything had happened, it would be too late. It would also take a lot more of your energy, so you wouldn't be able to analyze as much information overall.

What does that mean in cybersecurity, and why does it matter?

1. Detecting in real time

First, we'll talk about this idea of real time. Sure, we could run detection rules through databases of old logs to correlate past events, like reading through books in a library. However, this method is simply not fast enough. By the time we would've found a problem, the system would have already suffered any attacks or threats that we found. You can understand that in today's fast-paced environment, this method is no longer practical.

That's why the Correlator works in real time. Operating in the data stream means that our component tracks events as they happen and analyzes them right away, gathering and organizing data continuously. Another way of describing this contrast between methods is "first store, then analyze" versus "first analyze, then store." The second method, analyzing first when events come in, then storing data, rather than analyzing data stored in the past, is what the Correlator does. It's instantaneous, and any follow-up activity, such as an alert or notification, happens as soon as the threat is detected.

2. Detecting on a large scale

Second, we'll talk scale, meaning the size of a system and amount of data it has. If you're spending all of your energy combing through those books in the library, you can't do very many things at once.

It's the same in a cybersecurity situation; accessing and analyzing all that information from a database would involve a huge amount of processing power, but operating in the data stream is more flexbile and allows many detections to happen simultaneously over enormous quantities of data. Plus, everything the Correlator does still gets backed up to your system's storage disk, so you won't lose data in the event of an unexpected system shutdown.

How does the Correlator work?

Now you know why we need the Correlator, and what's special about it. But how does it work its magic? For that, let's venture outside the realm of cybersecurity.

1. Keeping track of what's happening

Imagine that a new food delivery app is doing a soft launch. As this delivery service is still in its early stages, it's launching in just four neighborhoods, or delivery zones, in a city. The app's success depends on efficiently managing orders, delivery times, and customer satisfaction.

The logistics manager of the app's team wants to know more about each delivery zone, so she decides to track every order placed by zone and by time. (Thanks to this small-scale launch, this task is still possible to do manually.) Every time an order comes in, the manager makes a mark in a table, where each delivery zone has its own row, and each timeframe has its own column.

Let's say an order is placed for delivery zone C once at 18:35, and once at 18:38. The logistics manager will go the C row and the 18:30-18:40 column and make two marks.

How does this translate to cybersecurity?

The orders being placed to the delivery app are like events, or logs, in a cybersecurity system. Just like each order includes much more information than just which delivery zone it's going to (such as which restaurant is being ordered from and what customer is placing the order), events in a cybersecurity system contain lots of information, some which is only relevant depending on what we're trying to accomplish. The Correlator, just like the logistics manager, can focus on a specific characteristic of an event, and sort events by that characteristc while keeping track of exactly when the events happen.

Now that we've covered how events are tracked, let's take it a step further.

2. Analyzing what's happening

If we can track events and when they happen, what else can we do?

The delivery app has gotten feedback that some orders are taking too long to deliver. The problem is that the app's logistics manager doesn't know how many couriers to allocate to each delivery zone. She decides to set a benchmark. If any delivery zone gets 10 orders or more in 30 minutes, she'll send an extra courier to that zone.

How does the manager count up the events? She looks at her table of timeframes and counts the orders 30 minutes at a time by adding up the totals of 3 consecutive 10-minute timeframes. You can imagine that she has an "analysis window" that spreads across three timeframes at a time. She moves the across the rows, and she can continue adding new orders to the table simultaneously.

When the manager finds 10 orders in a 30-minute timeframe, she takes action. You can see here that she's found 2 orders for Zone 3 between 18:30 and 18:40, 3 orders between 18:40 and 18:50, and 6 orders between 18:50 and 19:00. Thus, she'll allocate an additional courier to make up for the increased order traffic in Zone 3.

How does this translate to cybersecurity?

• The orders
  The orders of food are logs. Like a cybersecurity system, the delivery app has many things happening at once. Each order of food doesn't mean much individually, but a combination of orders provides information. Just like a log, each order has more information than what dish was ordered and at what time (for example, the restaurant, the food, the price, payment details), but only some information is relevant to what we want to know.

• The logistics manager
  The manager and is like the Correlator. The manager collects and organizes the information. Just like the manager, the Correlator tracks and analyzes events in a table.

• The conditions
  10 orders to a single delivery sone in 30 minutes is the condition the manager is looking for. In cybersecurity, conditions are set by detection rules, which define the parameters of the events and actions that the Correlator tracks and searches for. We can set infinite combinations of conditions, from very simple to very complex.

• Actions taken
  The action the manager takes once she finds that a single delivery zone has received 10 orders in 30 minutes is like the trigger that happens once the threat or suspicious activity is detected in a cybersecurity situation. This action is defined in detection rules. A triggered action could be an alert, a notification, or the creation of a new log that describes the detected behavior.

Examples in cybersecurity

Are you getting the hang of it? Let's dive in to a simple example in a cybersecurity environment. We mentioned login attempts earlier, and that's a great place to start. Many failed login attempts in a short time could mean that someone is trying to access a user account that isn't theirs, because they're guessing the password over and over, possibly using a specialized software to do so. This type of incident is called a brute force attack.

A detection rule against a brute force attack like this could set these conditions: If a single user account experiences 20 failed login attempts within a 5 minute timeframe, then the user will be denied access to their account for 10 minutes, and the system administrator will receive an email about the incident.

For this detection, the Correlator will:

• Keep track only of events that describe a failed login attempt
• Analyze 5 1-minute timeframes at once by adding up the total failed login attempts per user account
• Trigger the resulting action (access denial and adminstrator email) if behavior is detected

Here's what that table looks like:

You can see in the table that the Correlator detected 20 failed login attempts for the third user between 01:00 and 06:00.

Here are a few more situations that would use the same counting and time window method:

Data download volume: Downloading too much data in a short period of time could indicate a cybersecurity event called data exfiltration, or someone trying to steal large amounts of personal information from a system. The conditions to prevent data exfiltration could be: A single user downloads more than 10GB of data in a 12-hour period.

Counting emails sent: Sending emails can be an important indicator of user activity. It's not typical, for example, for most users to send hundreds of emails in a short period of time (unless that's specifically part of their job role). An unusually high volume of emails sent could indicate that someone's email address has been compromised and is being used to send spam. The conditions to detect potential email account compromise: A single email address sends 100 individual emails to separate recipients within 5 minutes.

More than counting

The examples we've covered so far only use the Correlator's "sum" capability, or its ability to count and add events together (20 failed login attempts, 10GB of data, 100 emails). However, the Correlator can use other mathematical functions too, such as calculating the median, average, standard deviation, variance, mean spike, and more.

When might we use a different aggregator function? While using the sum is the most common in detection rules, we might want to use another method when the context requires understanding patterns or behaviors that are not simply about amount. For example, if a system keeps track of the average rate at which new accounts are created on a website, then a detection rule could set conditions to inform an administrator if the account creation rate average goes above a certain level over the course of the whole day or week, and warn against potential automation or fraud attacks. Since this type of approach allows for more nuance, focusing not just on quantity but on deviation and more complex patterns, we can detect more sophisticated threats and get deeper insight into system activity.

How can you get the Correlator?

Now that you've seen how the Correlator can detect patterns, anomalies, and threats, are you ready to level up your detection power? If you already have a cybersecurity system, the Correlator can be integrated into your processes as an independent component. Or, if you're on the lookout for your first cybersecurity solution, TeskaLabs LogMan.io offers excellent data exploration, intuitive charts and graphs, and detections with the Correlator all in a user-friendly web app.

Whether you'd like to use our library of pre-written detection rules or craft customized solutions tailored to your unique challenges, we're here to support you on your journey towards a more secure, insightful cybersecurity posture.

If you're interested in the Correlator or LogMan.io, contact us! We'll be happy to help you get started.