Cyber Crimes and The Confusion Matrix
Machine learning is an application of artificial intelligence (AI) that provides systems the ability to automatically learn and improve from experience without being explicitly programmed. Machine learning focuses on the development of computer programs that can access data and use it to learn for themselves.
What is a confusion matrix and why do we need it?
When it comes to Machine Learning, performance measurement has always been one of the most essential means of how capable your machine learning model actually is. There are many methods of performance measurement, however when it comes to the classification type of problem in machine learning, the method of performance measurement through the use of a confusion matrix is the one of the best their is.
The performance measurement may also be referred to as the accuracy score of the machine learning model.
Suppose the dataset you used to test the performance of your model has ’N’ target classes, ( An example of a class can be the result(target) of student ie, pass or fail — two classes) then a confusion matrix is an N x N matrix used for evaluating the performance of a classification model. The matrix compares the actual target values to those predicted by the machine learning model.
The confusion matrix is a big topic altogether however, for today, we are only going to discuss what the confusion matrix actually means and what is it’s purpose.
This matrix makes it easy to understand whether or not the system is confusing the actual values and the predicted values, and thus the name Confusion matrix.
What is Cyber Crime?
Data is considered a commodity online and offline by both legal and illegal actors (Maras, 2016). For this reason, data is a primary target of cyber criminals.
Data plays an integral role in the commission of many cyber crimes, primarily because it is not adequately protected and can be illicitly accessed and obtained.
Cyber crime violates individuals’ privacy and the security of their data, particularly hacking, malware, identity theft, financial fraud, medical fraud, and certain offenses against persons that involve the revealing of personal information, messages, images, and video and audio recordings without individuals’ consent or permission (e.g., cyber stalking, cyber harassment, and cyber bullying).
Cyber crimes and confusion matrices
To better understand what a confusion matrix means lets take an example keeping it as simple as possible :
As for today’s discussion we are looking into how confusion matrices and cyber crime are related so lets keep our example based on cyber security ( cyber attack detection ). Also cyber attack detection is a binary classification problem means it will have only two target classes i.e. a cyber attack occurred and no cyber attack occurred.
Suppose you are working for a security operations organization or company. Your team’s primary objective is to detect and stop any kind of hacker attack.
For this task your team deploys a Intrusion Detection System (IDS), a program purely based on machine learning. An intrusion detection system is a monitor-only application designed to identify and report on anomalies before hackers can damage your network infrastructure.
Lets say the IDS system made 200 predictions in total. And the system reports that out of these 200 predictions, 80 were possible hacker attacks, and the rest 120 were not.
Let’s make the following definitions:
- “A hacker attack happening” is a positive class.
- “A hacker attack not happening” is a negative class.
After a proper analysis of results manually to verify the results obtained, four different types of values were extracted from the model:
- True Positive (TP): The number of times our actual positive values are equal to the predicted positive. You predicted a positive value, and it is correct.
- False Positive (FP): The number of times our model wrongly predicts negative values as positives. You predicted a negative value, and it is actually positive.
- True Negative (TN): The number of times our actual negative values are equal to predicted negative values. You predicted a negative value, and it is actually negative.
- False Negative (FN): The number of times our model wrongly predicts negative values as positives. You predicted a negative value, and it is actually positive.
Based on the analysis a 2 x 2 matrix was created in a certain pattern. This is the confusion matrix.
Confusion matrices have two types of errors: Type I and Type II:
- Type I Error:
Type I error refers to the False Positive error (FP). To better understand this, lets observe carefully. This is type of error is not as much as dangerous as the Type II error, but can be pretty troublesome. A false positive error occurs when the model predicted a negative value, and it is actually positive.
In our case, A false positive state is when the IDS identifies an activity as an attack but the activity is acceptable behavior. A false positive is a false alarm. This type of error always leads to the FATAL cases of data breaches, malware attacks, and many other types of cyber attacks.
2. Type II Error:
The Type II error refers to the False Negative error (FN). Out of the two types of errors this type of error is the most dangerous to have.
A false negative state is the most serious and dangerous state. This is when the IDS identifies an activity as acceptable when the activity is actually an attack. That is, a false negative is when the IDS fails to catch an attack. This is the most dangerous state since the security professional has no idea that an attack took place. This type of error always leads to the FATAL cases of data breaches, malware attacks, and many other types of cyber attacks.
False positives are an inconvenience at best and can cause significant issues. However, with the right amount of overhead, false positives can be successfully adjudicated; false negatives cannot.
How to use the confusion matrix?
Use of certain metrics is the one of the best ways to use a confusion matrix.
Confusion Metrics
From our confusion matrix, we can calculate five different metrics measuring the validity of our model. Obtaining metrics can be a crucial part in various types of observations.
- Accuracy (all correct / all) = (TP + TN) /( TP + TN + FP + FN)
- Misclassification (all incorrect / all) = (FP + FN )/( TP + TN + FP + FN)
- Precision (true positives / predicted positives) = (TP) /( TP + FP)
- Sensitivity aka Recall (true positives / all actual positives) = (TP) / (TP + FN)
- Specificity (true negatives / all actual negatives) =(TN) / (TN + FP)
A few references of confusion matrix in cyber crime:
“ Close to half of security analyst teams battle false positive rates of 50% or higher from their security tooling. Meantime, another report from the Ponemon Institute shows that as much as 25% of a security analyst’s time is spent chasing false positives — sifting through erroneous security alerts or false indicators of confidence — before being able to tackle real findings.
That means that every hour an analyst spends on the job, they’re wasting 15 minutes on false positives. On average, the typical organization wastes anywhere between 424 hours and 286 hours per week on false positives. “
In spite of the representational power of the confusion matrix in classification, it is not a very useful tool for the sake of comparison of the IDSs. To solve this problem, different performance metrics are defined in terms of the confusion matrix variables. These metrics produce some numeric values that are easily comparable.
Thus, in order to evaluate the effectiveness of an IDS, we need to measure its ability to correctly classify events as normal or intrusive along with other performance objectives, such as the economy in resource usage, resilience to stress and ability to resist attacks directed at the IDS.
Measuring these abilities o f IDS is important to both industry as well as the research community. It helps us to tune t h e IDS in a better way as well as compare different IDSs. As discussed above, there exist many metrics that measure different aspects of IDS, but n o single metric seems sufficient to measure the capability of the IDSs objectively. As per statistics of a survey c o n d u c t e d by ( Tavallaee, 2011), the most widely used metrics b y the intrusion detection research community are True Positive Rate (TPR) and False Positive Rate (FPR) along with the ROC.
Originated from signal detection theory (Tavallaee, 2011), ROC curves are used on the one hand to visualize the relation between detection rate and false positive rate of a classifier while tuning it, and on the other hand to compare the accuracy of several classifiers. Although this measure i s very effective, it has some limitations. The first limitation is that it is dependent on the ratio of attacks to normal traffic. The comparison of various classifiers based upon ROC works fine for the s a me dataset. However, the comparison of the IDSs done on various data sets is completely wrong, unless they have the same ratio of attack to normal instances. The second problem with ROC curves is that they might be misleading and simply incomplete for understanding the strengths and weaknesses of the candidate system.
Sometimes it is difficult to determine which IDS is better than another in terms of only FPR and TPR. For example, IDS1 can detect 10% more attacks, but IDS2 can produce 10% lower false alarms. Which one is better? In order to solve the problem, (Gu et al., 2006) suggested a single unified objective metric called intrusion detection capability (CID) based upon base rate, positive predictive value, or Bayesian detection rate (PPV) and negative predictive value (NPV). Such metric is used to select the best IDS configuration for an operational environment and to evaluate different IDSs.
Thank you!!!
How can you combat false positives and negatives?
There are several approaches to consider when it comes to reducing the number of bogus security threats, including network analysis, enacting policies that reduce the opportunity for cyber attacks, beefing up your overall security measures, and taking a look at how modern AI technology could help.
Analyze Network Traffic
Look through information in your network logs to spot unfamiliar usernames, odd connection details, and suspicious trends in the duration and frequency of communication to uncover security threats the old-fashioned way. You may be able to detect more false negatives than if you hadn’t looked, but this process is prone to suffer from human error and can become quite time-consuming.
Limit Network Access on IoT Devices
As a matter of protocol, consider implementing a policy that limits the network access of IoT devices. When IoT devices are given restricted network access, your security software is more likely to recognize unusual behavior and should issue more accurate alerts.
Use Web Application Firewalls
A large percentage of data breaches are targeted at web application vulnerabilities. While the commonly-deployed Web Application Firewall can reduce these instances, this type of firewall can hog network resources when used to detect false negatives and positives.
Research Artificial Intelligence Solutions
Reducing the number of false positives from the start can save you needless headache. Equipping your network security team with a tool that will help them better analyze emerging trends and widespread security threats will leave your company less vulnerable to actual security breaches.
Conclusion
Understanding the differences between false positives and false negatives, and how they’re related to cybersecurity is important for anyone working in information security.
The existence of both false positives and false negatives begs the question: Does your cybersecurity strategy include proactive measures? Most security programs rely on preventative and reactive components— establishing strong defenses against the attacks those tools know exist. On the other hand, proactive security measures include implementing incident response policies and procedures and proactively hunting for hidden/unknown attacks.
