Over the 3 years period , I’ve created numerous articles on forensic tools and incident response (IR). This time, I want to take a step back and focus on how to handle an incident investigation.
This guide will specifically highlight incident response workflows using Velociraptor and KAPE. If you're looking for the forensic part of investigations, check out my other articles—or let me know, and I’ll create one soon!
For those unfamiliar, I’ve written a series of articles diving deep into Velociraptor: from configuration to advanced usage. You can find those articles
Now, let’s dive into incident response without overcomplicating things
-------------------------------------------------------------------------------------------------------------
Why Velociraptor’s Labels Matter
One of Velociraptor's standout features is Labels, which play a critical role in investigations.
They help you categorize, organize, and quickly identify relevant endpoints.
While I can’t show you live client data due to privacy reasons, I'll provide detailed examples to help you understand the process.
Remember, this article assumes you’ve read my previous Velociraptor articles; they provide the foundational knowledge you'll need here.
-------------------------------------------------------------------------------------------------------------
Scenario: Investigating a Phishing Attack
Imagine you’re responding to an incident involving a client with 100 endpoints.
Attack Overview:
The client fell victim to a phishing email.
An attachment in the email was opened, initiating the attack.
The client has isolated the environment by cutting off external connectivity.
Their key questions are: (Before forensic)
How many users opened the attachment?
What files were created, and where?
Which endpoints were infected?
The client doesn’t have EDR or SIEM tools.
Yes, it’s not ideal, but in the real world, this happens more often than you’d think.
-------------------------------------------------------------------------------------------------------------
Deploying Velociraptor Agents
First, configure your Velociraptor server (refer to my previous articles for detailed steps). Provide the client with the necessary token and instructions to roll out Velociraptor using GPO (Group Policy Objects).
Key Points:
Velociraptor isn’t a typical agent-based EDR.
It doesn’t modify the system drastically, making it less intrusive and easier to handle.
Once the client deploys the agents across endpoints, all devices will begin appearing in your Velociraptor console.
-------------------------------------------------------------------------------------------------------------
Automating Labels for Large Environments
Let’s say the client has rolled out Velociraptor to 100 endpoints. Manually assigning labels to each endpoint is impractical. Instead, you can automate this process:
Click the eye icon in Velociraptor.
Search for Server.Monitor.Autolabeling.Clients.
Launch the rule.
With this rule enabled, Velociraptor will automatically assign labels to new clients as they connect, streamlining your workflow.
-------------------------------------------------------------------------------------------------------------
Investigating the Malicious Attachment
The client informs you of the attachment name (e.g., 123.ps1). Your goal is to determine:
How many endpoints have this file.
The file's location on each endpoint.
Here’s how to proceed:
Step 1: Create a Hunt
Navigate to the Hunts section.
Use the FileFinder artifact to configure the hunt.
Configuration Example:
If you’re looking for 123.ps1, set the search parameter as:
Step 2: Launch the Hunt
Once launched, Velociraptor will search for the specified file across all endpoints. You can view the results under the Notebook tab.
Output:
Improving Readability of Results
By default, the output may not be user-friendly, especially if it contains 20-30 artifacts. To make the data more readable:
Click the edit icon in the Notebook.
Paste the following query:
SELECT Fqdn, OSPath, BTime AS CreatedTime, MTime AS ModifiedTime
FROM source(artifact="Windows.Search.FileFinder")
LIMIT 50Run the query, and you'll see a neatly formatted output.
Now after this see below screenshot (How good and easy view it have become right?)
-------------------------------------------------------------------------------------------------------------
Labeling Infected Endpoints
Let’s say you identify 20 infected endpoints out of 100. To make tracking easier, label these endpoints as Phishing.
Here’s the query to do so:
SELECT Fqdn, OSPath, BTime AS CreatedTime, MTime AS ModifiedTime,
label(client_id=ClientId, labels=['Phishing'], op='set') AS SetLabel FROM source(artifact="Windows.Search.FileFinder")This automatically assigns the label Phishing to all infected devices(20 device), simplifying your investigation.
Example: In my case
Before running automate query:
After running the note book:
After Running query in notebook
Once you have identified 20 endpoints that have opened the file or downloaded the attachment, the next step is to determine which users these endpoints are associated with for further investigation. There are multiple ways to accomplish this.
Like:
Asking Client each endpoint belong to which user or using velociraptor live query method:
Running live query:
Create hunt for only endpoint which label is phishing or crowdstrike previously(This is where label become more useful instead of running hunt to all endpoint we can run hunt on only labelled endpoints to get data see how useful label become)
Select Hunt you want to run in this case Windows.sys.allusers
Launch the hunt and you will get the endpoint belong to which user( this user information will be usefull in our next hunt)
Once you run this hunt, use a notebook to extract a list of all affected users and their respective laptops. This initial step helps you identify around 20 laptops belonging to users who potentially acted as "patient zero."
-------------------------------------------------------------------------------------------------------------
Tracing Lateral Movement
Next, we investigate whether these 20 users logged into other laptops beyond their assigned ones. To do this:
Launch a hunt using Windows Event Logs: RDP Authentication.
While configuring the hunt, use a regular expression (regex) to include the usernames of the 20 suspected users. For example:
Above example(Screenshot) is for single user
If you want to add multiple users use below regex
.*(Dean\.amberose|hana\.wealth|Chad\.seen|jaye\.Ward).*This pattern helps track these users across multiple endpoints. However, this step may produce a large dataset with many false positives. To refine the results, analyze the output in a notebook.
Output Before running notebook
As u see screenshot when i run the query I got 60 result u can see why we need to minimize it(Because if u run same query on 20 endpoints in real scenario output will be very intense)
Minimizing False Positives
To reduce noise, use a carefully crafted query. For example:
SELECT EventTime, Computer, Channel, EventID, UserName, LogonType, SourceIP, Description, Message, Fqdn
FROM source(artifact="Windows.EventLogs.RDPAuth")
WHERE (
(UserName =~ "akash" AND NOT Computer =~ "Akash-Laptop")
OR (UserName =~ "hana.wealth" AND NOT Computer =~ "Doctor")
)
AND NOT EventID = 4634 -- Exclude logoff events
AND NOT (Computer =~ "domaincontroller" OR Computer =~ "exchangeserver" OR Computer =~ "fileserver")
ORDER BY EventTimeThis query excludes routine logons to systems like domain controllers or file servers, focusing on suspicious activity. Modify it further based on your environment to suit your needs.
Output After running the notebook
If needed, run additional hunts, such as UAL (User Access Logs) for servers. You can use below hunt from velociraptor. By analyzing these logs, you can map which accounts accessed which systems, providing insights into lateral movement. for server as well. Use this information to update labels, marking new suspected endpoints for further investigation.
If you want to learn more about UAL how to parse and analyse it, check out my article below:
-------------------------------------------------------------------------------------------------------------
Hunting for Suspicious Processes and Services
To understand the attack's scope and detect malicious activities, examine the processes and services running across all endpoints.
For service use below hunt and for processes i will show practice:
Let start with processes:
Automated Process Hunting
We will run this Process hunting in two way
First without using notebook hunt itself:
Run a hunt using Windows.System.Pslist.
When configuring parameters, check the option to focus on "untrusted authenticated code."
This will flag processes not signed by trusted authorities, providing their hash values.
Second running hunt first (without untrusted authenticated code box check) and then using Notebook :
Lets run same hunt as previously without untrusted authenticated code box check
As soon as i run the hunt i got 291 processes on one endpoint
lets suppose if u run this hunt on 100 endpoints how many processes u will get damn analysis will be worst:
Worry not if used second method, I have a notebook for you to make analysis easy
Query:
For a more detailed approach, use this query in a notebook:
SELECT Name,Exe,CommandLine,Hash.SHA256 AS SHA256, Authenticode.Trusted, Username, Fqdn, count() AS Count FROM source(WHERE Authenticode.Trusted = "untrusted" // unsigned binaries
// List of environment-specific processes to exclude
AND NOT Exe = "C:\\Program Files\\filebeat-rss\\filebeat.exe"
AND NOT Exe = "C:\\Program Files\\winlogbeat-rss\\winlogbeat.exe"
AND NOT Exe = "C:\\macfee\\macfee.exe"
AND NOT Exe = "C:\\test\\bin\\python.exe"
// Stack for prevalence analysis
GROUP BY Exe
// Sort results ascending
ORDER BY CountOutput after running above notebook
Only 3 detection's with clean output
-------------------------------------------------------------------------------------------------------------
Hunting for Suspicious Processes with automated Virus total Scan
Imagine you've scanned 100 endpoints and discovered 50 untrusted Processes. Checking their hashes manually would be frustrating and time-consuming. Here's how to simplify this:
Output before Virus total automation:
Before this keep in mind first you have to run hunt like above once you get output after that use below query or notebook to automate analysis with Virustotal.
Use the following query to cross-reference file hashes with VirusTotal, reducing manual overhead(Using note book)
// Get a free VirusTotal API key
LET VTKey <= "your_api_key_here"
// Build the list of untrusted processes
LET Results = SELECT Name, CommandLine, Exe, Hash.SHA256 AS SHA256, count() AS Count FROM source()
WHERE Authenticode.Trusted = "untrusted"
AND SHA256 // only entries with SHA256 hashes
// Exclude environment-specific processes
AND NOT Exe = "C:\\Sentinelone\\sentinel.exe"
GROUP BY Exe, SHA256
// Combine with VirusTotal enrichment query
SELECT *, {SELECT VTRating FROM Artifact.Server.Enrichment.Virustotal(VirustotalKey=VTKey, Hash=SHA256)} AS VTResults
FROM foreach(row=Results)
WHERE Count < 5
ORDER BY VTResults DESCOutcome:
After running the query, you get VirusTotal results with file ratings, making it easier to prioritize your efforts. No more manual hash-checking!
------------------------------------------------------------------------------------------------------------
Tracing Parent Processes
Once you’ve identified malicious processes, the next step is to trace their origins. Here’s how:
Set Up a Parent Process Hunt: Suppose you’ve identified these malicious processes:
IGCC.exe
WidgetService.exe
IGCCTray.exe
Use the Generic.System.PsTree hunt to map their parent processes.
Configure the parameters
Configure the parameters by adding the malicious processes in a regex format like this:
.*(IGCCTray.exe|WidgetService.exe|IGCC.exe).*In our case
Outcome:
The output will show the process call chain, helping you identify the parent processes and their origins. This insight is crucial for understanding how attackers gained initial access and their lateral movement within the network.
------------------------------------------------------------------------------------------------------------
Investigating Persistence Mechanisms
Persistence is a common tactic used by attackers to maintain access. Let's focus on startup items.
Startup Items Hunt: Running this hunt on 100 endpoints can generate a huge amount of data. Use hunt (Windows.Sys.StartupItems)
For instance, a single endpoint yield 22 startup items(Screenshot below), and across 100 endpoints, the dataset becomes unmanageable.
Filter Common False Positives: Narrow down results what we do is create a notebook or a query which will exclude the files or path client is using in their environment or they aware about or we can assume that those are mostly legit like macfee, ondrive, vmware right.
LET Results = SELECT count() AS Count, Fqdn, Name, OSPath, Details
FROM source(artifact="Windows.Sys.StartupItems")
// Exclude common false positives
WHERE NOT OSPath =~ "vmware-tray.exe"
AND NOT OSPath =~ "desktop.ini"
AND NOT (Name =~ "OneDrive" AND OSPath =~ "OneDrive" AND Details =~ "OneDrive")
// Stack and filter results
GROUP BY Name, OSPath, Details
SELECT * FROM Results
WHERE Count < 10
ORDER BY CountOutput after running above notebook:
Outcome
The refined output is structured, significantly reducing the data volume and allowing you to focus on potential threats. For example, the filtered results might now show only 15 entries instead of hundreds. (You can narrow those down)
------------------------------------------------------------------------------------------------------------
Documentation Is Key
Throughout the process:
Document all malicious processes, paths, infected endpoints, and related findings.
Organize your notes for efficient forensic investigation and reporting.
------------------------------------------------------------------------------------------------------------
Investigating Scheduled Tasks
Scheduled tasks often serve as a persistence mechanism for attackers. Here's how to efficiently analyze using velociraptor:
Use the hunt Windows.System.TaskScheduler/Analysis artifact to collect scheduled task data.
Once the data is collected, run the following query to exclude known legitimate entries from your environment: Query:
LET Results = SELECT OSPath, Command, Arguments, Fqdn, count() AS Count
FROM source(artifact="Windows.System.TaskScheduler/Analysis")WHERE Command AND Arguments AND NOT Command =~ "ASUS"AND NOT (Command = "C:\\Program Files (x86)\\Common Files\\Adobe\\AdobeGCClient\\AGCInvokerUtility.exe" OR OSPath =~ "Adobe")
AND NOT Command =~ "OneDrive"
AND NOT OSPath =~ "McAfee"
AND NOT OSPath =~ "Microsoft"
GROUP BY OSPath, Command, Arguments
SELECT * FROM Results
WHERE Count < 5
ORDER BY Count // sorts ascendingOutcome:
By running this query, you’ll exclude known false positives (e.g., ASUS, Adobe, OneDrive), significantly reducing the dataset and narrowing your focus to potentially suspicious tasks.
Environment-Specific Adjustments:
Tailor the query to your specific environment by adding more exclusions based on legitimate scheduled tasks in your network.
------------------------------------------------------------------------------------------------------------
Analyzing Using Autorun tool
Autorun tools is another common tool which can identify attackers seeking persistence. Here's how to analyze them efficiently:
Use the hunt Windows.Sysinternals.Autoruns artifact in Velociraptor to gather autorun data across endpoints.
Refine Results with a Notebook Query:Autorun entries often generate a large amount of data. Use the following query to focus on suspicious entries (In notebook):
Query:
LET Results = SELECT count() AS Count, Fqdn, Entry, Category, Profile, Description, `Image Path` AS ImagePath, `Launch String` AS LaunchString, `SHA-256` AS SHA256 FROM source()
WHERE NOT Signer
AND Enabled = "enabled"
GROUP BY ImagePath, LaunchString
SELECT * FROM Results
WHERE Count < 5 // return entries present on fewer than 5 systems
ORDER BY CountOutcome:
This query filters out signed entries and narrows the results allowing you to focus on anomalies while discarding likely false positives.
Customization:
Like the scheduled task query, modify this query to include exclusions specific to your environment for more accurate results.
------------------------------------------------------------------------------------------------------------
Document Everything
Keep a record of all suspicious entries, including file paths, hashes, and endpoints where they were found. This documentation is essential for both immediate remediation and forensic reporting.
Iterate and Adjust
Each organization has unique software and configurations. Continuously refine your queries to adapt to legitimate processes and new threats.
------------------------------------------------------------------------------------------------------------
So far, we’ve gathered substantial data from scheduled tasks, autorun entries, and identified potential malicious artifacts. Now, let’s take it a step further to ensure that no other endpoints in the environment are compromised
Identifying Additional Compromised Endpoints
Once we have identified malicious files or processes from our analysis, the next step is to ensure they aren’t present on any other endpoints.
We’ll use the Windows.Search.FileFinder artifact to search for malicious file names across all endpoints.
This is the same artifact we’ve used previously, but now we’ll populate it with the suspicious file paths or names identified in the earlier stages.
Example paths (for demonstration purposes):
Launch the Hunt
Run the hunt across 100 endpoints or more to check if the identified malicious files exist elsewhere.
Reviewing the Output:
Once the hunt completes, you’ll see a detailed list of endpoints where these files are found. If the files are present on other endpoints, label those endpoints as “compromised” or “attacked” for further investigation.
Labeling Compromised Endpoints:
Use the following query to label endpoints automatically:
Query:
SELECT Fqdn, OSPath, BTime AS CreatedTime, MTime AS ModifiedTime,
label(client_id=ClientId, labels=['Phishing'], op='set') AS SetLabel
FROM source(artifact="Windows.Search.FileFinder")------------------------------------------------------------------------------------------------------------
Next Steps Based on Findings:
If no additional compromised endpoints are found, you can move forward with the analysis of the initially identified endpoints.
If more compromised endpoints are identified, label them and consider isolating or rebuilding them to eliminate the risk of reinfection.
------------------------------------------------------------------------------------------------------------
YARA Scans for Advanced Threat Detection
Once you’ve identified the potentially malicious files and endpoints, the final step is to run a YARA rule scan across the environment. This helps detect specific malware families or identify links to Advanced Persistent Threat (APT) groups.
Running a YARA Hunt
Use the Windows.Detection.Yara.Process artifact for this hunt.
Configuring Parameters:
If you don’t provide a custom YARA rule, Velociraptor will default to scanning for Cobalt Strike indicators.
To run a specific YARA rule (e.g., for detecting APT activity), upload the rule or provide its URL in the configuration.
Example of adding a custom rule URL:
Launching the YARA Scan:
Once configured, launch the hunt. Velociraptor will scan all endpoints and flag any files or processes matching the specified YARA rules.
Reviewing the Results:
If hits are detected, you can identify the malware family or APT group involved based on the rule triggered.
If no hits are found, you can confirm that the environment is clean for the specified indicators.
-----------------------------------------------------------------------------------------------------------
Now that you’ve identified infected endpoints and labeled the “patient zero,” it’s time to move to the triage, containment, and recovery phases.
KAPE Triage Imaging
The next logical step is to capture a triage image of the compromised endpoints. This allows you to collect crucial artifacts for further investigation.
Triage Imaging via Velociraptor:
Velociraptor simplifies this process by allowing you to run KAPE (Target Filed) directly on the infected endpoint. Create a hunt to initiate KAPE (Target) collection, targeting the relevant artifacts needed for forensic analysis.
Collect key forensic artifacts such as registry hives, event logs, and file system metadata.
Ensure the image is stored securely for further examination.
Manual Imaging (Optional):
If Velociraptor isn’t an option, you can run KAPE manually on the infected machine to create a comprehensive triage image.
Quarantining Infected Endpoints
Once the imaging process is complete, it’s critical to keep isolated the compromised systems or if not done yet isolate endpoints from the network to prevent further spread or communication with potential Command and Control (C2) servers.
Using Velociraptor for Quarantine: Velociraptor can quarantine endpoints by blocking all network communications except to the Velociraptor server.
Create a hunt to execute the quarantine action.
This ensures the endpoint is unable to communicate externally while still being accessible for analysis.
Benefits of Quarantine:
Prevents lateral movement within the network.
Ensures minimal disruption to the ongoing investigation.
Recovery and Reimaging
After quarantining the compromised endpoints:
Reimage the Systems:
Reimaging cleans the endpoint, restoring it to a known good state.
Deploy it back into the production environment only after ensuring the threat is eradicated.
Forensic Analysis (Optional):If deeper investigation is required, forensic specialists can analyze the collected artifacts.
Velociraptor for Forensics: Velociraptor supports advanced forensic capabilities, allowing you to parse and analyze collected data.
Manual Analysis: Some professionals like me prefer using tools like KAPE, parsing artifacts manually for an in-depth understanding of the attack.
Additional Hunting (Optional)
Before wrapping up, you can perform further hunting on the infected endpoints to gather more details about the attack. For example:
Command History:Identify commands executed on the endpoints, such as psexec or PowerShell commands, to understand the attacker's actions.
Network Activity:Investigate network connections to detect communication with suspicious IPs or domains.
Persistence Mechanisms:Look for persistence techniques like registry changes or scheduled tasks.
Velociraptor offers an array of artifacts and queries for such investigations. Explore these capabilities to uncover additional insights.
-----------------------------------------------------------------------------------------------------------
Final Thoughts
With the steps outlined, you’ve gone through a comprehensive process to identify, contain, and recover from an endpoint compromise. From advanced hunting to quarantining infected systems, Velociraptor proves to be a powerful tool for incident response.
While this article doesn’t delve into detailed forensic analysis, it’s worth noting that Velociraptor can handle a wide range of forensic tasks. You can collect, parse, and analyze artifacts directly within the platform, making it an all-in-one solution for responders.
For those who prefer hands-on forensic work, tools like KAPE and manual parsing remain excellent options.
What’s Next?
This article is just the beginning. Velociraptor offers many more possibilities for proactive hunting and investigation. Experiment with its capabilities to uncover hidden threats in your environment.
Stay tuned for the next article, where we’ll dive deeper into another exciting topic in cybersecurity. Until then, happy hunting! 🚀
Dean