Logging Made Easier
The NCSC has announced that it is retiring its Logging Made Easy (LME) project.
https://www.ncsc.gov.uk/blog-post/ncsc-to-retire-logging-made-easy
LME provides small to medium organisations with the ability to create a basic SIEM solution, and to perform basic security auditing & logging of what is happening on systems on the network. The latest build on the platform focused on gathering data from Microsoft based systems and providing a platform to correlate and integrate those logs. It was great solution for teams who did not know where to start or were on a tight budget.
When talking to customers who have deployed LME or put together a similar ELK (Elasticsearch, Logstash, Kibana) solution, they all run into the same problems; scalability, and resource issues (be it personnel or hardware). Most people will have likely deployed LME on existing local hypervisor infrastructure. They have got logs coming in but don’t have the man hours to spend working out what logs are important for their environment and most importantly, where are their blind spots.
IP Performance were tasked with a similar problem around 4 years ago by one of our customers, before Logging Made Easy was available. They wanted a logging solution that met the following requirements:
- Network Monitoring across a sprawling MPLS Network
- Monitoring of individual hosts and what they are doing
- Ingestion of as many logs as possible
- Maximum retention of those logs
- All for a reasonable cost
These requirements will not be unfamiliar to most security professionals. However, like the organisations who stipulate all the above, they soon discover that something must give.
With many of the security solutions we evaluated, we initially discovered that in isolation the products were excellent. However, when scaled out, costs became completely unaffordable.
So, as a company we decided to create our own solution, Swarm-SecOps. The goal of this project was not to reinvent the wheel for security, or promise that it would solve all security and operation problems. The goal was to create a solution out of already existing open-source tools and run it on dedicated commodity hardware, to provide a service that works for our customer and keeps costs respectable.
The open-source tools available to IT and security teams today are the best they have ever been and, in some cases, can even go toe-to-toe with the best products on the market. The trade off with using these tools is the reduced availability for upstream support when things go wrong. IT and security teams are already stretched and do not have the time to spend an endless number of days troubleshooting an issue that may not be considered a top priority by senior management.
So, what then is Swarm-SecOps?
Much like ‘Logging Made Easy’ the solution has Elasticsearch at its core. This offers a SIEM dashboard that has clearly become a key focus for the Elasticsearch team, with a set of pre-built rules that are easy to understand.
The platform has the capability to endlessly scale horizontally with no software lock on the number of events that can be processed. We found that nothing can beat providing the system with dedicated hardware such as NVME drives for quick ingestion and look ups on data from the last week, and large spinning disks for data retention. There is also the added benefit for upstream support and advanced security features if they are required in the future.
Swarm-SecOps also utilises custom built network sensors made using commodity hardware. ‘Zeek’ network monitoring software is a brilliant tool for capturing network data on packets traversing the wire. Zeek is able to keep all the important metadata; the protocol information, source/destination IPs and number of packets transferred while ignoring the body of the data (most of which is encrypted anyway) to keep storage and transmission costs down.
These sensors can all be managed from a central device that we have deemed the ‘Queen’.
In addition to the sensor for network monitoring, we also leverage ‘Sysmon’ and Fleet agents for low level Windows and Linux system monitoring on the network.
The combination of these elements can allow an organisation to create a powerful security and logging solution, all whilst keeping costs at a minimum.
Whilst this is the primary benefit, there is also an elephant in the room that needs to be addressed; the complexity of integrating all these systems and configuring them for optimal log ingestion, data retention and alert baselining.
What IP Performance has been able to offer our customers is our experience of setting up an open source SIEM solution, and provision of a solution ultimately designed to augment and integrate with existing teams. They can focus on what is important and teams can decide on the level of service they need, whether its just the build and setup the solution, provision of ongoing baselining and optimisation, or delivery of alert triage and reporting on what has been seen on the network.
From many discussions with customers around SOC services, budgetary constraints have made this a ‘nice to have’ rather than something they can realistically afford. However, many organisations just want a helping hand and insight into the thousands of alerts that are shown on their screen. IP Performance currently offers our customers a ‘SOC-Lite’ solution with 24-7 alerting and 9-5 eyes on screen, where queries can be raised with professionals already familiar with their environment.
Our goal is to offer as many organisations as we can the fundamental baseline for security, and tools to further progress in their security journey that can fit within budget. If this sounds like it would be useful for your organisation or would just like to have a chat about the solution we have been working with customers to develop feel free to reach out at
-
IP Performance enabled us to trial the system in a test environment and then between Dubai and Edinburgh. They provided us with full support and fast response to any problems.
Gavin Wilson, Systems Development Manager,
Heriot Watt University
