Using ARMO Kubescape to scale kubernetes security adoption across an enterprise

Note - This is not an introduction to Kubernetes.
It is expected that the reader is already aware of what Kubernetes is and how it works.

Agenda / Topics of discussion

• Kubernetes is everywhere !!
• Challenges of Kubernetes security adoption at scale across enterprises.
• Strategies to solve container security adoption challenges.
• Conclusion.

Kubernetes is everywhere

Enterprise visionaries and thought leaders do not agree on a lot of things, but even they have kind of conceded that Kubernetes is quickly becoming the de-facto standard for application delivery across the IT landscape, from startups to mid-sized companies to big enterprises.

The 2021 Cloud Native Survey (https://www.cncf.io/reports/cncf-annual-survey-2021/), organized by CNCF, shows that the usage of Kubernetes is continuing to grow, and isn't likely to stop.
Some data points of interest are -

• 96% of IT organizations are either evaluating or already using kubernetes.
• 5.6 million developers in the world actively use kubernetes.

However, at the same time it is a well-known fact, that Kubernetes is extremely hard to implement, or get right.
And, quite frankly, the hardest part of Kubernetes is getting its security right.
As a result, Kubernetes security has become a hot topic and rightly so.

The State of Kubernetes security report, published by RedHat ( https://www.redhat.com/en/resources/state-kubernetes-security-report), really highlights a lot of challenges that are evident with the current kubernetes ecosystem.
Some interesting data points covered in the report are -

• 55% of enterprise IT companies have confirmed that kubernetes security concerns have delayed or slowed down production deployment.
• 94% of respondents to the survey confirmed that they have faced at least 1 security incident in their kubernetes environment in the last 12 months.
• 59% of companies have mentioned that their main concern about adopting container strategies is how to secure them, and maintain a strong security posture.

The reason I am trying to highlight these points is to enforce the opinion that Kubernetes is hard, and implementing Kubernetes security is harder.

It is very important that an enterprise has a clear, and well-understood strategy on how to handle the myriad challenges that are evident in managing security of container deployments on an orchestration platform like Kubernetes.

However, the good thing is -
in Enterprise IT, most problems, if not all, have solutions.

In recent past, there has been a strong focus by vendors, cloud service providers, etc on tools, practices, and processes that can seamlessly mitigate the challenges associated with implementing a comprehensive container security strategy.

ARMO (https://github.com/armosec/kubescape), is one such company, which is doing impressive work in this sector.

Their flagship product kubescape is one of the most comprehensive and easy to use container security solutions available in the market today.

What is kubescape ?

Kubescape is a k8 open-source tool providing a multi-cloud K8s single pane of glass, including risk analysis, security compliance, RBAC visualizer, and image vulnerabilities scanning.

Kubescape scans K8s clusters, YAML files, and HELM charts, detecting misconfigurations according to multiple frameworks (such as the NSA-CISA, MITRE ATT&CK), finding software vulnerabilities, and showing RBAC (role-based-access-control) violations at early stages of the CI/CD pipeline. It calculates risk scores instantly and shows risk trends over time.

You can find more details about kubescape at - https://github.com/armosec/kubescape, and https://www.armosec.io/blog/kubescape-the-first-tool-for-running-nsa-and-cisa-kubernetes-hardening-tests/

Why should I choose kubescape over other tools?

• User-friendly UI for streamlined scans and test management.
• An instantly calculated risk score based on the current scan
• Easy access to a history of past scans.
• Exceptions management, allowing Kubernetes admins to mark acceptable risk levels.
• Build and create customized compliance frameworks.

I have used it personally and it is pretty easy to get onboarded with kubescape, literally in 5 minutes.

Their UI is extremely elegant, and intuitive, and the range of controls that they have implemented as a default offering is comprehensive, and includes some of the most common security controls available in the market today like MITRE, NSA, and so on.

Please find below a snapshot of their scan output that I ran on my personal cluster -

Challenges of Kubernetes security adoption at scale across enterprises

In this blog, I am not going through a tutorial of kubescape - because there are already wonderful blog posts, and YouTube videos that can guide you through the entire process, step by step, and quite frankly, the way the tool is designed, it is extremely easy to get started.

What I want to focus on this blog, is on the challenge of how to use kubescape to scale the adoption of a standard security tool and enforce a consistent Kubernetes security posture across an enterprise.

To, understand the solution, we must first be cognizant of the problem that exists.

First, you need to understand that enterprises are not small entities comprising of 2-3 teams, or 50 people. So, how do we define enterprises -

• They have multiple departments and locations.
• Hundreds of teams and departments spread across the company, possibly globally distributed.
• Everyone has clear responsibilities, and hierarchies.

In a big enterprise, an initiative like implementing security is much more difficult, and prolonged than enforcing the same controls in a small, or medium sized company.

Mostly, what happens in these cases, is that due to lack of central governance, controls, and policies in place, each team implements security controls in their kubernetes clusters in their own way, resulting in divergence and chaos.

I call this the Kubernetes Security Divide -

As you can see from the above image -

• Each team implements their own security policies.
• There is no standardization, or cohesive approach across the enterprise.
• Every team is working in silos.
• However, the target state of the enterprises is a situation where all teams will follow a standard set of guidelines, practices and patterns.

But, it is really very hard to move from left-to-right, across the divide.

So, in a nutshell, some of the challenges of trying to scale a kubernetes security initiative across hundreds of teams in an enterprise could be -

Strategies to solve container security adoption challenges

How can kubescape help ?

Enterprise security architects can leverage kubescape as a tool to consolidate security practices across an enterprise, and ensure that all teams are adhering to a standard set of security guidelines, and policies.

Let's try to evaluate the different strategies that can be leveraged to solve these challenges -

In a nutshell, we will discuss the below strategies to increase the adoption of a consistent security posture for containerized workloads across an enterprise, leveraging kubescape by ARMO-

Strategy 1 - Having an enterprise wide container/kubernetes security framework in place.

Every enterprise has their own requirements. No two companies are alike. The applications within the enterprise might vary, but they must still follow a standard set of policies that is a must across the enterprise.

Kubescape makes it very easy for an enterprise to create a custom security framework, that the enterprise can push to all service lines, and teams as a must-have.

Kubescape offers 4 security frameworks out-of-the-box , as shown below -

At the same time, it also offers 70 pre-built security controls, shared across the above 4 frameworks.

However, an enterprise might need a custom framework of their own, where they will select their own set of controls, out of the 70 pre-built controls, that have been provided out-of-the-box.

This could be due to a recommendation from the enterprise InfoSec team, or due to external regulators that have mandated those controls to the enterprise.

The point is - there can be instances where a custom combination of the provided security controls is needed to align with the security objectives of an enterprise, as shown below -

And, the best thing is that, this is pretty easy to do in Kubescape.

As you can see in the below image, I have created a custom framework for my enterprise.
In this framework, I only included the critical controls available. So, the idea could be that, any cluster deployment in my enterprise, must ensure that they pass the above critical controls, included as part of my enterprise framework.

Without that, I might not allow the deployments to be pushed to PROD, or something along those lines.

The best thing about kubescape, is that if I know which controls I need to include in my custom framework, it takes only 5 minutes, to create a custom framework, and we are good to go.

As of today, we can select any of the available 70 controls to be included in our custom framework, but going forward, there could potentially be more controls included as part of the default offering.

Also, since the controls are categorized as - Critical OR High OR Medium OR Low, it is easy for a security engineer with minimal idea of kubernetes to decide which ones should be included in the custom framework.

So, now we have a custom framework for our own enterprise in place, what to do now ?

Strategy 2 - Provide a shift-left security platform, with focus on enhancing developer experience.

Most enterprises make the wrong assumption that their security posture depends on their security teams. While that is true, at some level, but mostly, it is up to the developers to ensure how security will be implemented at the enterprise level.

Mostly, developers do not always care for security.

Indeed, nowadays the situation has improved a lot, with individual developers being more aware of security controls than ever before, but at a high level, you can consider that developers are not that concerned about security, as they are about developing and pushing their features into production.

So, say a developer does all his work of creating and writing the kubernetes application source code, they YAML files for deploying the application, the docker file for building the image and so on. At this point, the developer has no clue about whether the implementation that he is doing, is compliant with the custom kubernetes security framework that is enforced by the InfoSec team at an enterprise level.

So, the developer will do the changes in his local branch, push them to dev, raise a PR. Maybe, when the CI tests will run, many issues concerning insecure kubernetes implementations will get flagged, which means, the developer will again have to spend considerable time in fixing them, again pushing code, again waiting for the CI tests to finish, and so on.

This is not at all aligned with the agile software delivery lifecycle, that we want enterprise(s) to follow.

With that in mind, it would be best if we could push the analysis/scanning of the custom security framework to the left, meaning if the developer could get a feedback of some issues ( at least, the easy-to-understand static ones), directly when he/she is coding in his IDE, that would be best.

This is also called Shift-Left security, and you can use the VSCode extension for kubescape to do the same. With the VSCode extension, the dev can directly scan his kubernetes YAML files during development phase, utilizing the full power of kubescape, without having to leave his IDE - this can result in tremendous productivity increase, with no more waiting for costly CI tests to run, to get feedback.

Kubescape team has published a wonderful blog and video on how to get started with shift-left security using kubescape VSCode extension , you can find details at - https://www.armosec.io/blog/find-kubernetes-security-issues-while-coding/

This is something, that is very important to understand - the security tools that you are trying to enforce, at scale, across your enterprise, must be aligned with the developer.

You can see many tools in the markets, with hundreds of custom controls, dashboards, predictive analysis , etc, but most of such tools are directed at the security teams, and not at the developers.

Developer buy-in is a must to ensure, that your at scale adoption exercise is successful.

If you provide your developers with an easy-to-integrate tool, and a clean, functional user-interface, they will surely use your tool, and fix security vulnerabilities at the source.

This is where, from my perspective, kubescape shines - it has a very intuitiveand easy-to-use interface, and at the same time, it has equal focus on both local development teams, and central security teams.

Strategy 3 - Standardization of security controls and tools across an enterprise

One thing that is very important to ensure that at-scale adoption of your security initiative using kubescape succeeds, is to enforce standardization across the board.

In most mature enterprises, k8 cluster deployment, management, and operation is not performed directly via a CSP(Cloud Service Provider) or via manual scripts.
Most enterprises have a self-service catalog, using which an end-user can request to provision a fully functional cluster. A typical self-service portal might look like below -

Such an enterprise-wide offering of standard kubernetes components like clusters, namespaces, etc ensures that there is a level of standardization that is pre-enforced, across all deployments, regardless of the teams/business units involved.

Also, such offerings typically handle all cross-cutting concerns, which are common across all teams in the enterprise, for example-

As you can see, one of the primary cross-cutting concerns is Security , and this is where kubescape comes into picture.

All kubernetes installations/cluster setups across the enterprise must include ARMO kubescape pre-installed, as a default.

This will ensure that, project/dev teams do no have to do this as an additional step.

A kubescape pre-installed k8 cluster will go a long way in enforcing kubescape as the standard of choice for enforcing container security in the enterprise.

This can be easily done using the kubescape helm chart. This will ensure that kubescape gets deployed in a separate namespace within the cluster.

So, basically if the enterprise uses a custom script or process to provision new k8 clusters for end-users based on requirement, they need to add the section to install the kubescape helm chart, which can be done pretty easily.

Once this is done, kubescape will become one of the built-in tools available as part of the enterprise kubernetes offering.

As you can see, in-cluster deployment using kubescape is pretty simple, and easy to get started with -

Strategy 4 - Develop/invest in a central excellence team dedicated to kubernetes security.

This team will be in-charge of the overall security strategies, policy enforcement, and security posture management across the entire enterprise.

The members of this team should dictate the kubescape controls that will be enforced as part of the custom enterprise framework, ensure that all the security standards are enforced correctly, provide training an guidance on kubescape usage to different development teams, evangelize kubescape adoption as a single source of truth for kubernetes security.

This team will be placed horizontally, and interact with the different product teams, belonging to the different SL(s). So, all kubernetes security topics will be managed by this team, centrally, and then pushed to the different product/project teams within the enterprise, as shown below -

However, this team should include representation from different enterprise teams, to ensure that all parts of the enterprise, and key stakeholders are onboard.

At the same time, this team should have a strong collaboration with the vendor team - in this kubescape customer success/pre-sales/product team.

Only then, can the security initiative at scale, leveraging kubescape be successful.

A sample segregation/structuring of this this team, including different interactions, could be as shown below -

Strategy 5 - Inject security initiatives into automation/CICD practices

No security adoption initiative can scale, or be successful if it is not automated.
With kubescape, you have pre-built integrations available, so that you can directly inject kubescape into CICD platforms like Jenkins, and Azure DevOps.

The best thing about kubescape, is like always, the integration with other ecosystem providers and tools is extremely seamless and elegant.

For example, you can easily integrate kubescape with Jenkins CI/CD, CircleCI, GitLab, GitHub Actions, and Azure DevOps.

Let me show you a small example, using Azure DevOps( more details can be found at - https://hub.armo.cloud/docs/azure-devops-pipeline)

trigger:- masterpool:  vmImage: 'ubuntu-18.04'container: jmferrer/azure-devops-agent:lateststeps:- script:  |    mkdir $HOME/.local/bin    export PATH=$PATH:$HOME/.local/bin    curl -s https://raw.githubusercontent.com/armosec/kubescape/master/install.sh | /bin/bash    kubescape scan framework nsa *.yaml    displayName: 'Run Kubescape'

As you can see above, just adding the appropriate script task in the pipeline, will ensure that the k8 objects in the YAML files are scanned, as part of the pipeline.

Once, the pipeline runs, you can see the results in the Azure DevOps console, as logs-

In most enterprises, there is a separate DevOps practice, or COE who manage all the DevOps pipelines, and CI/CD processes and practices across the enterprise.

Sometimes, these teams use standard pipeline templates, to get started with a project, rather than starting from scratch. For example, there could be a standard DevOps/CICD pipeline, which already includes the different security components - for example SAST tools like Checkmarx/ Veracode, Code quality tools like SonarQube, and so on.

In such cases, the kubescape task should also be added as part of the pre-built integrations, so that whenever any team wants to create a new CICD pipeline for cloud-native, or containerized applications, the kubescape plugin will get activated, by default, and all k8 YAML files/helm packages will be scanned by default, as part of the pipeline.

Again, the message that I want to push here is that the k8 security practices, should be pushed as part of the enterprise-wide standards, so that all teams, irrespective of where they are in their cloud-native journey, can leverage those security standards, in this case - ARMO kubescape.

Conclusion

If one does a simple google search, anyone can find hundreds, or even thousands of articles, whitepapers, blogs, and tutorials on kubernetes security, and approaches for implementing the same.

However, those articles do not focus on why/how an enterprise needs to harmonize the choice of a kubernetes security tool, with a strategy, or a set of approaches, that can be followed to ensure that the tool is leveraged, at scale, across an enterprise, in the intended way.

I hope the readers, can focus on the below 2 key takeaways, from this article-

• Choosing the right tool for the job is pretty important - In this blog, I demonstrate how you can use an awesome open-source project called kubescape by ARMO, and leverage it to implement and enforce a set of security standards, practices, patterns and principles across the entire enterprise.

• However, just having a tool is counter-productive, if you do not have a consistent vision and streamlined strategy to support your initiatives.

Always remember one thing - Kubernetes is hard , Kubernetes security is harder, but scaling a kubernetes security initiative across an enterprise is the hardest.

But, I hope that some of the approaches that I have laid out in this article, can help in solving some of these challenges, and ensure that the reader does not face the same issues/problems that I had to go through when trying to implement a similar exercise in one of my past companies.

References

• Special thanks to Kunal Kushwaha, whose video on kubescape introduced me to the kubernetes security week challenge. You can find more details here - https://www.youtube.com/watch?v=SDpacCd5518
• Awesome blogs and tutorials at ARMOSEC blog by Ben Hirschberg, Jonathan Kaftzan, and Leonid Sandler at https://www.armosec.io/blog/, that helped me to get started with, and implement kubescape.Specific article I referenced : https://www.armosec.io/blog/find-kubernetes-security-issues-while-coding/
• Kubescape git repo which has an abundance of relevant and critical information related to the project, and kubernetes security in general, you can find more details at - https://github.com/armosec/kubescape
• All diagrams are drawn by me, using the awesome excalidraw tool (https://excalidraw.com/)