What is the paved road?
Paved Roads in Information security is the epitome of Secure Defaults. Simply put, when it comes to Application Security at least, the Paved Road ensures that the quickest path to production is also the most secure. This allows engineering teams to develop, build, and deploy their apps in a continuous manner with little active input from Information Security. It takes quite a bit of up front work and cooperation among many different tech teams, but the outcome is a combination of high security assurance and rapid development with no bottle-necking.
The term ‘Paved Road’ was originally coined by the Application Security team at Netflix. The focus of their Application Security team was to create a single view for development that allows them to develop and create value for the business, while maintaining application security throughout the SDLC in an automated fashion.
What goes into the paved road?
Let’s dive into what goes into creating the paved road for an organization. Each of these topics are a major task in-of-themselves, so be prepared.
Compliance Like any good Information Security presence within a technology organization, starting with good security policies is tantamount. When you enforce changes on technology due to insecurity, there should be a policy document that states what is being enforced and why, and has also been signed off by senior leadership.
Development (Self-Service) Also known as a Pre-Commit stage, this is the opportunity to allow development to help themselves build more secure code earlier. Offer tools and services to the development staff that can be utilized for quick security linting with little overhead. Security linting plugins for the IDE can be officially offered by the security team to quickly find security issues in the developer’s environment.
Additionally, pre-commit hooks can be installed in the developer machines that can perform security tasks and scans before code is pushed to source control. For example, git-secrets can be installed in the git client to prevent adding secrets like passwords and encryption keys to repositories.
Your organization should also be disclosing a list of approved languages and frameworks for the development staff to utilize; emphasize their usage over third-party alternatives. This is especially important for your internal security tools and libraries that were purpose built to make security easier to implement. You do not want to see projects attempting to recreate the wheel for security as this will drive up your risk for vulnerabilities.
Peer reviewing code changes is also a crucial process. Code changes should be reviewed by other engineers on the same team to ensure code quality before merging changes. These peer reviewing engineers should also be offered additional code security training to sharpen their skills noticing potential issues.
And on that note, offering security training that is informative and relevant to the developers role is also a great way to ensure additional code security.
Commit Stage/Source Control Once code has been pushed to Source Control, there are various other checks that can be done at this stage, the foremost one being Static Analysis Security Testing (SAST). We have mentioned SAST a number of times on Hella Secure blog, and this is the best area to implement it for a few reasons. One, it’s simple to automate the cloning of repos and kicking off scans at this point. Secondly, since SAST scans can be slow, it’s best that these are done in parallel with the standard pipeline, and not inline. This can be highly dependent on the language types and scanners, as some are faster than others. But also the ‘falsey-ness’ of SAST findings need to be taken into account. These scanner findings can be reported to the engineers and the high likelihood findings can possibly stop a build process.
During this stage is a good time to encourage the development team to begin thinking about creating ‘Security Unit Tests.’ In general, unit testing checks that the functionality of an application is working as intended. But sometimes we overlook the fact that we can also create unit tests for security based issues. For example, you can create unit tests for your security based functions to see how it would react when the values are maliciously tampered with or receive malformed or malicious data.
Lastly, this is the time to also think about dependency management. Security vulnerabilities in third-party libraries is a quickly escalating problem the more time goes on, and being able to identify possible known issues in your dependencies is essential. Luckily there is a substantial number of tools that can be wired into your SDLC pipeline to find these possible issues. Some artifact repositories even have this built in, including Sonatype Nexus and Jfrog Artifactory. Then there are repository-agnostic tools like Snyk, BlackDuck, and Dependency-Checker.
Acceptance Stage The gap between Source Control and Production is where your security automation options vastly open up. Using a CI/CD tool like Jenkins or GitLab allows AppSec to set up automated security acceptance stages. Work closely with the build and release engineers, but also ensure that your security tests do not hold up a pipeline build for too long. Anything longer than a minute (depending on application criticality you may be able to get away with longer) may be better left to an ancillary process that works in parallel.
One critical thing to not overlook in your CI/CD tool are infrastructure projects, deployment of infrastructure and containers. The security scanning of these are just as important as application projects. Infrastructure as code (IaC) like Puppet, Chef, Ansible, Terraform, or CloudFormation can also contain security misconfigurations and should be checked against your security policies. There is at least one security scanning tool for each IaC language and wiring this into the security acceptance stage helps to maintain a hardening baseline for infrastructure.
Utilizing containers or container orchestration can also ‘contain’ misconfigurations which can expose them for attackers. Docker images can be scanned for compliance and security before they are pushed to your registry of choice to ensure there is a hardening baseline. Some of these tools include:
Anchore
Collector
CoreOS Clair
Dagda
Docker Bench
Docker Actuary
Lastly, dynamic fuzzing and runtime security acceptance testing is possible. These are integration style tests for the purpose of testing for security controls at runtime. These can all be automated and wired into the CI/CD tool. Tools like Guantlt or Selenium mixed with Burp Suite or OWASP Zap can be automated to test for known security vulnerabilities. These can spit out reports and fail builds on particular tests.
Production Once we hit the production step it comes down to ensuring the machines and environments we are deploying in and into are properly hardened. This goes a little out of scope of this article, but I still want to mention a couple of items.
Now that we have all this new and shiny CI/CD goodness in place, we think to ourselves that’s it… the Paved Road is complete! Well not just yet. The tools that are generally used in a CI/CD pipeline, namely source control, repositories, building and deploying apps are sometimes notoriously insecure. It’s a funny irony that we set up all this cool tech to create secure, automated pipelines for production applications, while leaving the security configuration of these very tools by the wayside. Attackers and intruders love to see these types of tools under-protected and they become excellent targets for pivoting further into the network. Be absolutely confident that these CI/DC tools have the same security love as has been applied to the apps that go through them.
Finally, let’s talk about secrets management. Every environment has them and the difference between good security and great security is how the organization manages its secrets. There are a number of tools to help with this. Hashicorp Vault is a popular secret store for on-prem networks. If the organization is running infrastructure in the cloud, each of them sells their own secret storage service, like AWS KMS, Azure Key Vault, or Google Cloud Secrets Management.
What this means for Application Security
As mentioned in another post, there is a big advantage to automated CI/CD. As a security person myself it can feel like you’re giving up control of the security posture of the application space and introducing risk to allow developers to build and deploy on their own. However, this comes with the advantage that, if there is an urgent need for patching, patches can be put out to production in a minimum amount of time. Instead of just looking at vulnerability metrics across the enterprise for your risk rating, also look at the ‘Mean time to Remediate’ (MttR). If you have a modest MttR score, you know that even if vulnerabilities do crop up, they can be corrected much faster.
And by ensuring that the fastest path to production is also the most secure, this removes bottlenecks from security. Development is not as likely to ignore security controls when it’s all baked into the process, and they don’t even need to think about it.
Once you get to the point that you have a Paved Road ready as the primary pathway to production, AppSec can then begin using a questionnaire for the development staff when they create new features or make changes. These questionnaire answers can then modify the Paved Road if the criticality of the app is higher or lower.
When the road branches Now unfortunately, the road to production can be a winding and branching path, with a multitude of different teams using different languages, frameworks, deployment software to different environments and cloud resources. Forcing a tech organization down a single paved road, while necessary to get started, will not work for every team and every project. This is when it may become necessary for the AppSec team to perform a “Road Paving” request with development teams.
When a new project wants to get started up, or an existing project is under review, find out if they will be utilizing any of the set Paved Roads that have already been established. If they cannot for some reason, it should be discussed why they cannot. If they have a valid business reason, then it may warrant a Paved Road review. The AppSec team can review the proposed CI/CD pipeline for this project and implement new security requirements that can turn it into a new Paved Road. When that review work is complete, that now becomes a new possible Paved Road that can be utilized by other development teams.
It is crucial of the AppSec team and security engineering to catalog and publicize this information to the greater tech organization so that development is aware of the possible Paved Roads that teams can follow to promote their projects to productions. These can also be reiterated as part of the original security policies to strengthen their enforcement. Anyone caught not following a paved road can be marked as ‘off roading’ and can be marked as a risk to the greater organization.
Measuring risk and improving the roads
When measuring risk for the organization, Paved Roads give you the ability to reduce risk across the tech organization. As more projects adopt the Paved Road, risk is driven down.
Lastly, just like any real road, they require maintenance from time to time. Be sure to keep your ears open for comments and issues from the development staff and take surveys of the tech organization to find ways to improve the Paved Roads in the organization.
Summary
Using the Paved Road method, we can help drive down risk in an organization by making the quickest and painless paths to production the most secure. Remember the following points:
Start with the policy
Offer IDE security plugins
Build self-service security tools for development
Introduce security pre-commit hooks
Scan the code autonomously with Static Analysis Security Testing tools
Create security unit tests
Manage vulnerabilities in third-party software
Scan Infrastructure as Code and containers
Implement automated dynamic fuzzing and testing
Does your organization do something like Paved Road already or are you thinking about implementing it? Let’s discuss on Twitter.
Links
About the Author
Aaron is an Application Security Engineer with over 10 years of experience. His unorthodox career path has led to many unique insights in the security industry.