Meetup callback

It’s More Ops Than You Think
June 2018
Bob Balaban, CA/Veracode

Copyright © 2018 CA. All rights reserved.
Agenda
• The “promise” of serverless
• Veracode Dynamic Analysis
• Replacing a “server” with a “service”
– Advantages
– Dev and DevOps processes
– Demos
– Dev effort summary
– Ops effort summary
• Lessons Learned (including…security!)
• Q & A

Advantages of Serverless (from Wayne)
 Automated Scalability
 Built-in Fault Tolerance
 LessOps – No infrastructure to manage
 Lower Cost - Pay only for what you use
 Faster development - Small functions easier to code and business logic focused
 Lower Risk – responsibility transferred to provider
 Integrated security – No patching, remote into O/S (SSH/RDP)
 Automatic logging
We will come back to this later

Veracode Dynamic Analysis
• Scans a “target app” (website) looking for security
vulnerabilities
– Cross-site scripting (XSS)
– SQL Injection (SQLi)
– XML External Entity (XXE)
– Remote File Inclusion (RFI)
– OS Command Injection
– Server-side Request Forgery (SSRF)
– Directory Traversal
– …and many more
• See OWASP: http://www.owasp.org

How Do We Find Vulnerabilities?
• “Ethical hacking”
• We “attack” the target (in ways that don’t damage data or
functionality)
• If an attack succeeds, we report a vulnerability
– And help you fix it!
• Example: SQL Injection
– “Select (sleep(15)) where true”
– We can tell if it works because the response is sloooowwww

What a “callback” server does
1. Scan engine attacks a target app with (ethically)
malicious XML, Javascript, etc.
2. If successful, the attack causes the app to make a call to
the Veracode “callback server”
3. The server records the successful attack
4. The scan engine polls the server later to fetch data on
successful attacks, and reports vulnerabilities

Callback Server (“Before”)
Callback
Server
Target App
Scan Engine
1. Inject attack,
e.g., “include remote
PHP file from callback
service”
3. Get results
2. “I was
hacked!”

Existing Callback Server
• AWS EC2, golang
• Simple “GET” response to target apps
– Serves static file
– Records successful attacks
• Simple REST API for scanner engine
– Register/unregister scanners
– Fetch successful attack data
– Clean up attack data
• Postgres RDB for attack data
– Auto-purges after 7 days

New Callback SERVICE Objectives
Re-architect as “serverless”:
1. Lambda functions only run when invoked
2. AWS API Gateway offloads authorization, firewall and load
balancing
3. DynamoDB nosql database easy to use from lambdas
4. Both Lambda and DynamoDB usage can auto-scale
5. Simplify REST API, otherwise maintain the same interfaces
6. Add some metrics tracking (count of invocations)

New Callback Service (“After”)
Callback
Service
Target App
Scan Engine
1. Inject attack
3. Get results
2. “I was
hacked!”

Callback Service Components (AWS)
DynamoDb
Query attack info
Capture attacks,
Serve text
A
P
I
G
a
t
e
w
a
y
From target app
From scanner
λ
λ

Tour of new Callback Service

Advantages of New Architecture
• DynamoDB TTL feature means we can simplify the
service’s REST API
– No need for cleanup calls
– No need for Postgres
• Lambdas do not get billed for idle time
• API Gateway will auto-scale lambda invocations when
needed (no need for auto-scaling groups)
• Logging calls go automatically to CloudWatch

Development Process Changes
• Smaller, independent functions (good!)
• No runtime debugging (bad!) – didn’t try Cloud9
– Fallback to “Printf debugging”
– Also required to see what your function is being called with (e.g.,
http event data structure not documented)
• Create AWS resources manually, the first time
– API Gateway (can start with a Swagger definition)
– Lambdas
– IAM
– DynamoDB tables
– S3 (backup Terraform db file)

Development Process Changes - 2
• Dev/test cycle
– Make code changes
– Manually (or semi-manually) deploy to AWS
• Need to figure out how to get code from IDE to AWS
• We use gitlab and Terraform
• Terraform can be used locally to push changes to AWS
• (Have to zip .js files first!)
– Test (Postman/curl)
– Commit changes to repo
– Repeat

Devops!
• Manual deployment is not sustainable long-term
• Must automate!
• Chose Terraform to script AWS resource
creation/configuration
– Injectable “variables” for AWS keys, config parameters,
environment variables
– Resource dependencies (e.g., create lambdas before api
gateway)
– Incremental changes! (if you keep a copy of the tf db)
• Chose Gitlab for CI/CD scripting
– Triggered on merge to Master branch
– Build zip files, manage Terraform invocation

Devops - 2
• Ongoing re-factoring of the project
– When and how to supply production account AWS keys
– Coordinate Gitlab deploys and “legacy” (Bitbucket) builds
– Who decides on gateway “stages”?
• Who monitors the logs?
– Some teams ship their logs to Sumo (more lambdas)
• Who watches scalability of the gateway and lambdas?
• Oh, and who watches cost?

Time Commitment – Dev and DevOps
• Dev was about 20% of “time to beta”
• DevOps (pipeline automation, config automation…) was
80%
• Why?
– The pipeline automation tools were new to us, we had to learn
them
– Some AWS features were new

Stuff We Had to Learn From Scratch
• API Gateway setup/config
• Lambda deployment/config
– Permissions, environment variables
• Querying DynamoDB efficiently
• Terraform – create/update entire AWS infrastructure
• Gitlab CI tools

Skills We Had to Improve
• Node.js
– Coding
– Debugging (via console.log() output, ew)
• Cloudwatch
• IAM for inter-component authorization
– E.g., API Gateway calling Lambda
– Lambda calling Cloudwatch and DynamoDB
• PostMan/curl
– Create http POST and GET tests

Recap: Advantages of Serverless
 Automated Scalability √
 Built-in Fault Tolerance √
 LessOps – No infrastructure to manage meh
 Lower Cost - Pay only for what you use √
 Faster development - Small functions easier to code and business logic focused √
 Lower Risk – responsibility transferred to provider meh
 Integrated security – No patching, remote into O/S (SSH/RDP) √
 Automatic logging meh

Lessons Learned
• Serverless is great, where appropriate
– Keep it simple
• Figure out the DevOps as you go
– Assume you will automate everything:
• Build
• Unit test
• Deploy
• Integration test
• You will re-factor the project at least as often as you
refactor your code

And Oh Yeah, Security!
“Security is like performance, you bolt it on at the end of the project”

“Security is like performance, you bolt it on at the end of the project”
(Said no one we want to listen to)

Severless Security: Application Dependencies
Application dependencies are going to become one of the main vectors for attackers
to compromise your FaaS infrastructure. It is crucial that you scan third party libraries
for vulnerabilities.
 To drive home that point - a researcher recently found a way to get push rights to over 14% of
NPM Packages and mainstream ones at that, e.g. : debug, request, react, gulp... How? It was a
combination of brute force and credential leaks from GitHub.
 In an evaluation of 1,000 open-source serverless projects, a threat research team revealed that
21% of them contained one or more critical vulnerabilities or misconfigurations, allowing
attackers to manipulate applications and perform various malicious actions.
Mitigations
 Maintaining an inventory list of software packages and other dependencies and their versions
 Scanning libraries for known vulnerable dependencies. Vulnerability scanning should be done as
part of your ongoing CI/CD process

Serverless Security: At-Rest and In-Transit equally vulnerable!
Serverless exudes a false sense of security due to secure API calls and encrypted
databases.
1. OWASP top 10 is still as relevant as ever!
2. SQL injection and other forms of injection attacks are still possible in the
serverless world, as are cross-site scripting attacks.
3. Keep your secrets in an Key Management System, not code.
4. Ensure your services have the least amount of privilege to read/write to your
databases.
5. Use direct/private connections between your VPCs and Databases wherever
possible.
Mitigations
 Always use safe APIs that either sanitize or validate user input, or APIs which provide a
mechanism for binding variables or parameterizing them for underlying infrastructure (e.g.
stored procedures or prepared statements in the case of SQL queries)
 Always use safe APIs that either sanitize or validate user input, or APIs which provide a
mechanism for binding variables or parameterizing them for underlying infrastructure.

Severless Security: Monitoring becomes extremely important
The one thing that the advent of serverless has made more difficult is monitoring and the
lack of understanding of security implications for various serverless assets.
 Logging is key. Ensure you have CloudTrail enabled on all accounts for all regions. Don’t
assume a developer or malicious actor won’t create an asset outside of us-east-1.
 The days of agents are gone! You now have to use platform-specific tools to monitor…
 …using Cloudwatch Metrics (with alarms) to monitor resources and set limits on either calls or
spend. When alerted, investigate when/why those limits are being exceeded.
 If you have a SIEM Tool or a 3rd-party monitoring service, send CloudWatch/Cloudtrail logs to
their services to be monitored. You’ll be amazed at what is already actively scanning your
public assets.
 RYOM – (rolling your own monitoring) – Lambda’s written in Python using the Boto SDK are
extremely powerful to monitor/modify your assets and alert based on customized parameters.
e.g. - A developer adds their home ip address to connect to an asset for debugging
from home, but a metric was defined to only allow connections of a specific CIDR range and
kicked off the Lambda written with Boto to remove that home ip entry automatically from
accessing that asset.

Thank you!
We’re hiring!
Q & A

Meetup callback

Recommended

More Related Content

Similar to Meetup callback (20)

Recently uploaded (20)

Meetup callback

Editor's Notes