In previous Foray into Jenkins, Puppet, Docker, and Photon posts, I was able to clone a Photon OS VM (part 1), deploy a Docker container into the Photon OS VM (part 2), and do automated load testing (part 3). Now it is time to look at improving the security of my Git repository. Given the number of scripts out there to look through GitHub for API keys, usernames, and passwords in order to rack up serious bills on Amazon and other cloud services, it behooves us to be extra vigilant.
The simple fact of the matter is that we embed quite a few things into our source code files that we should not. Such items include but are not limited to:
- API keys
- PII (even for testing)
- Passwords
- Usernames
- IP addresses.
The list is fairly long and growing daily. For the most common problems, however, we can easily scan the code, report any findings, and deny a commit or even push if they exist.
Why is this required? Because no code should be placed in a public location that contains sensitive information. As such, we need to be made aware of such findings, they should be reported, and they should not happen. Being able to add such tools to the commit process of any source code control mechanism allows us a great amount of power and flexibility.
However, this is not always enough. For auditing purposes, the information on commit problems should be gathered, sifted, and made part of any security governance. This way, issues can be brought up and studied in order to provide the proper awareness training to developers.
The simple case is to be aware that you may be placing into GitHub the keys to your kingdom. If that happens, you can be sure that your AWS account will be hacked and something created to rack up your expenses. Not only are alerts required within AWS, but you need to ensure that your repositories do not contain the data.
I wrote this simple Git hook to determine whether I am sending to my Git repositories anything that would be an issue. The script will follow, but first, how do I set this up?
Step 1: Create the script named pre-commit within /usr/share/git-core/templates (that script is at the end of this post).
Step 2: Ensure the script is executable: sudo chmod +x /usr/share/git-core/templates/pre-commit
Step 3: Clone a repository—any repository will do.
Step 4: Verify that the hook was copied to the proper location, which is .git/hooks/pre-commit within the repository you cloned.
Voila, you now have a pre-commit script that will scan your code for API keys, PII, IP addresses, etc., before you commit it to Git. On finding an error, it will exit with warning messages and log those messages to syslog. Syslog logging allows other tools to track such issues as well. Since there is a way within Git to not call the pre-commit script, the logging allows others to monitor what is happening.
Refer to https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks for more information on how to use the hooks provided. There are even server side hooks that work on a push and will not allow the push to succeed if the scan shows that the files being uploaded contain unwanted items.
Also, check out https://securosis.com/blog/my-500-cloud-security-screwup for things to do on the Amazon side to help warn of API issues.
The script follows and has a TODO list that is growing:
- Scan for IPV6 IP addresses
- Perform static code analysis for common security issues, such as buffer overruns
- Add more PII scans
- Add more hash scans
- Improve username scans for all forms of usernames, not just those domain users that are coded with user@domain.
#!/bin/sh
#
# Copyright (c) 2015 AstroArch Consulting, Inc. All rights reserved
#
#
# A hook script to verify what is about to be committed.
# -- Looks for IPV4 Addresses
# -- Looks for Usernames
# -- Looks for Domain Names
# -- Looks for Passwords
# -- Looks for Hashes/API Keys
# Called by "git commit" with no arguments. The hook should
# exit with non-zero status after issuing an appropriate message if
# it wants to stop the commit.
#
# Reference: http://www.unix-ninja.com/p/A_cheat-sheet_for_password_crackers
#
# Redirect output to stderr.
exec 1>&2
# First Check for IP
regex4='([0-9]{1,3}[\.]){3}[0-9]{1,3}'
# Hashchecks including base64 and API Keys
hashchecks='[0-7][0-9a-f]\{7\}[0-7][0-9a-f]\{7\} \$2a\\$\08\\$\(.\)\{75\} ([0-9a-zA-Z]{32}):(\w{16,32}) ([0-9a-zA-Z]{32}):(\S{3,32}) \$H\$\S{31} \$P\$\S{31} \$S\$\S{52} \$1\$\w{8}\S{22} \$apr1\$\w{8}\S{22} \$6\$\w{8}\S{86} [0-9A-Za-z+/]{60,}= [0-9A-Za-z+/]{20,}'
# CC/SSN/Passport/etc
pii="'4[0-9]{3}[ -]?[0-9]{4}[ -]?[0-9]{4}[ -]?[0-9]{4}' '5[0-9]{3}[ -]?[0-9]{4}[ -]?[0-9]{4}[ -]?[0-9]{4}' '\b3[47][0-9]{13\b' '\b3(?:0[0-5]|[68][0-9])[0-9]{11}\' '6011[ -]?[0-9]{4}[ -]?[0-9]{4}[ -]?[0-9]{4}' '\(?:2131|1800|35\d{3})\d{11}\b' '3[47][0-9]{2}[ -]?[0-9]{6}[ -]?[0-9]{5}' '[0-9]{3}[ -]?[0-9]{2}[ -]?[0-9]{4}' '[0-9]{4}[ -]?[0-9]{2}[ -]?[0-9]{4}' 'C0[0-9]{7}' '[23][0-9]{8}'"
icount=0
ifile=""
dcount=0
dfile=""
hcount=0
hfile=""
pcount=0
pfile=""
for x in `/usr/bin/git diff-index --cached HEAD|/usr/bin/awk '{print $NF}'`
do
# check for IP Addresses in files
/bin/grep -Eo "$regex4" $x >& /dev/null
if [ $? = 0 ]
then
if [ $icount = 0 ]
then
ifile="$x" else
ifile="$ifile $x"
fi
let icount=$icount+1
fi
# check for user@domain
/bin/grep -Eo "[[:alnum:]]+\@[[:alnum:]]+" $x >& /dev/null
if [ $? = 0 ]
then
if [ $dcount = 0 ]
then
dfile="$x"
else
dfile="$dfile $x"
fi
let dcount=$dcount+1
fi
# Hashes
/bin/egrep -oE '(^|[^a-fA-F0-9])[a-fA-F0-9]{32}([^a-fA-F0-9]|$)' $x | /bin/egrep -o '[a-fA-F0-9]{32}' >& /dev/null
if [ $? = 0 ]
then
if [ $hcount = 0 ]
then
hfile="$x"
else
hfile="$hfile $x"
fi
let hcount=$hcount+1
fi
for y in $hashchecks
do
/bin/egrep -o "$y" $x >& /dev/null
if [ $? = 0 ]
then
if [ $hcount = 0 ]
then
hfile="$x"
else
hfile="$hfile $x"
fi
let hcount=$hcount+1
fi
done
for y in $pii
do
/bin/egrep -o "$y" $x >& /dev/null
if [ $? = 0 ]
then
if [ $pcount = 0 ]
then
pfile="$x"
else
pfile="$pfile $x"
fi
let pcount=$pcount+1
fi
done
done
doexit=0
logstr=""
if [ $icount -gt 0 ]
then
echo ""
echo "Cannot continue due to IP in file(s):"
for x in $ifile
do
echo " $x"
done
echo ""
/bin/logger "$USER attempted to commit with IP included: $ifile"
doexit=1
fi
if [ $dcount -gt 0 ]
then
echo ""
echo "Cannot continue due to Domain Users in file(s):"
for x in $dfile
do
echo " $x"
done
echo ""
/bin/logger "$USER attempted to commit with Domain Users included: $dfile"
doexit=1
fi
if [ $hcount -gt 0 ]
then
echo ""
echo "Cannot continue due to Password Hashes in file(s):"
for x in $hfile
do
echo " $x"
done
echo ""
/bin/logger "$USER attempted to commit with Hashes included: $hfile"
doexit=1
fi
if [ $pcount -gt 0 ]
then
echo ""
echo "Cannot continue due to PII in file(s):"
for x in $pfile
do
echo " $x"
done
echo ""
/bin/logger "$USER attempted to commit with PII included: $pfile"
doexit=1
fi
if [ $doexit -gt 0 ]
then
exit 1
fi
