In previous Foray into Jenkins, Puppet, Docker, and Photon posts, I was able to clone a Photon OS VM (part 1) and deploy a Docker container into the Photon OS VM (part 2). Now, it is time to do some automated load testing in order to load and security test the deployed application. Load testing is required to determine the upper limit of the load this one container can handle. Once I know that, I can properly scale out the environment. But I also need to ensure that known security holes do not exist.
Load and security testing can be accomplished a number of different ways, but I have the opportunity to play with an Ixia BreakingPoint VE appliance, so I will use that.
Simply put, I did the following:
- Used part 1 of this Foray to deploy a Photon OS VM with Remote Docker enabled as one build in Jenkins.
- Used part 2 of this Foray to deploy an NGINX Docker container as a downstream build that fires when part 1 is successful within Jenkins.
- Created a third build within Jenkins to fire when part 2 is successful.
This third build uses three scripts stored within a private Git repository to get the IP of the NGINX Docker Photon OS instance (just like it did within part 2). Another script is used to dig username and password out of the SDK credentials file for use to log into the predeployed Ixia BreakingPoint VE. It then runs a third script that does a load test against the NGINX instance.
Very simply put, I have Infrastructure as Code and Testing as Code, not to mention the ability to do Security Testing as Code, within a Jenkins build flow.
To do all this, I needed to add the BreakingPoint TCL shell to my Jenkins image. I did this by downloading BPSH direct from the BreakingPoint VE and placed it within the /usr/local/bin directory on the Jenkins server. However, it would not run immediately, as BPSH is a 32-bit application and needed glibc.i686 and libgcc.i686 to be installed as well to run properly using:
yum -y install glibc.i686 libgcc.i686
Now it was just a case of creating a simple test within the Ixia BreakingPoint interface, exporting that test, and using it as an import to the automated test I would run. This way, most of the heavy lifting was done, and all my script had to do was:
- Change the target IP for the test
- Change the URI to use during the test
- Change the number of sessions to run per second
- Change the session counts within the test
Four small things to do using just three scripts and a canned test. In effect, 95s until I know if there are some security issues or not. Does my parser fail? Does the web server fail? I get scale plus security testing from one test. Adding more from the Ixia library of tests will tell if, at scale, my server can handle the attacks.
If we tie this testing to an application performance management tool such as New Relic or Dynatrace Ruxit, we get even more data, including where any failures occur.
The first script gets the IP of the test target from vCenter using the Perl SDK. You can use PowerShell, but I had troubles with credential file permissions as the Jenkins user. You have to first create a credential store (see the VMware SDK cred_store.pl instructions).
Get the IP script#!/usr/bin/perl -w
#
# Copyright (c) 2015 AstroArch Consulting, Inc. All rights reserved
#
# requires credstore
use strict;
use VMware::VIRuntime;
use VMware::VILib;
use VMware::VICredStore;
use File::Basename;
my $vmname=$ARGV[0];
VMware::VICredStore::init(filename => "./vicredentials.xml");
my @server_list=VMware::VICredStore::get_hosts();
my @user_list=VMware::VICredStore::get_usernames(server => $server_list[0]);
my $password=VMware::VICredStore::get_password(server => $server_list[0], username => $user_list[0]);
my $url = "https://".$server_list[0]."/sdk/vimService";
VMware::VICredStore::close();
eval {
Vim::login(service_url => $url, user_name => $user_list[0],
password => $password);
};
if ($@) {
print "$@"; exit 3;
}
my $vdata = Vim::find_entity_views(view_type => 'VirtualMachine',
filter => {"config.name" => $vmname});
foreach (@$vdata) {
my $vm_view = $_;
if (defined $vm_view->guest->net) {
if (defined $vm_view->guest->net) {
my $net_len = @{$vm_view->guest->net};
my $cnt = 0;
while ($cnt < $net_len) {
if (defined $vm_view->guest->net->[$cnt]->ipAddress) {
my $ip_len = @{$vm_view->guest->net->[$cnt]->ipAddress};
my $cnt_ip = 0;
while ($cnt_ip < $ip_len) {
print $vm_view->guest->net->[$cnt]->ipAddress->[$cnt_ip]."\n";
$cnt_ip++;
}
}
$cnt++;
}
}
}
}
Vim::logout();
This second script gets the credentials from the credential store for use by the BreakingPoint script. It is set up to not be called from the command line directly.
Get Credentials Script#!/usr/bin/perl -w
#
# Copyright (c) 2015 AstroArch Consulting, Inc. All rights reserved
#
# requires credstore
# not designed to be used directly
use strict;
use VMware::VIRuntime;
use VMware::VILib;
use VMware::VICredStore;
use File::Basename;
sub ltrim { my $s = shift; $s =~ s/^\s+//; return $s };
sub rtrim { my $s = shift; $s =~ s/\s+$//; return $s };
sub trim { my $s = shift; $s =~ s/^\s+|\s+$//g; return $s };
my $parent_id;
foreach (`ps -ef`) {
my ($uid,$pid,$ppid) = split;
next unless ($pid eq $$);
$parent_id = $ppid;
last;
}
my $parent = (grep {/^\s*\d+/} (`ps -p $parent_id`))[0];
my $parent_name = trim((split /\s+/, $parent, 5)[4]);
if ($parent_name eq "bash") {
exit;
}
VMware::VICredStore::init(filename => "./vicredentials.xml");
my @server_list=VMware::VICredStore::get_hosts();
my @user_list=VMware::VICredStore::get_usernames(server => $server_list[0]);
my $password=VMware::VICredStore::get_password(server => $server_list[0], username => $user_list[0]);
my $url = "https://".$server_list[0]."/sdk/vimService";
VMware::VICredStore::close();
print $user_list[0]." ".$server_list[0]." ".$password;
What follows is the BreakingPoint script, which takes several arguments. The most important are the target IP, the IP of the Ixia BreakingPoint server, and the URL to attack. We also can pass in the sessions count we want and the maximum number of sessions per second. Actually, for this script you need all those entries. Using:
./bps_run.tcl TargetIP TestName IxiaServerIP LocalTestFileName SessionCount MaxPerSecond URL
This script allows us to use one script but pass many different URLs, each with a different attack embedded with in it, such as SQL injection, cross-site scripting, and fuzzed input.
Breaking Point Test Script#!/usr/local/bin/bpsh
#
# Copyright (c) 2015 AstroArch Consulting, Inc. All rights reserved
#
# -- BPSH comes from BreakingPoint VE after login at
# Ixia Web Apps -> Help -> Download
# -- BPSH is a 32-bit application and requires
# glibc.i686 and libgcc.i686 to be installed on x86_64 systems
####
# Get command line arguments
set testip [lindex $argv 0]
set testname [lindex $argv 1]
set testrig [lindex $argv 2]
set localtestname [lindex $argv 3]
set sessioncount [lindex $argv 4]
set maxpersecond [lindex $argv 5]
set uri [lindex $argv 6]
####
####
# We are getting the credentials to use from an encrypted store
# SECURITY: Never embed credentials unencrypted in code!
set creds [exec ./get_creds.pl]
set username [lindex $creds 0]
set password [lindex $creds 2]
# strip everything after @ as that is not allowed in BP VE
set au [split $username @]
set username [lindex $au 0]
####
####
# Base procedures for import/export
proc testimport {bpsobj} {
global testname
global localtestname
puts "Import the Test"
$bpsobj importTest $testname -file $localtestname -force
puts "Imported Test: $testname"
}
proc exportresults {testobject} {
puts "Starting Test Report Export"
$testobject exportReport -file MyTestReport.pdf
puts "Test Report Export Completed."
}
####
####
# Get the Breaking Point connection
puts "Start the connection to BPS"
set bps [bps::connect $testrig $username $password]
puts "Connected to BPS bps, connection ID = $bps"
####
####
# Work with the chassis to put ports into a group for use
set chsobj [$bps getChassis -onclose exit]; #creates the chassis object
#
# Reserve the ports / really should check reserve state
# -- defaults to group 1 needs to be user specific
# -- should look for open ports and assign to 'new group'
#set pstate [$chsobj getState]
#set lstate [eval "list $pstate"]
#array set astate $lstate
#foreach index [array names astate] {
# set x [string match {[0-9]} $index]
# set y [string match {[0-9][0-9]} $index]
# if { $x > 0 || $y > 0 } {
# set vstate [eval list $astate($index)]
# array set avstate $vstate
# foreach vndex [array names avstate] {
# puts $vndex
# }
# }
#}
# reserve everything <-- This should be smarter
set group 1
for {set i 1} {$i < 9} {incr i 1} {
for {set j 0} {$j < 8} {incr j 1} {
$chsobj reservePort $i $j
}
}
####
####
# Import existing test (created on Ixia Breakpoint and Exported)
testimport $bps
# Create the test from the import
set testobject [$bps createTest -template "$testname" -name "$testname"]
####
########
# Edit the Test! We need to
# -- change target
# -- change session counts
####
# Modify Network Neighborhood of the test by
# cloning the neighborhood
# changing the target IP of the test
puts "Modifying Network Neighborhood of Existing Test"
set neighborhood [$testobject cget -neighborhood]
set nnew [join [split $neighborhood "-"] ""]
set newNN [append nnew "New"]
puts "New Test Neighborhood Name: $newNN"
set nn [$bps createNetwork -template $neighborhood -name $newNN]
set ips [$nn getAll ip_external_hosts]
# should only be one
foreach {name ipObj} $ips {
puts "External Hosts Entry: [$ipObj cget -id] to -ip_address $testip"
$ipObj configure -ip_address $testip
}
$nn save -name $newNN -force
$testobject configure -network $newNN
####
####
# Get Test Components
puts "Getting Test Components"
set components [$testobject getComponents]
set lcomps [eval "list $components"]
array set acomps $lcomps
# Get New Superflow name
set sflow [$acomps(clientsimpreset_1) cget -superflow ]
set nflow [append sflow New]
####
####
# Modify Superflow
# change web address
puts "Modify Superflow $sflow to $nflow using Uri $uri"
set superflowobject [$bps createSuperflow -template $sflow -name $nflow]
$superflowobject modifyAction 1 -uri $uri
$superflowobject save -force
####
####
# Modify Test Components with new SuperFlow and Parameters
puts "Modify Test Components for Max Sessions of $sessioncount using Superflow $nflow"
$acomps(clientsimpreset_1) configure -superflow $nflow
$acomps(clientsimpreset_1) configure -sessions.max $sessioncount
$acomps(clientsimpreset_1) configure -sessions.maxPerSecond $maxpersecond
####
####
# save the test
$testobject save -force
####
########
####
# Run the saved test
puts "Starting the test: $testname with neighborhood: $newNN"
#$testobject run -group $group -progress "bps::textprogress stdout"
$testobject run -group $group
####
####
# Get the results
set rc [$acomps(clientsimpreset_1) result]
set aa [$rc get appAttempted]
set as [$rc get appSuccessful]
set au [$rc get appUnsuccessful]
puts "Test results: Attempted: $aa Successful: $as Unsuccessful: $au"
$testobject exportReport -file "$testname.pdf" -format pdf
####
####
# Close the connection, cleanup
# -- should clean up reserved ports based on 'group' used
for {set i 1} {$i < 9} {incr i 1} {
for {set j 0} {$j < 8} {incr j 1} {
$chsobj unreservePort $i $j
}
}
$bps delete
####
set res "passed"
if { $au > 0 } {
set res "failed"
}
puts "Test Completed with $res Result!!!!"
exit $au
