WordPress Hacked: Security Steps

Anyone who has a WordPress site may have been hacked at one point in time or another. A hack may happen regardless of how diligent you are, but what can we do to help us identify attacks before they occur and what can we do to secure the WordPress environment. I found some very helpful tools while perusing the internet for WordPress exploits and solutions. The following are what I have discovered as extremely useful, there are some rules if you been hacked, and then some tools you can use to find future attacks easier as well as prevent them.

First what do you do if you were hacked (assume working in the directory /usr/share/wordpress, but really wherever your wordpress install lives):

  • Make a backup of your database
  • Get a complete list of your existing themes and plugins
  • Download the latest WordPress zip file
  • Disable your existing WordPress installation (copy index.php to some other name)
  • Move your old WordPress install from /usr/share/wordpress to /usr/share/wordpress.old
  • Make a new /usr/share/wordpress directory and install the latest WordPress into this directory
  • Restore your existing database
  • Use the default theme
  • Compare the Exploit-DB for WordPress against your list of plugins and themes
  • Search Google for specific 0-day attacks against your version of WordPress, if any exist patch them as explained
  • Reinstall your Theme if not on the above list
  • Reinstall your Plugins if not on the aforementioned list, if you cannot find your plugins because they have gone the way of the dodo and become available, then inspect every file for any form of malware and then copy from your ‘hacked’ backup directory. However, if you do not know what malware looks like, then get a professional to help. <== NEW

After a few hours work your WordPress site should be back to normal hopefully without the latest list of exploits.

If you have bbpress also installed, repeat for bbpress as well as the two can be linked together in many ways. <== NEW

If you were not attacked, but wish to prevent it from happening, or if you were attacked how to prevent it from happening again. First, modify your installation just a bit to alleviate further attacks:

  • Change your Administrator and Editor passwords to a > 16 character pass phrase
  • Replace your NONCE salts within your wp-config.php file by getting a new set from https://api.wordpress.org/secret-key/1.1/salt/
  • Ensure no files or directories are world or group writable
  • Remove any unused themes and plugins (they can still be attack vectors even when not activated) <== NEW
  • Clear any Cache files in wp-content/cache/* or other locations <== NEW
  • Change the Permissions on /index.php to be non-writable (chmod ugo-w index.php) <== NEW

Then install some WordPress security plugins. These first are a must:

  • Ultimate Security Checker – Provides a list of easily fixable security vulnerabilities in the standard WordPress environment. Some of these end up being hard to pass without modifying other plugins. One in particular is W3 Total Cache, where you need to remove the ‘X-Powered-By’ headers.
  • Secure WP – Provides useful tools to disable attack vectors and giving up too much knowledge of your environment.
  • WP Security Scan – Use WP Security Scan to verify there are no vulnerabilities in your fresh installation or your database, use this tool to modify your administrative username and database table prefixes from their current settings. Also scan for other mis-configurations
  • Clean Options – Go through all your options and remove any orphaned and unused options while also inspecting the options for malware. Not only will this clean up your current options, but also speed up WordPress.
  • WordPress File Monitor Plus – This is the most important plugin. Configure it to run every hour or sooner so that you can get a list of changes to your WordPress files. If anything changes, inspect immediately for malware. If anything is added to your list of WordPress files, it could be malware. Simply look at the files for base64 encoded data that looks cryptic and not as plain PHP code. I then inspect my Apache log files for who perpetrated the attacks if possible and block them from happening again using firewall rules.
  • WordPress Firewall 2 adds some important but basic request verification against well known attack vectors <== NEW

These three add more logging into your WordPress environment and provide details that takes other knowledge to use:

  • WP MalWatch – Provides a way to look for malware in common places, while not always useful, it could expose critical issues.
  • Exploit Scanner – Provides a way to look for malware in uncommon locations, takes a bit of Javaascript and PHP coding knowledge to understand the output, but does bring to light certain critical issues.
  • WP Cron – Provides a way to inspect those cron jobs WordPress requires to work, ensure there are only expected jobs in the list.

In addition, to the above elements, I also made the following changes:

  • Disable the ability to directly access wp-config.php and other files using a .htaccess file (if using Apache)
  • Disable the ability to access non-image files from your uploads directory. Granted this only denies reading files of the appropriate extension, but you also need to ensure any upload software inspects for images only into this directory.
  • Changed permissions on critical WordPress files to be non-writable by all so that they cannot be modified. However, this does require you be diligent whenever you update WordPress to make the files readable once more, then revert them to non-writable by all.
  • Implement a web performance management suite such as New Relic RPM to detect when your WordPress site is accessing external web traffic. In Figure 1, for example we see an increase in ‘Web External’ traffic that represented a hack to a WordPress site. Most malware wants to call home, this one did to a recently created domain. New Relic RPM furthermore allows you to view all External Services your website calls, it is a very good idea to review that list to determine if all external traffic is expected or not. <==NEW
  • Sign Up for something like Website Defender to automatically scan the site every few days. <== NEW
Figure 1: New Relic RPM showing External Web Traffic

Given how WordPress sites are written today, some Web External traffic is to be expected, but when there is a sudden increase, it is also time to research to determine why this is happening.

Lastly, I created a development site into which I can install and test any new updates to plugins and WordPress. Until they get tested, they do not get onto the production WordPress installation.  By being diligent and relying on good attack intelligence you can stay on top of the current batch of WordPress attacks as well as prevent them from happening.  It is important to not only to review exploit databases for known vulnerabilities but to also understand how the attacks were perpetrated so that you can investigate and mitigate future security attacks.

We also need to remember to clean up any bbpress installation as well. bbpress, which is forum software, has links to WordPress when you use the two together.

vSphere 5 Upgrade Saga: Planning the Upgrade

With vSphere 5 there are some new considerations as well as the same old set of considerations for upgrading from older versions of vSphere and Virtual Infrastructures. The same old set of considerations is:

  • Can you upgrade your licenses, or did you let your SnS lapse?
  • Will you perform an in place upgrade? Or require a reinstall?
  • Is there a feature such as EVC that you need to enable during this refresh?
  • Do you need to wait until your third party and VMware software supports vSphere 5?
  • Do you need to upgrade any infrastructure VMs to support vSphere 5, ala databases, AD, etc?

However the new set of questions are:

  • Do you need more licensing per the vRAM pools license model?
  • Do you need to migrate clusters into a single vCenter to add those server licenses into your vRAM pool?
  • Do you need to federate your vCenter’s to increase your vRAM Pool?
  • Will you migrate from your existing vCenter implementation to the vCenter Appliance?

As we add more functionality within our virtual environments we need to consider how an upgrade will impact that functionality.

Upgrade: MacOS X Lion and Tool Upgrades

I recently upgraded my MacBook Pro from Snow Leopard to Lion as well as a Mac Mini which I did using Apple Remote Desktop ($80 well spent in my mind but that is another story). The upgrades on both went quite smoothly and I am pleasantly impressed with the new look and feel as well as how the tools I use once more ‘just worked’. There were a few tools I upgraded but not much. I had to upgrade the following tools:

  • Tunnelblick to access my VPN, I had to go searching for the upgrade
  • Oxygen Cloud, received an email about the upgrade, but the tool itself did not auto-upgrade
  • Codeweavers Crossover Office, received an email about this upgrade
  • Little Snitch, I did not receive an email nor did I notice it was not running, but once it started it said it needed to download a new version.
  • Livescribe Desktop, just required me to start the application and the download was ready. Which required an Adobe AIR update.
  • Flipshare, just required me to plugin my Flip and all was upgraded.
  • IOmega Storage Manager for my IX2, I had to check for the upgrade
  • Apple Remote Desktop, required an update that showed up in the AppStore

Given the upgrade just occurred, this is pretty good, all but one of my tools that required an upgrade actually emailed me that there was an upgrade available. I find that very helpful. However, the fact that Little Snitch, which is very useful security software did not start to tell me there was a download, I find to be a particular failure. Granted, I did notice it was not available and manually restarted the program.

The other tools I use, just worked, as expected.

  • Twitter
  • Growl
  • Fusion
  • RDC
  • SnagIt
  • Camtasia
  • arSync
  • Skype
  • Trillian

There is also an update to MacOS X Lion’s version of CUPS so that printing works properly for my Epson R1800 shared printer as documented in the article MacBook Pro: Update on Integration: Printing.

So far it has been a relatively painless upgrade with great new features such as the launchpad. While I personally cannot make use of the gesture computing mechanisms everyday, they are very cool to use. I just wish they had more pictures of Lions for the background. The Snow Leopard pictures were pretty amazing.

As for the Mac Mini, I did that upgrade using Apple Remote Desktop and that also while slower due to lack of SSD, also upgraded without a hitch even over quite a distance. Download, upgrade, reboot, reconnect, viola, successful remote upgrade of a Mac Mini.

Hurray for Apple, the upgrade Just Works.

MacBook Pro: Update on Integration: Printing

I use a 90-95% virtualized environment and here is a brief update of how I integrated my Mac Book Pro into this environment. Specifically about printing. Some notes first:

Print Server: Windows 2008 R2 running as a VM (why windows, because I needed an AD server so this is a AD/Print Server combination)

Printer connected to a Belkin FL5009 USB over IP device

This configuration seems fairly straight forward, but since I use passwords and heightened security I had to setup MacOS to properly communicate to the remote windows print server using some form of security.

To do this I first went to the following URL:

Printing Issues: CUPS/FL5009 – http://discussions.apple.com/message.jspa?messageID=6871761

This solution spoke about connecting to CUPs directly to configure the print queue to use a proper password. I did this by connecting directly to http://localhost:631 using the ‘root’ user password. Once into CUPs click on the Administration tab and then add printer. You will need to specify the printer in the form of smb://username:password@Host/printerName.

This finally allowed me to print from my MacOS desktop to my print server running as a virtual machine. However, this lead to a different problem with printing. Because I was using the Gutenprint and not the Epson R1800 driver all the printouts were faded and for months I wondered why. I did not find a solution until I went to print some graphics and photos. The problem is that the Epson would rather you used their driver but their driver is not setup to work via a print server.

After many hours using Google I found one article that helped by explaining what is happening. The issue is that this particular Epson R1800 printer, requires direct control by its driver to work properly and its driver does not work when you have shared the printer. At least not on the remote host, it works fine local to the print server.  All this boils down to is that I either print hose high quality required jobs direct from the print server or look for a printer that does have the concept of sharing its print driver.

Another option that does work is to connect the printer to the MacOS system via a USB over IP device as well and I know that works, but does not help others much when my MacBook Pro is on the road with me.

UPDATE: MacOS X Lion supports the EPSON SP R1800 driver which prints things as expected, without faded output. Very nice update! In essence, their now exists a CUPs driver that speaks the proper Epson language for shared printers.

How to succeed at at TechFieldDay!

While at Tech Field Day 6 in Boston, thank you Stephen for the invitation and Claire for setting up everything, I had the opportunity to make some observations of how best to succeed as a presenter to the Tech Field Day delegates.  Tech Field Day delegate love to talk about technology, but we also talk about bad presentations. This is not how you want us to remember your company. So without much further ado, here are the basic rules.

  • Know your audience! We are usually extremely technical people with in-depth knowledge of how the technology theme chosen for the specific Tech Field Day works. You may have delegates that have written books on the subject you want to discuss, who know intimate details of those technologies, etc.
  • Do your homework on who will be the delegates. Speak to them not the general audience. Know which ones use your products and assume we are looking at them in our environments during your discussion.
  • Pay attention to the live feed and understand the sorts of questions we have been asking. Be prepared to answer those same types of questions. View older live feeds to understand your audience better
  • Keep your presentations Technical. We were part of Tech Field Day because we are Technical people
  • Be interactive, repeat questions, and do not hide behind your tables. Keep our attention but be cognizant of where the camera is at all times.
  • Do not quote other analysts, we want details not analyst quotes
  • Do not tell us your market share, management team, etc. Do not show us marketing slides. We did our homework, why did you not do yours?
  • Stage your live demos so you get to the GOOD parts quickly, do not bore us with setting up users, etc., use videos appropriately
  • If we ask a question about comparison to other products, assume we did our homework and that we know what we are talking about, marketing answers are inappropriate
  • KNOW YOUR AUDIENCE.

All this sums up to one very good statement para-phrased from another conference, “Cut the —–, Show us the Product(s)”. We are there to see your product(s) so that we can fully understand what it is, and make our own decisions on how it fits into the overall fabric of the Tech Field Day Theme. We may even surprise you and show where your product(s) fit-in quite a bit differently than you expected.  Keep your slides to a minimum, use good contrasting colors and get to the technical details quickly!

vSphere Upgrade: vCenter Startup Issues

I recently rebooted my vCenter server for maintenance reasons. However, when I went to access vCenter, I could not as vCenter would not start. The vpxd.log file contained:

Datastore with invalid folder

Well that was a first for me. But what did it mean? Was it the non-existent iSCSI Server it had issues accessing? Or was it the Iomega StorCenter IX2 acting as an NFS share? So since I did not have a vCenter Server, I went to each node and removed those datastores as they were not actually in use.

This solution did not work. So it was time to look somewhere else? I tried to increase the verbosity of the logfiles for vCenter by adding the following to the <log></log> section of the vpxd.cfg file.

<trace><db><verbose>true</verbose></db></trace>

As well as setting the log level to ‘trivia’, but that just increased the log file output with no really useful data. So I reinstalled vCenter as my next task but kept the same database. This also had no effect. So in essence it was not a vCenter issue nor an issue easily cleaned up by removing datastores from the hosts. So what was it?

I then started investigating the vCenter Database not wanting to lose my existing performance data with a recreate (which would have also fixed the issue).  My theory is that the IX2 was the cause of the issue as it had on it non-vSphere created files. So I did the following SQL commands for my vCenter Database:

select * from dbo.VPX_DATASTORE;

This allowed me to find the IDs associated with the IX2 and the iSCSI Server. Their ID numbers were 661 and 435. So the next commands to run where the following for each of the IDs. Note this is NEVER recommended, and should only be done after first contacting your VMware support specialist.

delete from dbo.VPX_DATASTORE where ID=435;
delete from dbo.VPX_DS_ASSIGNMENT where DS_ID=435;
delete from dbo.VPX_DATASTORE_INFO where ID=435;
delete from dbo.VPXV_DATASTORE where ID=435;
delete from dbo.VPXV_ENTITY where ID=435;

Now I was able to boot vCenter properly. Finding that the datastore ID was also in VPXV_ENTITY took the most time. I had to review quite a few tables til I hit on this one. I kept getting an error about /vpx/group/#s24 not able to access the IX2 NFS server.  Well, the group #24 was within VPXV_ENTITY.

In finding this result, I had to add back in the IX2 as an NFS share. So I think the real culprit was not the IX2 but the iSCSI Server that no longer was available.  Why it was not available was not the real issue, but if a Datastore could not be reached, why would vCenter not pull that information from the ESX/ESXi host instead of failing to start?

As an aside, a future issue forced me to also reinstall my vCenter database, which would also fix this issue.