New Location

My website has moved to http://www.jasonwhaley.com. Please visit there for the latest and only remain here for legacy content.

Wednesday, June 25, 2008

securechannelfacade - scp wrapper

So I'm making some decent progress on my jsch facade that intended on writing some time back. Here's the project page over at google code. My goal is to simplify doing things with jsch, which is a pure java library for ssh. This evening I added the ability to scp multiple files over the same ssh session.

Sunday, June 8, 2008

Load Balancing Your QA/Test Environment

For the purposes of this post, assume there are four distinct environments that are available to host a suite of network applications - some being end user applications that deliver html/flash/etc. and others being applications that are really web service containers. These four environments are partioned logically as follows:

  1. Development - playground where applications are first deployed for initial intergration testing. This environment is always in a constant state of flux.
  2. QA - area where snapshots of the development environment are deployed for a full set of tests by a QA team.
  3. Staging/Pre-Production - area where the suite of builds blessed by QA as being production ready are initially deployed as a "practice" round and final testing is conducted. This environment is meant to simulate production as much as possible.
  4. Production - environment that is live to the rest of the world.
In the Production environment it is quite likely that these applications will be load balanced or clustered, which is expected since this is the only way any application can achieve high availability and scale properly. Since the Staging environment is meant to be a mirror of Production, it too would be expected to be load balanced. Staging may likely be exposed to the outside world from time to time as well, for the purposes of giving demos of things not yet live or to test integrations with external 3rd parties that can't be achieved otherwise - another reason to make sure there is some redundancy and high availability enabled here.

But what about that QA environment? Its purpose is ultimately to just allow a team of QA testers to thoroughly test all functionality. There's really no good reason to load balance this environment since it is an internal environment that shouldn't experience a high workload and an uptime of virtually 100% isn't a mission critical requirement for it. Combine this with the potential cost in man hours and hardware/licenses that may be required to load balancer and/or cluster in QA and it seems quite apparent that providing such redundancy in the QA environment is worthless idea... right?

Wrong.

At my present place of employment, we actually load balance our primary test environment. Within the last 2 months alone, we've observed multiple instances of cases where there are bugs with a web application that only manifest themselves when the application is used in a load balanced environment. I won't share the gory details, but it has mostly involved a) not being able to control session hopping due to ineffeciences in a vendor's framework and toolset and b) proxying by a client side application to a server side application not working at all in the load balanced environment. The latter case took at least 20 total man hours to analyze and correct.

So, in a world (or at least my world), where development cycles are performed in the matter of 2-3 months where there is a QA phase that lasts nearly the whole time, but a pre-production phase that is typically only a week in length, would you rather like to find out about these bugs during those 2-3 months development and QA is still occuring so that you actually have adequate time to fix and resolve the problem? Or would you rather discover these bugs within a handful of days before the committed go-live date where, most likely, you'll likely already have a slew of bugs not discovered until late in the game still pending and/or you will have to make concessions to your infrastructure to accommodate an application that doesn't work fully, or at all, in a production environment configured for high availability.

The answer here is to be able to see these problems as soon as possible. Not doing so puts a project at risk. As was the case in the two instances I've described, and probably countless others, having a QA environment actually be configured for load balancing allowed the problems to be detected, and fixed, rather early - thus lessening the number of major or blocking issues that may arise just before a launch is scheduled to occur.