Please note that the bits and pieces of information come from various sources and things may or may not be exactly as we think they are.

In order to understand what HipHop for PHP does it is important to understand how PHP itself gets executed. In this article I will give a rough overview of the PHP execution process and make a few educated guesses in what aspect HipHop for PHP differs from there.

Based on the little information available online I'll discuss how HipHop for PHP is suitable for large businesses and what might turn out to be obstacles that are yet to remove.

The PHP execution process

PHP code can not be executed directly, a property it shares with all other scripting languages such as Python, Ruby and JavaScript. What makes PHP executable is the Zend Engine, and it does so in a two stage process.

1. The scripts are parsed and analyzed, checked for syntactical and semantical errors and if all goes well the application is transformed into byte code.
2. Since the byte code is still not executable on a CPU, it is interpreted further into machine code (that is CPU dependent) and then executed.

For most PHP applications the first parsing stage takes most of the total time. The second execution stage is comparatively fast. So in order to speed up a PHP application, most code optimizers hook into the Zend Engine after the first stage has been completed and save the generated byte code into a cache. In a consecutive call to the webserver the cache will be fed back into the Zend Engine, thus skipping the entire first stage of the execution process.

There are quite a few products on the market that behave in such a way. Zend itself offers the Zend Optimizer (now part of Zend Guard), there is eaccelerator and APC, which comes as a PECL package.

The execution process with HipHop for PHP

What sets HipHop for PHP apart from these three (and no doubt other) accelerators is that it does not act as a cache. Instead, HipHop for PHP will transform your PHP application into C++ code and compile that directly into a binary which is executable.

In that sense, HipHop for PHP is probably very similar to Roadsend PHP which also generates executable code from PHP, albeit not by transforming it into C++ first.

In essence, HipHop for PHP will get rid not only of the first stage of the usual execution stage, but also of a big part of the second stage, namely the interpretation of byte code into machine code. Generally I believe it is safe to assume that HipHop for PHP has the potential to run an application much faster than any of the previously mentioned accelerators, simply because there is less for to be done at runtime. Time will tell.

For detailed information on the internals of HipHop for PHP make sure you watch the Facebook Technology Tasting video on ustream.

Possible limitations

The requirements
The requirements to run HipHop for PHP are available online. Nothing out of the ordinary at first sight. The page claims that HipHop for PHP is tested on CentOS and Fedora. Both are venerable distributions, no doubt. But both also come without vendor support which may or may not be a problem for your organization.

Granted, companies who's codebase can actually benefit from HipHop for PHP will no doubt have a team of excellent system administrators which maintain Linux in their sleep. But those companies might also be in the situation that their customers demand a Linux distribution with commercial support, such as SLES, RedHat etc.

So this is more an organizational issue rather than a technical. That being said it is not clear yet whether or not packages will be released by other Linux distribution vendors once HipHop for PHP is out. If at all, it probably won't happen soon.

Coming back to the requirements for HipHop for PHP, it requires customized packages of libcurl and libevent, which again may or may not be an issue for your environment. Here I see the same organizational issue, but also a possible technical issue - you might be in the situation that you need the original libcurl or libevent for some other software on the server. Only a practical test will tell.

Unsupported language constructs
As with most PHP caches and compilers, a few PHP language constructs are not supported. Most commonly known is the eval() construct which is simply not available in C++. Personally I believe that eval() is a bad thing to begin with and you can get around it in almost every cases. Sadly many PHP frameworks such as TYPO3, many PEAR packages, Drupal and Joomla make use of eval().

If you're intending to run one of these through HipHop for PHP you might need to wait for an update from the community, patch it yourself, or remove the code from the system if you don't need it. In any case this is a time consuming task.

Aside from eval() a few more language constructs may not be supported or may not benefit from the compilation in terms of performance, such as call_user_func_array() and other 'dynamic' constructs like dynamic variables, dynamic includes or built-in functions like function_exists(), class_exists(), get_declared_classes() etc.

HipHop for PHP is no different from other caches and compilers in these regards, so please don't make that affect your choice. Just be aware that things might not be as easy as you'd like them to be.

No Apache?
This is speculative at this point but the Running HipHop page on GitHub seems to indicate that HipHop for PHP will not make use of the Apache webserver but instead come with its own webserver. My assumption is that this means we'll have to handle HipHop for PHP through Apache, much in the same way that we frequently do with Tomcat. This is not necessarily an issue.

The potential of HipHop for PHP

Despite the limitations I mentioned above, I believe HipHop for PHP has the potential to take PHP much further into the enterprise environment than many of the existing solutions. If it can deliver what it promises it is definitely something we will take a very close look at to give our eCommerce framework a serious performance boost.

Very high on my wishlist is to run the transformation process from PHP to C++ but skip the compilation. If that's possible I see the chance to make debugging compile errors much easier.

Consider the following scenario: You have a group of classes, each responsible for returning a specific statistic. All these classes implement the same API, as given by a common abstract class.

Now, some of these statistics are publicly accessible, others require a certain access control permission to be set.

The logical approach would be to implement a static property to the abstract class, denying access by default. Those classes that are indeed public will overwrite the static property with the correct setting and become accessible.

The property is static to gain speed: There is no need to instantiate the class, if the access control is not satisfied.

Stripped down to the bare essentials the whole set of classes to outline the issue looks like this:

abstract class StatisticBase {
    public static $acl = '0';
} // end: class StatisticBase

class PublicStatistic extends StatisticBase {
    public static $acl = '1';
} // end: class StatisticBase

class AdminStatistic extends StatisticBase {
    // default value for $acl is fine
} // end: class AdminStatistic

AdminStatistic is an implemented class that is private, indicated by the $acl property being 0. PublicStatistic on the other hand is public, hence $acl is 1.

Now before instantiating a specific class we can check whether or not the access condition is satisfied:

if ($user->isAdmin == true || PublicStatistic::$acl == 1) {
    // retrieve admin statistic...
} // end: if

But naturally this is cumbersome as soon as you have more than a handful of statistics - you don't want to add another condition to your if-statement whenever you add a statistic. So the logical solution is a loop:

$statistics = array(
    'PublicStatistic',
    'AdminStatistic'
);

foreach ($statistics as $statClassName) {

    if ($user->acl == 1 || $statClassName::$acl == 0) {
        // retrieve current statistic...
    } // end: if
} // end: foreach

Here PHP 5.2 will fail with a fatal error.

You can't access static class properties by means of using a variable for the classname. PHP 5.3 can. There is no really elegant solution for this. (If you know one, post it in the comments!)

While I'm glad that PHP 5.3 has one more OOP feature covered, It leaves me with the feeling that we still have a long way to go until PHP fully supports object orientation.

Now, if Novel would please release a PHP 5.3 package for SLES 10 and SLES 11... thank you

The AddThis "Social Bookmarking & Sharing Service" provides the possibility to share my blog to virtually all social bookmark services (currently over 200) with a single (!) button and also has an easy option to send links via email. Similar to ShareThis, AddThis provides me with an analytics-like overview of which articles have been shared when and where. Pretty neat. The documentation on AddThis is better than the one for ShareThis though so I went for the former.

Since with removing the Tweet This plugin the "auto-post to Twitter" option is gone I installed WP To Twitter to compensate for that. In fact WP To Twitter has more options than Tweet This, making things even more flexible for me.

If social bookmarking is a concern for you, check out AddThis and WP To Twitter.

However, when it comes to SEO and bandwidth optimization and standard compliance many of the plugins fall short and require some additional attention. The good news is that usually many plugins are available to scratch a single itch. Picking the right one often avoids many glitches.

Social network plugins and social bookmarks

I want to reach out to people with this blog. This includes enabling my honored readers to quickly share my articles with their community through the various social networks.

TweetMeMe Button and Facebook Share are two plugins that put nifty buttons in your blog that include counters to boast about how many people shared your content. Both are really great and work perfectly, but I encountered some issues due to which I removed both from this website after a test period of a couple of days.

I had to remove the TweetMeMe Button simply because it loads content from the tweetmeme.com server which is currently very slow, thus slowing down my website as well. There is nothing the author of this plugin can do about it I guess.

Facebook Share on the other hand was removed because it uses an id="" attribute for the button it displays, even if the button is on the page multiple times. Since this is violating W3C compliance this plugin had to go, too.

In addition to that, both plugins cause my layout to look a little bit messy and 'cheap', which I try to avoid. Again, that is nothing that can be fixed by the plugin authors since it is just my personal opinion. I really liked both plugins but these minor glitches made me move on to...

Tweet This - CSS sprites

I found another nice extension for social bookmarks called Tweet This. The author Richard X. Thripp did a really great job on this, offering bookmark icons for not one but many social networks. That covers my bases.

However I enhanced this plugin slightly in order to utilize CSS sprites for the social bookmark icons. My friend Chris recently talked about CSS sprites on his blog SwiftLizard.com. Here is the concept in a nutshell:

If you display ten icons on your website the traditional way the browser has to request ten files from the server - one for each icon. Since usually browsers do only a handful of requests at a time this slows down your website. With CSS sprites however, you can utilize CSS and one single graphic containing all icons to achieve the same effect with only one image being requested. This results in website response times that are much lower, thus enabling a smoother surfing experience.

Further optimization of Social bookmarks [added]

Since I posted this article I further enhanced the way I handle social bookmarks in this blog, you might want to check that out.

WP Super Cache

The plugin WP Super Cache almost needs no introduction. Written by Donncha O. Caoimh it enables us to circumvent PHP completely and serve static content in many cases. Naturally, this is much faster. The plugin is a must for anyone taking performance seriously. WP Super Cache alone reduced the initial response of my blog (time-to-first-byte) by 80%.

iPhone and mobile browsers

As the owner of a technology related website I believe it is reasonable (and my access statistics seem to agree) to believe that some of my visitors will visit the website from a mobile device such as the iPhone or a Blackberry. While technically the layout I chose for this blog looks ok in my iPhone, there is a lot of clutter that I don't want to wait for on my mobile device.

Thankfully there is a plugin called WPtouch which displays the blog in a highly stripped down but yet good looking manner when viewed with a recognized mobile browser. Not only do mobile visitors get their content faster by removing clutter, the usability is also highly increased. Perfect.

Combining JavaScript and CSS files

Similar to what I said earlier about many images slowing down a website, the same is true for many JavaScript or CSS files. Regardless of the file type you should keep your number of requests as low as possible. With Wordpress and its many plugins that is next to impossible though, unless you want to rewrite every plugin you install.

But there is hope. What CSS sprites achieve for icons, the Head Cleaner plugin does for JavaScript and CSS files. It can combine all JavaScript files into one and do the same for CSS. Further, JavaScript can be minimized and both file types can be compressed, reducing not the number of requests but the bandwidth as well.

Image size

This is a classic. Double check the images you have on your website for size - the size measured in bytes, not pixels. If you're not sure how to reduce images byte-wise, there are online services like the one on dynamic drive which can guide you (and save you lots of manual trial-and-error time).

General rule of thumb is that a png will be smaller than a gif and images with only a few colors should never be stored in a jpg. For example the ScrumMaster badge you see on the upper right was initially a jpg of 25kB. My all means the png I use now looks identical and is only 13kB.

Results

Given all the measures above, I was able to save over 50% of bandwidth and 75% of requests on the frontpage of my blog. No matter how fast your broadband connection is, you feel the difference. Even better, the website is still W3C compliant.

When in doubt what else you can do, I can recommend registering for Google Webmaster Tools and install their Page Speed Firefox plugin. It will hook into the famous Firebug plugin and provide you with an additional tab that gives very useful hints how to improve your website speed.

We're getting closer and closer to full test automation and continuous integration. The last article introduced you to how to write PHPUnit tests for SeleniumRC so that we can test website frontends through PHPUnit. After that we wrote our own phing build script that ran our backend as well as the frontend tests automatically, generated an code coverage report based on Xdebug as well as an comprehensive APIDoc based on PHPDoc comments.

The phing build script we used for this - test.xml - was executed manually by calling

phing -f test.xml

on the command line. Everything happened on the development virtual host.

The next logical step is to automate this and perform the additional tasks required to run continuous integration.

The first step is to do a manual svn checkout on the integration virtual host, so we have setup that is identical to the development virtual host.

Now, what the automation will have to do is a Subversion update of the integration virtual host in order to retrieve the latest copy. If the revision changes in that process (that is, if we indeed fetched newer files from Subversion) we again run the tests and generate the reports. If all tests succeed we can safely assume that we have a running system, and we can decide to create a tarball from it or update the staging system for our project manager / client to check it out. On the other hand, if a test fails we'd like to be notified by email.

This is exactly what a second phing build script will do.

Introducing more flexibility

Before we can do all that, we need to make a small but very important change to the Selenium RC test case. The setUp() method of out tests/TestForm.php file configures SeleniumRC so that it points to http://integration.local which is our development environment. This means that no matter where we run the frontend tests, this test would always test our development environment! Clearly not what we want.

If the developer runs the test manually (by executing phing -f test.xml on the commandline), the development environment has to be used. For the automation, the integration host http://integrate.integration.local has to be used.

We accomplish this by changing the setUp() method to this:

function setUp() {
  // determine hostname based on phing property
  $startDir = Phing::getProperty('application.startdir');
  preg_match('/\/(default|integration|demo)/', $startDir, $vhostMainDir);

  switch($vhostMainDir[1]) {
    case 'default':
      $browserUrl = 'http://integration.local/';
      break;

    case 'integration':
      $browserUrl = 'http://integrate.integration.local/';
      break;

    case 'demo':
      $browserUrl = 'http://demo.integration.local/';
      break;

    default:
  } // end: switch

  $this->screenshotUrl    = $browserUrl . 'reports/selenium';
  $this->screenshotPath    = $_ENV['PWD'] . '/../reports/selenium';
  $this->setBrowser('*firefox');
  $this->setBrowserUrl($browserUrl);
} // end: function setUp()

We retrieve the application.startdir property and with this information we can decide in which environment we run to set the browser URL and the screenshot path correctly.

Committing and retrieve everything to Subversion.

With that flexibility issue in the test out of the way we can commit the whole development environment /var/www/vhosts/default to Subversion:

svn import /var/www/vhosts/default https://your.subversion.server/repository/path/trunk
svn checkout --force https://your.subversion.server/repository/path/trunk /var/www/vhosts/default

Your development environment now is connected to the subversion repository.

Next, you have to do another svn checkout from the same repository URL, this time to the integration environment.

svn checkout --force https://your.subversion.server/repository/path/trunk /var/www/vhosts/integration

Now the integration environment is identical to the development environment.

Note that the virtual host configuration file for apache will not be overwritten, since the filenames are different. However the integration environment now contains two .conf files for apache. Since only the correct one is linked to /etc/apache2/sites-available, no harm is done though.

The continuous integration build script.

The single addition between our manual tests from the test.xml phing build file and the continuous integration is another phing build file that you will find at buildScripts/updateAndBuild.xml. If you open it in an editor it will look like this.

<?xml version="1.0" encoding="UTF-8"?>
<project basedir="." default="runTests" name="updateWorkingCopy">
  <property name="demoDir"      value="/var/www/vhosts/demo" />
  <property name="baseDir"      value="/var/www/vhosts/integration" />
  <property name="htdocsDir"    value="${baseDir}/htdocs" />
  <property name="testDir"      value="${baseDir}/tests/" />
  <property name="reportDir"    value="${htdocsDir}/reports" />
  <property name="seleniumDir"  value="${reportDir}/selenium" />
  <property name="masterFile"   value="updateAndBuild" />
  <property name="buildVersion" value="1.6.1" />
  <property name="srcDir"       value="/var/www/vhosts/integration/htdocs" />
  <property name="svnUser"      value="your_svn_user" />
  <property name="svnPass"      value="your_svn_password" />

  <!--fetch current revision of working copy -->
  <target name="getOriginalRevision">
    <svnlastrevision workingcopy="${baseDir}" propertyname="svnOriginalRevision" />
    <echo msg="Current revision of this working copy: ${svnOriginalRevision}" />
  </target>

  <!-- update local working copy -->
  <target name="svnUpdate">
    <svnupdate username="${svnUser}" password="${svnPass}" nocache="true" todir="${baseDir}" />
  </target>

  <!-- fetch revision of working copy again -->
  <target name="getCurrentRevision" depends="svnUpdate">
    <svnlastrevision workingcopy="${baseDir}" propertyname="svnCurrentRevision" />
    <echo msg="Revision of this working copy after svn update: ${svnCurrentRevision}" />
  </target>

  <!-- create the tarball from the integration VHost-->
  <target name="tar">
    <echo msg="Creating archive..." />
    <tar destfile="${baseDir}/builds/build-${buildVersion}.r${svnCurrentRevision}.tar.gz" compression="gzip">
      <fileset dir="${htdocsDir}">
        <include name="**" />
        <exclude name="reports/**" />
      </fileset>
    </tar>
    <echo msg="Build copied and compressed into directory!" />
  </target>

  <!-- updates the demo VHost after all tests have succeeded -->
  <target name="updateDemo">
    <!-- change baseDir to demoDir -->
    <property name="baseDir" value="${demoDir}" override="true" />
    <phingcall target="svnUpdate" />
  </target>

  <!-- compare original and current working copy revisions -->
  <target name="runTests"  depends="getOriginalRevision,getCurrentRevision">
    <if>
      <equals arg1="${svnOriginalRevision}" arg2="${svnCurrentRevision}" />
      <then>
        <echo msg="Working copy is up to date." />
      </then>
      <else>
        <echo msg="Updated working copy from SVN, starting integration tests." />
        <phing phingfile="test.xml" haltonfailure="true" />
        <phingcall target="tar" />
        <phingcall target="updateDemo" />
      </else>
    </if>
  </target>
</project>

Let's go through this script step by step.

1. The target <getOriginalRevision> gets called first.
   Thankfully phing provides a ready-made task to retrieve the Subversion revision of a given path. We store that revision in the property ${svnOriginalRevision}.
2. Next, the <svnUpdate> target performs a Subversion update on the whole integration environment, ensuring we get any updates that have been committed since the last run. Since this environment will never be edited there will never be any merge conflicts.
3. Then the< svnCurrentRevision> target fetches the Subversion revision number of the integration environment again and stores it in a property, this time in ${svnCurrentRevision}. Note that this target is identical to <getOriginalRevision>, except for the property used.
4. After we have ensured that the integration environment is up to date, the main target <runTests> compares the ${svnOriginalRevision} with ${svnCurrentRevision} to check if a newer version has arrived. If not, this continuous integration loop stops. No need to test again what was already tested.
5. If the execution did not stop - that is, if a newer source code version has arrived - we use the <phing> task to execute the test.xml build file we introduced in the third part of this series.
   The buildAndUpdate.xml uses the same properties as the test.xml and because of that the latter will now run in the integration environment.
6. In case the test.xml script runs through without error a tarball is created automatically, containing everything in the htdocs directory.
7. Last not least we perform a SVN update on the demo environment.

Note that the test.xml build file will now send an email if one of the tests fails!

This is due to the fact that the updateAndBuild.xml sets the ${masterFile} property to a value that causes an <if> task in test.xml (line 112) to pass. This way the developers will automatically be alerted if the current revision in the integration environment is broken. If the test.xml is executed manually the if-condition will not apply and no mail is sent.

Testing the phing script.

You can direct your SSH shell to /var/www/vhosts/integration and execute

phing -f buildScripts/updateAndBuild.xml

but this will not do much good since the integration environment is up to date.

What I would recommend is to create a new file in the development environment /var/www/vhosts/default and fill it with some arbitrary text, only to add and commit that file to Subversion. Then return to the integration environment and run the same command.

Voila! The phing script updated the integration environment, recognized the update and started all tests (this time in the integration environment). Once more, a reports folder has been created (again in the integration environment) and filled. A new folder /var/www/vhosts/integration/builds will have been created, containing a .tar.gz file with the latest build.

Note that this folder may get huge very fast, depending on project complexity. It is probably advisable to deactivate the automatic tarball generation task for a real life environment.

If you add an error to one of the tests, you should get an email. Double check the ${errorMail} property in test.xml and make sure the server can successfully sent mails if you don't.

The last step for you to do is to set up a cronjob to run the phing command periodically. Make sure the cron environment is forwarding the X server display and otherwise is as close as your shell environment as possible, because otherwise you will run into all sorts of issues. This is the nature of cron...

This concludes my tutorial on continuous integration for PHP.

Leave me a comment if you liked the series, if you have questions or suggestions for improvements. I will work on a fifth and last article about my own thoughts for possible improvements, to be published in a bit.

This will be a series of several posts where I will try to cover all basic aspects of continuous integration for PHP:

An informational article about what continuous integration is all about.

After that I'll give you a step-by-step guide on how to set up a continuous integration environment using a virtual environment.

I will provide you with an example build script for automated local tests in the third article.

The fourth article will put all things together and establish continuous integration.

This article will cover the first topic "What is continuous integration all about?".

Continuous integration is an important aspect in agile development methodologies and test driven development.

The whole idea behind continuous integration is to have a process running in the background that integrates and tests the various pieces of source code that you generated continuously and automatically. In a nutshell, that's it.

Why would we want to automate our testing? Well I believe that is obvious:

1. Testing is hard.
2. Testing is boring.
3. Testing is time consuming.

All three aspects - and there might be more - make it impractical to achieve complete test coverage manually even in a medium sized project. It becomes much more feasible to invest the effort of setting up an automated test environment and writing the tests once which from then on will run day after day after day.

Which components are required for continuous integration with PHP?

There are a few alternatives to what I'm going to discuss here (I give some of them in parenthesis) but roughly you'll want:

• Unittest scripts that will cover your regression tests. PHPUnit is the tool of choice here (SimpleTest also does a good job)
• A build tool such as phing. It will automate the building of your test environment.
• For the continuity aspect a cronjob is enough. (CruiseControl & phpUnderControl are much more sophisticated. Check them out if a cronjob is not cutting it for you)
• You will need some sort of revision control. My tool of choice is Subversion. (CVS, and many others are available)
• I strongly recommend using your own integration environment over testing on your development machine. VMWare is a great free virtualizer.

Is there more that we can do with continuous integration?

Hell yes! Here are a few examples:

• Frontend testing (yes, automated) with Selenium IDE and Selenium Remote Control. Note that you will also need some sort of graphical environment for that. Xvfb is usually good enough.
• You could generate your API doc for PHP with phpDocumentor.
• You certainly can generate an API doc for JavaScript as well.
• You could perform unittests for JavaScript or your jQuery plugins.
• and last not least why don't you push everything on a staging system for your client to see when everything goes well?

Now that we have that covered...

How does an environment with continuous integration look like?
How do we have to work in order to get the most out of it?

As usual, there are many ways to do this. My personal approach consists of two, better three environments:

First, the development machine(s). That is where we develop our code.
Second, the integration machine, where the continuous integration is happening.
Third, optionally a system for staging. This gives you constant feedback from the client.

Overall the workflow looks like this:

My continuous integration environment

The 'devel' environment is local on the developers' machine. A virtual machine as I pointed out earlier but that is not relevant at this moment. If there is more than one developer, each will have his own 'devel' environment. The development (1) is done locally. All developers commit to and update  from a central Subversion repository (2).

The 'int' (integration) environment usually sits on a different piece of hardware. I prefer the 'int' and the 'stage' system to be on two different virtual hosts on one machine. However your setup might be, on the 'int' environment a cronjob is running continuously and pulls the latest revision of the product out of Subversion (3). If a new revision is found, the integration (4) starts, running all tests, generating all reports, writing API docs and so on.

If an error occurs during the integration test, a mail is being sent to all developers. I've heard of teams who change the color of an ambient orb to red in this case, or switch on a lava lamp. Do as you please. A mail is a good start, however. The team can then utilize all reports on 'int' (available thru the apache virtual host) to inspect the problem and fix it.

Because the integration is continuous, the feedback about a failing test is fast.

Not much code has to be revised in order to find and fix the bug.

Only when the integration runs without errors, the step (5) starts: Deploying that revision to the 'stage' environment where the client can test it. If you think that might probably not be a good idea in your case / with your client, you might be right. Releasing every other week, at the end of a sprint works just as well. This last step (5) is optional.

That concludes the first part in my series on setting up continuous integration for PHP. Continue with the second part: A installation howto.