composer pain.json

July 15th, 2013

So tonight I decided to try and use Composer, a “dependency management” tool for PHP. I’ve had my eye on doing this for a while, and finally got to it tonight.

I use a custom rolled version of PHP 5.5, because I want to actually use the new functionality that PHP has finally added. So, on to the install I went.

curl -sS https://getcomposer.org/installer | php

Already, the first critique here, I’m downloading a file and piping it right on into PHP. I’m not going to go into that, because this guy already has. So, on to the next step, after I took a quick poke through the installer code I downloaded.

Woops, need to recompile my PHP binary without --disable-phar. Now, done that, I ran the installer again, and had a clean install this time.

Next step, create a composer.json file after running through the Composer install steps. I wanted to use Zend Framework, so I copied the snippets of composer JSON from this site and assembled them into a coherent composer file (not the easiest thing, since the Composer manual didn’t seem to have a nice example file linked somewhere that I could find).

My basic composer.json file:
[code]
{
"repositories": [
{
"type": "composer",
"url": "https://packages.zendframework.com/"
}
],
"require": {
"zendframework/zend-config": "2.*"
}
}
[/code]

Next, ran a composer validate composer.json on my file, and got this output:
composer.json is valid for simple usage with composer but has strict errors that make it unable to be published as a package:
See http://getcomposer.org/doc/04-schema.md for details on the schema
name : is missing and it is required
description : is missing and it is required
No license specified, it is recommended to do so. For closed-source software you may use "proprietary" as license.

After a quick look over the messages, I decided to soldier on anyway, crossed my fingers, and ran composer install.

The installer seemed to laugh at me after it downloaded the files, and told me that I needed to --enable-zip before I could install the packages. THANK YOU COMPOSER, for mentioning that on install.

Another recompile, and finally I managed to get Composer to behave.

Lessons Learned

  • Composer needs both PHAR and Zip enabled to work, but doesn’t like to mention those things
  • Custom PHP is fun to do, but can be a pain when you need to turn things on that you didn’t plan on using

Globally Scoped

July 14th, 2013

Global variables are a code smell. This statement is both true and false. What is true is that they are a problem when they are no longer the right variables to be global.

Today I had to re-factor a function to be able to use multiple servers inside one script instance. Initially, the server connection was created and stored in one variable that was then referenced. This worked well, because the code that called this function didn’t have to worry about what server connection was involved.

Of course, the problem started when I added a second server into the mix. Now, this function had to be able to access either server, selected by the code in some of the places that called it. The easiest way to solve this would have been to keep the global and add a second optional parameter to override it, but that would not be as clean a fix as just fixing the global.

And so I re-factored the function call to pass in the server connection to use. This makes the function more flexible, because I can add new servers to the mix and just pass in the variable for the one I want to use. The other benefit is that in eliminating the global, this function is now finally completely self-contained, making it that much easier to test and fix.

Lesson Learned
Globals can be useful, but will make it harder to properly test a function, and complicates matters if the function that uses them is widely used and has to be changed.

Choose carefully how to use them.

The Smallest Database Problems

July 11th, 2013

A few months ago, after some reports of lag with one of the systems in a distributed architecture I built and maintain, I discovered how important it was to pay attention to how queries were constructed. One day, near the end of a shift, after some complaints from staff members, I observed a high server load in the database server that backs this architecture, and traced it back to its source.

One service (nanny) that runs in this architecture was used to monitor and maintain the health of the processes on each server. This service depended on some information from the database, which was fetched by another service that processes RPC (remote procedure calls). Digging into the RPC service, I discovered several problems that weren’t obvious when it was first built. These problems became more obvious as more systems were added to the pool, increasing the load on the RPC service in question.

At it’s peak, the database server backing this process was showing a peak load of 3.2 and an average of over 1. After digging into it and fixing the problems, the server now sits at a peak load of 0.18 and an average load of 0.04.

Fixes Made

  • Modified nanny service to only request some service information on every other run instead of every run.
    This reduced the overall load placed on the RPC service.
  • Added a LIMIT to the RPC service query in question.
    Before applying the LIMIT, this query was generating around 100k results, but only being used to get the latest information.
  • Added a composite index to improve the query time.
    This was the single best gain in query speed, resulting in the query time going from a maximum of 1.7 seconds to a maximum of 0.02 seconds.

Tools
tload – A tool that generates a ASCII CPU load graph in a console.

Lessons Learned

  • Always limit your queries if you only fetch one row from them.
  • Revisit your queries to examine and update indexing to be more suitable for the data and access patterns.
  • Visualization over time is helpful in diagnosing problems.

Zend DB Query Dump

June 25th, 2013

If you ever need to see the SQL that Zend DB has created, so that you can run it manually or any other reason, here’s how to do it.

[php]
// Turn on the profiler
$db->getProfiler()->setEnabled(true);

// Run your query
$db->update(‘someTable’, array(
‘thisField’ => 0.12551,
‘anotherField’ => ‘xxxxxxxxxxxx’
), ‘whatId = ‘ . $cdsDB->quote(‘james’));

// Output the last query
echo "\n" . $db->getProfiler()->getLastQueryProfile()->getQuery() . "\n";

// And the parameters for it
print_r($db->getProfiler()->getLastQueryProfile()->getQueryParams());

// Then turn the profiler off, unless you want to do more
$db->getProfiler()->setEnabled(false);

/*
And the output is

UPDATE `someTable` SET `thisField` = ?, `anotherField` = ? WHERE (whatId = 186459)
Array
(
[1] => 0.12551
[2] => xxxxxxxxxxxx
)
*/
[/php]

Strip Whitespace Out Of A File

June 6th, 2013

sed is a wonderful tool. Just used this command today to remove most of the whitespace from a file.

sed 's/ //g' test.txt > test2.txt

WordPress - Entries (RSS) and Comments (RSS) - © 2011 Ben Dauphinee