composer pain.json

July 15th, 2013

So tonight I decided to try and use Composer, a “dependency management” tool for PHP. I’ve had my eye on doing this for a while, and finally got to it tonight.

I use a custom rolled version of PHP 5.5, because I want to actually use the new functionality that PHP has finally added. So, on to the install I went.

curl -sS https://getcomposer.org/installer | php

Already, the first critique here, I’m downloading a file and piping it right on into PHP. I’m not going to go into that, because this guy already has. So, on to the next step, after I took a quick poke through the installer code I downloaded.

Woops, need to recompile my PHP binary without --disable-phar. Now, done that, I ran the installer again, and had a clean install this time.

Next step, create a composer.json file after running through the Composer install steps. I wanted to use Zend Framework, so I copied the snippets of composer JSON from this site and assembled them into a coherent composer file (not the easiest thing, since the Composer manual didn’t seem to have a nice example file linked somewhere that I could find).

My basic composer.json file:
[code]
{
"repositories": [
{
"type": "composer",
"url": "https://packages.zendframework.com/"
}
],
"require": {
"zendframework/zend-config": "2.*"
}
}
[/code]

Next, ran a composer validate composer.json on my file, and got this output:
composer.json is valid for simple usage with composer but has strict errors that make it unable to be published as a package:
See http://getcomposer.org/doc/04-schema.md for details on the schema
name : is missing and it is required
description : is missing and it is required
No license specified, it is recommended to do so. For closed-source software you may use "proprietary" as license.

After a quick look over the messages, I decided to soldier on anyway, crossed my fingers, and ran composer install.

The installer seemed to laugh at me after it downloaded the files, and told me that I needed to --enable-zip before I could install the packages. THANK YOU COMPOSER, for mentioning that on install.

Another recompile, and finally I managed to get Composer to behave.

Lessons Learned

  • Composer needs both PHAR and Zip enabled to work, but doesn’t like to mention those things
  • Custom PHP is fun to do, but can be a pain when you need to turn things on that you didn’t plan on using

The Smallest Database Problems

July 11th, 2013

A few months ago, after some reports of lag with one of the systems in a distributed architecture I built and maintain, I discovered how important it was to pay attention to how queries were constructed. One day, near the end of a shift, after some complaints from staff members, I observed a high server load in the database server that backs this architecture, and traced it back to its source.

One service (nanny) that runs in this architecture was used to monitor and maintain the health of the processes on each server. This service depended on some information from the database, which was fetched by another service that processes RPC (remote procedure calls). Digging into the RPC service, I discovered several problems that weren’t obvious when it was first built. These problems became more obvious as more systems were added to the pool, increasing the load on the RPC service in question.

At it’s peak, the database server backing this process was showing a peak load of 3.2 and an average of over 1. After digging into it and fixing the problems, the server now sits at a peak load of 0.18 and an average load of 0.04.

Fixes Made

  • Modified nanny service to only request some service information on every other run instead of every run.
    This reduced the overall load placed on the RPC service.
  • Added a LIMIT to the RPC service query in question.
    Before applying the LIMIT, this query was generating around 100k results, but only being used to get the latest information.
  • Added a composite index to improve the query time.
    This was the single best gain in query speed, resulting in the query time going from a maximum of 1.7 seconds to a maximum of 0.02 seconds.

Tools
tload – A tool that generates a ASCII CPU load graph in a console.

Lessons Learned

  • Always limit your queries if you only fetch one row from them.
  • Revisit your queries to examine and update indexing to be more suitable for the data and access patterns.
  • Visualization over time is helpful in diagnosing problems.

Strip Whitespace Out Of A File

June 6th, 2013

sed is a wonderful tool. Just used this command today to remove most of the whitespace from a file.

sed 's/ //g' test.txt > test2.txt

Making A USB Hard Drive Bootable

June 25th, 2011

Vista and Windows 7 DVDs include a utility to make a USB hard drive bootable so you can install from them.

The command for this is:
[code]E:\boot\bootsect.exe /nt60 F:[/code]
where E is the DVD drive, and F is the hard drive that you want bootable.

Once you’ve done this, you can copy the contents of the DVD to the USB hard drive, and install from that. Great for when you didn’t buy a SATA DVD drive for your new computer 😀

WordPress - Entries (RSS) and Comments (RSS) - © 2011 Ben Dauphinee