PHP Log2Files Advanced Logger v 1.0

This is main documentation file for Kemu Log2Files Advanced Logger v 1.0.

Introduction

This logger is an advanced library, which dumps PHP-Application logs to file in production environment.

The main purpose is to ensure the logger is very fast and so can be used in production environment.
Multiple PHP execution threads log to consecutive files at the same time, to avoid synchronization and locking problems.
There is no need to do any additional setup. Just pure PHP (no mysql even!) and some simple API calls.

Why

Logging in simple application, which is working for one user on a development server is extremely easy.
Just open a file, write to it, close (PHP: fopen(), fwrite(), fclose) or even simply file_put_contents()... then open it in any text viewer, and its done - you've got the log. No library, nor framework is really needed.

However, in real-time, multi-user, multi-script production environment, this won't work - simply because there are tens of scripts executing at the same time in different threads or even processes:


For most of advanced applications, the scripts start and finish in very non-trivial (not to say "random") order:


Logging engine needs synchronization - and the synchronization itself should have minimal impact to the script timings.
The K_Log flow is synchronized only in one very small piece of code - during log construction it takes the unique number which is synchronized by flock().
So the script waits only a few micro-seconds or doesn't wait at all (if the lock is ready-to-take) - this is almost fastest possible method in PHP (don't even try to compare it to any mySQL-made logs).
There is also some other synchronization point at the finish of a script, called "Log compacting and clean up" - but this part is executed after the script flushes its output, so it have almost no performance impact at all.

SUMMARY: K_Log ensures that each execution log have its own unique id, is consistent, is not fragmented and is stored even if the PHP script aborts for some reason.

Features

• the simplicity
  
  • simple "open log with some configuration" and then "write to log"
  • no additional dependencies (no PEAR, no frameworks), just one PHP file with one class to include in your script
  • two simplicity levels:
    
    • simple mode: the textual form - doesn't need any special parsing
    • advanced mode: the BJSON form - stores all data with binary format which is very simple to parse and traverse. This can be used by advanced stats and monitoring tools.
  • can be used with almost anything else, easily includable into any other library, framework or existing application
• performance
  
  • minimal impact to the script timing,
  • no SQL, no databases, just PHP and filesystem,
  • aware of multi-threading, multiple calls from same web-client at the time, multiple scripts running at a time with very various duration (one in 10ms, while the other in 10secs),
  • very little thread/process switching and synchronization, allow OS caches (no flushes etc.),
  • so timing-cheap, that log-levels definition is not really needed (simply log all what may be needed),
• dedicated for production environment (megabytes of log in an hour, tens of scripts executing at a time),
• dedicated API for handling arrays (only values) and maps (key=>value),
• pre-made implementation for storage of: sessions, cookies, get&post, time of whole script execution,
• easy to transport storage form (copy, delete, etc.): no thousands of files,

How it works

There are some simple steps the K_Log follow:

• Log construction:
  Each time the new K_Log object is constructed, it receives its own unique index - which is simply the incrementing integer number.
  This number identifies particular script execution session which would generate one execution log.
  The execution log is simply a textual file (in simple mode) or binary file(in advanced mode).
• Logging
  Next the PHP script calls logging methods to store its log entries.
  Every call to one of logging methods adds one entry of given data (e.g. a string, a number, a key=>value pair, an array, a script timing stamp etc.).
  Each log entry is stored as a text in "simple mode" or as binary BJSON record in "advanced mode".
• Log compacting and clean up
  After the PHP script finishes, the execution log may have few bytes, up to few megabytes depending of size of logged data.
  To avoid having thousands of small files (which may be difficult to mantain, download, archive etc.) the execution log is merged to a bigger file called "storage file" (default size is 16MB).
  In simple mode, each execution log stored in storage file is delimeted by some pre-configured string (default is: '==*===*===*===*===*=').
  In advanced mode, each execution log is simply one BJSON array. BJSON array header contains its size in bytes, so it's extremelly easy to traverse through even very-big storage files.

Configuration

The configuration of the Logger should be provided in the constructor as an associative array (key=>value).
There is only one mandatory field (logPath) which must be filled... All other fields are optional - they have default values.

Ensure that all paths passed in config are writable by script.
For your own convenience - give the logs they own separate directory (e.g. 'logs/log').

Config fields

• logPath [MANDATORY] - e.g. '/logs/theLog_'
• storagePath [OPTIONAL, default is taken from 'logPath' with suffix '_store_'] - e.g. '/logs/storedLog_'
• storageFileMaxSize [OPTIONAL, in bytes, default is 16MB] - e.g. 16*1024*1024 which equals to 16MB.
  Give big number here if you log "everything".
  Give small number, if you log only some small specific things.
• advancedMode [OPTIONAL, 0 or 1, default is 0]
  
  • If "1" each log entry is serialized in "advanced" way using BJSON (read more about "advanced mode").
    This should be used for non-trivial logs, where whole data structures may be stored and advanced parsing of log is needed (e.g. for stats).
  • If "0" (default) the log is written as plain text value using "entryDelimeter" after each entry (as a separator) and "logSeparator" after each executed log script.
  Note: advanced mode should never be mixed with simple mode in the same log file, because this will break the storage files consistency.
  
• startTime [OPTIONAL, default is taken from call to microtime(1)] - by setting this value, you can setup the startTime of your script different that the one generated during k_log construction.
  This may be very useful, if the K_Log instance is created a bit later than the real script start point (see examples).
• logPredefinedValues [OPTIONAL, 0 or 1, default is 0] - if enabled, each execution log is started with some default variables dump:
  arrays _GET, _POST, _COOKIE, _SERVER and unix timestamp are logged.

Config fields in simple mode

These are ignored in advanced mode.

• entryDelimeter [OPTIONAL, string, default is chr(10) (linux line-break)] - the string that will be put after each log entry in "simple" mode.
• logSeparator [OPTIONAL, string, default is defined later] - the separator that will be used for joining logs to a "store file" in "simple" mode.
  For default this is '==*===*===*===*===*=' plus entryDelimeter.

Configuration examples

The simplest log possible:
would create a log object, which will log execution-log files as myLog000001, myLog000002, etc. and then will compact them to myLog_store_ files

The not-so-simple code:
would create a log object, which will log execution-log files as data/logs/log000001, data/logs/log000002, etc. and then will compact them to data/logs/logstored_ files. Log stored files will be rotated (new one would be created) after 8 KiB (8*1024 bytes).

The config with specified delimeters:
would create a log object similar to previous one, but after each log entry the windows line-break ("\r\n") will be added. Also consecutive execution-logs will be separated by "----\r\n" string.

The config for BJSON logs:
would log in binary mode.

Usage

Usage examples

After construction of a log, simply call log() methods to add entries:

Demo

The link to online demo would be provided in next few days.

API Reference

new K_Log($config)

The constructor of K_Log.
It interprets provided config, then starts log.
Read more about $config in CONFIGURATION section.

function generateFilePathForLog($idx)

Returns the full path to log file of given index, using file-name generation template.

function getCurrentLogFilePath()

Returns the path to the file log is writing to.

function getWholeLog()

Returns the current execution log as a string (even in BJSON mode).
This may be useful, because the file is flock'ed by the log, so direct file read may be impossible (especially on Windows servers, where flock is mandatory).

function log($entry)

Adds the entry to the log.

function logArray($in)

Logs array values (keys are ignored).
This call forces the array to be dumped as "indexed array" - so keys simply are ignored - opposite is logMap() call.

function logMap($in)

Logs array as map of key=>value.

function logAsString($entry)

Adds the entry to the log as string.
This call forces conversion of variable to string before adding to log.

function logAsInt($entry)

Adds the entry to the log as int.
This call forces conversion of variable to integer before adding to log.

function logKeyValue($key, $value)

Adds the entry to the log as a pair key=>value.
The $key and $value may be any type.
In BJSON mode, this is encoded as a map with only one key=>value entry.

function logTime($label)

Logs the named timestamp.
The time is number of microseconds elapsed from log start (the log start time may be overriden by setting "startTime" in configuration passed during log construction).
In BJSON mode, this is encoded as an array with three elements:
- the hardcoded string 'time',
- the $label,
- the integer time,

function shutdown()

A PHP shutdown callback - It is not a log finish function !
This should never be called directly - it's called after PHP script finishes.

Other

FAQ

Q: Can I have two or more logs created in one script ?
A: Yes ! You may have as many different logs as you want - each configured independently.
E.g. you may log user actions to one log, and some time-stats to different one.
Just ensure they have different filenames or filepaths - the 'logPath' field in configuration passed to constructor.

Q: Why there are no timestamps in each log entry ?
A: Auto-adding timestamps into each entry would improve the profiling abilities of the log. However, this would also make log a lot bigger, harder to parse and the logging itself would be slower. You can add a timestamp by simply calling logTime().

Roadmap

The Kemu Log2Files is under continous improvement process.
We will soon:
- provide the online demo of K_Log,
- publish some performance tests and comparison to mysql calls,

License

All copyrights reserved by Kemu Studio, Pietrzak 'yosh' Roman (visit yosh.ke.mu).
Source code and documentation available from CodeCanyon