Frank DENIS random thoughts.

DNS records and TTL - how long does a second actually last?

2011-11-17T00:00:00+01:00

In a DNS zone, every record carries its own time-to-live, so that it can be cached, yet still changed if necessary.

This information is originally served by authoritative servers for the related zone. The TTL is represented as an integer number of seconds.

At first sight, the mechanism looks straightforward: if the www.example.com record has a TTL of 30, it's only valid up to 30 second, and caches must fetch it again if requested after this delay.

Chained caches

DNS caches can be chained: instead of directly querying authoritative servers, a cache can forward queries to a another server, and cache the result.

Having 3 or 4 chained caches is actually very common. Web browsers, operating systems and routers can cache and forward DNS a query. Eventually, this query will be sent to an upstream cache, like the ISP cache or a third-party service like OpenDNS. And these caches can also actually hide multiple chained caches.

In order to respect the original TTL, caches are modifying records as they forward them to clients. The response to a query that has been sitting in a cache for 10 second will be served with a TTL reduced by 10 second. That way, if the original TTL was 30 second, the whole chain is guaranteed to consider this record as expired as the same time: the original meaning of the TTL is retained no matter how many resolvers there are in the way.

Well, not exactly. A TTL is just a time interval, not an absolute date. Unlike a HTTP response, a DNS response doesn't contain any timestamp. Thus, requests processing and network latency are causing caches to keep a record longer than they actually should in order to respect the initial TTL.

A TTL being an integer value makes things even worse: a chain of N caches can introduce a N second bias.

In practice, this is rarely an issue: TTLs as served by authoritative servers are considered indicative, and not as something to depend on when accurate timing is required.

The decay of a TTL, as seen by different pieces of software

How does the TTL of a record served by a DNS cache decays over time?

Surprisingly, different implementations exhibit different behaviors.

For a record initially served with a TTL equal to N by authoritative servers:

Google DNS serves it with a TTL in the interval [0, N-1]
dnscache is serving it with a TTL in the interval [0, N]
Unbound serves it with a TTL in the interval [0, N]
Bind serves it with a TTL in the interval [1, N]
PowerDNS Recursor always serves it with a TTL of N

Hold on... Does it mean that how frequently an entry will actually be refreshed depends on what software resolvers are running?

Sadly, yes. Given a record with a TTL equal to N:

Bind and PowerDNS Recursor refresh it every N second if necessary
Unbound and dnscache are only refreshing it every N + 1 seconds at best.

TTL Zero

Although a TTL of zero can cause interoperability issues, most DNS caches are considering records with a TTL of zero as records that should not be cached.

This perfectly makes sense when the TTL of zero is the original TTL, as served by authoritative servers.

However, when a cache artificially changes the TTL to zero, it changes a record that had been designed for being cached to an uncachable record that contaminates the rest of the chain.

dnscache and a 1 second record

To illustrate this, a Linux box has been setup with a local DNS cache. Unbound has been chosen, but the last component of the chain actually makes little difference. Even web browsers caches have a very similar behavior.

Queries are forwarded to an upstream cache on the same LAN, running dnscache, and outgoing queries are recorded with ngrep. The same query, whose response has a TTL of 1, is made at a 10 queries per second rate.

The X axis represents outgoing queries, whereas the Y axis is the time elapsed since the previous query.

Even though a TTL of 1 is served by authoritative servers for this record, a large amount of responses are cached and served for 2 second. This is due to the local Unbound resolver.

However, there is also quite a lot of responses that haven't been cached at all. It happens when dnscache serves a response with a TTL of 0. Since it happens one third of the time, this is suboptimal and not on par with the intent of the authoritate record, which is served with non-zero TTL.

And although dnscache and Unbound are handling TTLs the same way, we can't expect their caches to be perfectly synchronized. When the local cache considers a record as expired and issues an outgoing query, the upstream server can consider it as not expired yet, just in the middle of the last second. What we get is a constant race between caches, causing jitter and outgoing queries sent after 1 second.

How different implementations can affect the number of outgoing queries

The following experiment has been made by observing outgoing queries when sending 10,000 queries for a record with a TTL of 2, at an average 10 qps rate, to a local cache, using Google, OpenDNS and Level 3 as upstream resolvers.

Here is the number of outgoing queries that were required to complete the 10,000 local queries using different services:

Service	Outgoing queries
Google DNS	1383
OpenDNS	632
Level 3	388

When using Google DNS (blue dots), queries are effectively never cached more than 2 second, even locally. This is due to the max TTL returned by Google DNS being the initial TTL minus one.

When Google DNS returns a TTL of zero, we observe the same jitter and the same slew of queries that couldn't be served from the local cache.

OpenDNS (orange dots) has the same behavior as dnscache and Unbound., with TTLs in the [0, N] interval. Our initial TTL with a value of 2 actually causes 3, 2, and 1 second delays between request, plus a sensitive amount of consecutive outgoing queries due a null TTL.

Level 3 (green dots) is running Bind, which returns a TTL in the [1, N] interval. Bind caches records one second less than dnscache and Unbound. Because our local resolver is Unbound, queries received with a TTL of N are actually cached N + 1 second. But as expected, the frequency of required outgoing queries very rarely exceeds 3 second.

What the correct behavior is, is out of scope of this article. All major implementations are probably correct.

But from a user perspective, with only 2 caches in the chain, and for a given record, the same set of queries can require up to 3.5 more outgoing queries to get resolved, depending on what software the remote cache is running.

With CDNs and popular web sites having records with a very low TTL (Facebook has a 30 second TTL, Skyrock has a 10 second TTL), the way a cache handles TTLs can have a sensible impact on performance. That said, some resolvers can be configured to pre-fetch records before they expire, effectively mitigating this problem.

The OSX cache

MacOS X provides a system-wide name cache, which is enabled by default.

Its behavior is quite surprising, though. It seems to enforce a minimum TTL of 12.5 seconds, while still requiring some outgoing queries delayed by the initial TTL value, and some consecutive ones. Resolving the 10,000 queries from the previous test took only an average of 232 outgoing queries.

Using Google DNS, OpenDNS and Level 3 as a remote resolver produce the same result, with the exception on Level 3 (Bind) avoiding frequent (less than 1 sec) consecutive queries during the time other return a TTL of zero.

Programmatically changing network configuration on OSX

2011-08-14T00:00:00+02:00

Programmatically changing DNS resolvers, IP addresses and proxies on OSX isn't rocket science but finding documentation on that is like looking for a needle in a haystack.

Changing the DNS configuration could have been as simple as updating /etc/resolv.conf, as OSX does provide an /etc/resolv.conf file. But unlike every other Unix variant out there, OSX doesn't actually read this file, as it is only there for compatibility with some legacy tools.

Meet configd

"The configd daemon is responsible for many configuration aspects of the local system. configd maintains data reflecting the desired and current state of the system, provides notifications to applications when this data changes, and hosts a number of configuration agents in the form of loadable bundles."

Think about it as a centralized key/value registry. Applications can watch for changes, add new key/value pairs and modify existing entries.

The scutil(8) command provides a command-line interface to this registry:

$ scutil
> list
  subKey [0] = Plugin:IPConfiguration
  subKey [1] = Plugin:InterfaceNamer
  subKey [2] = Setup:
  subKey [3] = Setup:/
  subKey [4] = Setup:/Network/BackToMyMac
  subKey [5] = Setup:/Network/Global/IPv4
  subKey [6] = Setup:/Network/HostNames
  subKey [7] = Setup:/Network/Interface/en0/AirPort
  subKey [8] = Setup:/Network/Service/0CE7A64C-9174-41AC-B634-41F4B6B5FE02
  subKey [9] = Setup:/Network/Service/0CE7A64C-9174-41AC-B634-41F4B6B5FE02/IPv4
  subKey [10] = Setup:/Network/Service/0CE7A64C-9174-41AC-B634-41F4B6B5FE02/IPv6
  subKey [11] = Setup:/Network/Service/0CE7A64C-9174-41AC-B634-41F4B6B5FE02/Interface
  subKey [12] = Setup:/Network/Service/0CE7A64C-9174-41AC-B634-41F4B6B5FE02/Proxies
...  
  subKey [32] = Setup:/Network/Service/527FC752-8B1D-4111-A205-4E81F757704E
  subKey [33] = Setup:/Network/Service/527FC752-8B1D-4111-A205-4E81F757704E/IPv4
  subKey [34] = Setup:/Network/Service/527FC752-8B1D-4111-A205-4E81F757704E/IPv6
  subKey [35] = Setup:/Network/Service/527FC752-8B1D-4111-A205-4E81F757704E/Interface
  subKey [36] = Setup:/Network/Service/527FC752-8B1D-4111-A205-4E81F757704E/Proxies
...
  subKey [58] = Setup:/System
  subKey [59] = State:/IOKit/LowBatteryWarning
  subKey [60] = State:/IOKit/Power/CPUPower
  subKey [61] = State:/IOKit/PowerAdapter
  subKey [62] = State:/IOKit/PowerManagement/Assertions
  subKey [63] = State:/IOKit/PowerManagement/CurrentSettings
  subKey [64] = State:/IOKit/PowerManagement/SystemLoad
  subKey [65] = State:/IOKit/PowerManagement/SystemLoad/Detailed
  subKey [66] = State:/IOKit/PowerSources/InternalBattery-0
  subKey [67] = State:/IOKit/SystemPowerCapabilities
  subKey [68] = State:/Network/BackToMyMac
  subKey [69] = State:/Network/Connectivity
  subKey [70] = State:/Network/Global/DNS
  subKey [71] = State:/Network/Global/IPv4
  subKey [72] = State:/Network/Global/Proxies
  subKey [73] = State:/Network/Interface
  subKey [74] = State:/Network/Interface/en0/AirPort
  subKey [75] = State:/Network/Interface/en0/CaptiveNetwork
  subKey [76] = State:/Network/Interface/en0/IPv4
  subKey [77] = State:/Network/Interface/en0/IPv6
  subKey [78] = State:/Network/Interface/en0/Link
  subKey [79] = State:/Network/Interface/lo0/IPv4
  subKey [80] = State:/Network/Interface/lo0/IPv6
  subKey [81] = State:/Network/Interface/p2p0/Link
  subKey [82] = State:/Network/Interface/utun0/IPv6
  subKey [83] = State:/Network/MulticastDNS
  subKey [84] = State:/Network/NetBIOS
  subKey [85] = State:/Network/PrivateDNS
  subKey [86] = State:/Network/Service/527FC752-8B1D-4111-A205-4E81F757704E/DHCP
  subKey [87] = State:/Network/Service/527FC752-8B1D-4111-A205-4E81F757704E/DNS
  subKey [88] = State:/Network/Service/527FC752-8B1D-4111-A205-4E81F757704E/IPv4
  subKey [89] = State:/Users/ConsoleUser
  subKey [90] = com.apple.DirectoryService.NotifyTypeStandard:DirectoryNodeAdded
  subKey [91] = com.apple.network.identification
  subKey [92] = com.apple.opendirectoryd.node:/Search
  
> get State:/Network/Global/DNS

> d.show
<dictionary> {
  ServerAddresses : <array> {
    0 : 208.67.220.220
    1 : 208.67.222.222
  }
}

As we can see, the registry holds most of the network settings, including DNS settings. The Setup:/* entries contain every available network configuration (as configured in the preferences pane), whereas State:/* entries include only what's currently active.

There are global DNS settings and configuration-specific DNS setings.

Programmatically accessing the registry

The overlooked DynamicStore API, which is part of the SystemConfiguration framework, let us access the configd registry.

The following code snippet opens the registry, looks for active DNS-related entries, and updates them with a new property list containing OpenDNS resolvers.

#include <SystemConfiguration/SystemConfiguration.h>

static bool setDNS(CFStringRef *resolvers, CFIndex resolvers_count)
{
    SCDynamicStoreRef ds = SCDynamicStoreCreate(NULL, CFSTR("setDNS"), NULL, NULL);
    
    CFArrayRef array = CFArrayCreate(NULL, (const void **) resolvers,
        resolvers_count, &kCFTypeArrayCallBacks);
    
    CFDictionaryRef dict = CFDictionaryCreate(NULL,
        (const void **) (CFStringRef []) { CFSTR("ServerAddresses") },
        (const void **) &array, 1, &kCFTypeDictionaryKeyCallBacks,
        &kCFTypeDictionaryValueCallBacks);    
    
    CFArrayRef list = SCDynamicStoreCopyKeyList(ds,
        CFSTR("State:/Network/(Service/.+|Global)/DNS"));
    
    CFIndex i = 0, j = CFArrayGetCount(list);
    if (j <= 0) {
        return FALSE;
    }
    bool ret = TRUE;
    while (i < j) {
        ret &= SCDynamicStoreSetValue(ds, CFArrayGetValueAtIndex(list, i), dict);
        i++;
    }
    return ret;
}

int main(int argc, const char * argv[])
{
    CFStringRef resolvers[] = {
        CFSTR("208.67.220.220"),
        CFSTR("208.67.222.222")
    };
    setDNS(resolvers, (CFIndex) (sizeof resolvers / sizeof resolvers[0]));
    
    return 0;
}

Of course, changing the registry requires root privileges, but that's all it takes.

As new entries can be seamlessly created, you can use the registry for centralizing configuration in your own apps, or just to save previous entries before altering them.

Changing network services

The technique describe above works fine. But the new settings aren't persistent. More often than once, this actually comes in handy, in order to apply temporary settings and to rest assured that once the user reboots his machine, network settings will always be back to a sane state.

Tweaking the system configuration database is not recomended by Apple because some daemons may not be aware of a configuration change. In addition, you may want to permanently apply the new settings.

Another way to tackle this problem is to actually alter the network services.

Under OSX, a network service is a configuration for a network interface. To apply a change globally, we need to step over all network services, check whether DNS (or whatever we want to change) is a support protocol, and make the related changes.

This achieves the same thing as what one can do from the Network preferences pane.

Almost.

After changing settings this way, you will notice that ths Network preferences pane properly reflects them. However, some apps won't have caught up with the changes. For example, the /etc/resolv.conf isn't automatically updated.

The trick here is to open the system configuration registry as described above, and then to close it without having done any actual change. Just "touching" the registry will trigger observers like configd so that they can do their own magic.

An empty DNS servers list or an empty IP/network address means that DHCP will be used in order to retrieve this information. But this doesn't happen automatically as the new settings are applied. Explicitly renewing the DHCP lease is required, by calling SCNetworkInterfaceForceConfigurationRefresh() on each interface.

@implementation DNSGlobalSettings

- (BOOL) DNSSupportForNetworkService: (SCNetworkServiceRef) networkService
{
    if (!SCNetworkServiceGetEnabled(networkService)) {
        return NO;
    }
    SCNetworkInterfaceRef interface = SCNetworkServiceGetInterface(networkService);
    if (!interface) {
        return NO;
    }
    NSArray *supportedProtocols = (__bridge_transfer NSArray *) SCNetworkInterfaceGetSupportedProtocolTypes(interface);
    if ([supportedProtocols indexOfObject: (__bridge NSString *) kSCNetworkProtocolTypeDNS] == NSNotFound) {
        return NO;
    }
    return YES;
}

- (BOOL) setResolversTo: (NSArray *) resolvers forNetworkService: (SCNetworkServiceRef) networkService
{
    if ([self DNSSupportForNetworkService: networkService] == NO) {
        return 0;
    }
    SCNetworkProtocolRef networkProtocol = SCNetworkServiceCopyProtocol(networkService, kSCNetworkProtocolTypeDNS);
    if (networkProtocol == nil || ! SCNetworkProtocolGetEnabled(networkProtocol)) {
        NSLog(@"disabled");
        return FALSE;
    }
    if (![resolvers isKindOfClass: [NSArray class]] || resolvers.count == 0) {
        resolvers = nil;
    }
    if (resolvers == nil) {
        NSLog(@"Setting this service's resolvers to DHCP");
    } else {
        NSLog(@"Setting this service's resolvers to [%@]", resolvers);
    }
    NSDictionary *DNSDict = (__bridge NSDictionary *) SCNetworkProtocolGetConfiguration(networkProtocol);
    NSMutableDictionary *newDNSDict;    
    if (DNSDict) {
        NSLog(@"This interface had an existing configuration");
        if (resolvers == nil) {
            SCNetworkProtocolRef networkProtocol = SCNetworkServiceCopyProtocol(networkService, kSCNetworkProtocolTypeDNS);
            if (networkProtocol == nil || ! SCNetworkProtocolGetEnabled(networkProtocol)) {
                return TRUE;
            }
            NSLog(@"Removing protocol from configuration");
            SCNetworkProtocolSetConfiguration(networkProtocol, NULL);
            return TRUE;
        }
        newDNSDict = [NSMutableDictionary dictionaryWithDictionary: DNSDict];
        if (resolvers == nil) {
            [newDNSDict removeObjectForKey: (__bridge NSString *) kSCPropNetDNSServerAddresses];
        } else {
            [newDNSDict setValue: resolvers forKey: (__bridge NSString *) kSCPropNetDNSServerAddresses];
        }
    } else {
        NSLog(@"This interface had no existing configuration");
        if (resolvers == nil) {            
            return TRUE;
        }
        newDNSDict = [NSMutableDictionary dictionaryWithObject: resolvers forKey: (__bridge NSString *) kSCPropNetDNSServerAddresses];
    }
    
    return SCNetworkProtocolSetConfiguration(networkProtocol, (__bridge CFDictionaryRef) newDNSDict);
}

- (BOOL) touchDynamicStore
{
    SCDynamicStoreRef ds = SCDynamicStoreCreate(NULL, CFSTR("myapp"), NULL, NULL);
    CFRelease(ds);
    NSLog(@"DynamicStore updated");
    
    return TRUE;
}

- (BOOL) forceDHCPUpdate
{
    NSLog(@"Forcing DHCP update");
    NSArray *interfaces = (__bridge_transfer NSArray *) SCNetworkInterfaceCopyAll();
    for (id interface_ in interfaces) {
        SCNetworkInterfaceRef interface = (__bridge SCNetworkInterfaceRef) interface_;
        SCNetworkInterfaceForceConfigurationRefresh(interface);
    }
    return TRUE;
}

- (BOOL) applyToNetworkServices: (BOOL (^)(SCNetworkServiceRef networkService)) cb andCommit: (BOOL) commit
{
    SCPreferencesRef preferences = SCPreferencesCreate(NULL, CFSTR("myapp"), NULL);
    SCPreferencesLock(preferences, TRUE);
    SCNetworkSetRef networkSet = SCNetworkSetCopyCurrent(preferences); 
    NSArray *networkSetServices = (__bridge_transfer NSArray *) SCNetworkSetCopyServices(networkSet);
    BOOL ret = TRUE;
    for (id networkService_ in networkSetServices) {
        SCNetworkServiceRef networkService = (__bridge SCNetworkServiceRef) networkService_;
        ret &= cb(networkService);
    }
    SCPreferencesUnlock(preferences);
    if (commit) {
        SCPreferencesCommitChanges(preferences);
        SCPreferencesApplyChanges(preferences);
    }
    CFRelease(networkSet);    
    CFRelease(preferences);
    if (commit) {
        [self touchDynamicStore];
    }
    return ret;
}

- (BOOL) setResolvers: (NSArray *) resolvers
{
    [self applyToNetworkServices: ^BOOL(SCNetworkServiceRef networkService) {
        return [self setResolversTo: resolvers forNetworkService: networkService];
    } andCommit: YES];   
    [[NSNotificationCenter defaultCenter] postNotificationName: @"CONFIGURATION_CHANGED" object: self userInfo: nil];

    return TRUE;
}

@end

Redis as a recommendation engine

2011-03-10T00:00:00+01:00

"You might be interested in red toilet paper, because you bought blue and green toilet paper, and people who also bought blue and green toilet paper tend to also buy red toilet paper".

Recommendation engines are now everywhere, from e-commerce to social networks.

Graphs databases like Neo4j are probably the best way to tackle the problem. But as an alternative, let's try building a recommendation engine on Redis.

Who bought what or who is following who could be stored in Redis sorted sets.

Let's take an example: a user buys A, and also B. We increment the score of the B item in the sorted set associated to key A. And we possibly also do it the other way round.

This is a trivial way to keep a list of what items have been bought, what are the related items and how many of them there are.

On a social network, the score associated to a relationship may reflect the level of trust.

Let's suppose we have the following relationships:

B -> C
D -> C
E -> F
A -> B
A -> D
A -> E

C is referenced by B and D. B and D are referenced by A. But C isn't referenced by A yet. This is presumably something to suggest.

F might also be a good candidate as a suggestion for A. However, since there's only a single path from A to F (through E), it's probably less relevant than C.

In order to limit the number of non-relevant results, we may want to add a threshold: nodes that have less possible paths than a cutoff value shouldn't be suggested.

How to do that with Redis, and only Redis?

Here's one possible way:

Use ZUNIONSTORE in order to build a temporary aggregate of all related items (scores are summed up), and remove the reference item from the set (cylic links).
Remove everything from the aggregate that is already directly related to the reference item. One way to achieve this on a sorted set is to use ZUNIONSTORE AGGREGATE MIN with a negative or null weight for the set we want to delete, followed by ZREMRANGEBYSCORE in order to actually remove everything from the reference set and everything below the cutoff score.
Sort and optionally trim the new set with ZREVRANGE et voila!

Maybe code speaks louder than words, so here we go for some code:

require "redis"

class Recommendation
  attr_reader :db
  
  def initialize
    @db = Redis.new  
  end
  
  def bought(id)
    Item.new(self, id)
  end
  
  def suggestions_for(id, options = { })
    Item.new(self, id).suggestions(options)
  end
  
  class Item
    def initialize(recommendation, id)
      @db, @id = recommendation.db, id
    end
    
    def self.key(id) "{rec}:items:#{id}" end
    
    def key() Item.key(@id) end
    
    def tmp_key(n = 1) "{rec}:_tmp:#{@id}:#{n}" end
    
    def and_also_bought(linked_id, weight = 1)
      @db.zincrby(key, weight, linked_id)
    end
    
    def suggestions(options = { })
      cutoff = options[:cutoff] || 1
      limit = options[:limit] || 50
      tmp, tmp2 = tmp_key, tmp_key(2)
      related_keys = @db.sort(key).collect { |item| Item.key(item) }
      return [ ] unless related_keys

      res = @db.multi do |m|
        m.zunionstore(tmp, related_keys, :aggregate => :sum)
        m.zrem(tmp, @id)
        m.zunionstore(tmp2, [ tmp, key ], :weights => [ 1, -1 ],
                                          :aggregate => :min)
        m.del(tmp)
        m.zremrangebyscore(tmp2, '-inf', cutoff)
        m.zrevrange(tmp2, 0, limit - 1)
        m.del(tmp2)
      end
      res[5]
    end
  end    
end

# Example

r = Recommendation.new

r.bought('b').and_also_bought('c')
r.bought('d').and_also_bought('c')
r.bought('e').and_also_bought('f')
r.bought('a').and_also_bought('b')
r.bought('a').and_also_bought('d')
r.bought('a').and_also_bought('e')

puts r.suggestions_for('a')

Pagination and HTTP caching

2011-02-25T00:00:00+01:00

Pagination is a simple concept, but from the developer's side, it's really tricky to get right.

Adding pagination to a fixed set of documents is a piece of cake. In such a situation, lookups have no valid reason not to be O(1) unless you're blindly using SELECT...LIMIT SQL statements.

But paginating dynamic data, where documents are constantly added, removed and modified, is slightly more challenging. Alas, discussion boards, blog engines and social networks have to suck it up.

In this post, I will focus on client-side caching of pagination. Googling for "pagination http cache" doesn't yield anything but articles about server-side caching.

This is a sad state of affairs. While browsing a blog or a forum, going back and forth between pages is an instinctive behavior. A lack of client-side caching significantly degrades a user's experience, is a waste of bandwidth and burns server CPU cycles for no valid reasons.

Don't do this at home

Wordpress, vBulletin, Ning and unfortunately a lot of other forum and blog engines and services, are using the worst possible approach.

Document A is posted. Later on comes document B and a bunch of other documents: C, D, E, and so on. Let's suppose document R is the freshest one and pages are designed to hold up to 5 documents.

Browsing http://example.com displays the newest 5 documents: R, Q, P, O, N. Good and totally expected.

Things start to go wrong when a user follows the "see older posts" link, leading to the next page.

R, Q, P, O and N are on http://example.com, which is an alias for http://example.com/page1 M, L, K, J, and I are served as http://example.com/page2. H, G, F, E, and D are served as http://example.com/page3 and so on.

So, as the page number grows, the content is getting older.

This is terrible.

Inserting a new document S means means that all pages immediately become inconsistent.

http://example.com should now hold S, R, Q, P and O. http://example.com/page1 should now hold N, M, L K and J. http://example.com/page2 should now hold I, H, G, F and E.

And this is even more terrible because inserting new documents is an operation that obviously has a high chance of being constantly performed.

This kind of pagination is inefficient. Users can't reliably bookmark a page, because if they open the bookmark later on, they're might reach other documents.

It ruins SEO. And to top it off, it makes pages virtually impossible to cache.

Wordpress, vBulletin, Ning and others are solving this by explicitly preventing a browser to cache a page.

So if a user reads page 1, then goes to page 2, then goes back to page 1, downloading the full page 1 will be necessary, even if nothing has changed at all.

Granted, there are "cacheability" addons for Wordpress and vBulletin, but all they do is to sometimes issue a "304 Not Modified" reply instead of providing URIs featuring a decent time to live.

How come having to rewrite every page when something is added to a text is something that never happens in the real life?

Well... the usual way to use a notebook is to start to write at page 1. Once page 1 is full, the pen holder keeps writing, but on page 2. Once page 2 is full, he keeps writing, that time on page 3. Thus, page 1 holds the oldest content and the last page of the book holds the content that was written last.

Sounds pretty straightforward, eh?

So why not do so on a web site?

The first article id

Page numbers were designed for books. More specifically, they were designed in order to manually find content according to a table of contents.

But on a computer, we never have to flick over pages in order to find the content associated to a link. We just click a link, or touch the screen.

Hence, there's no valid reasons to use a page number in order to refer to a page.

Overblog are using URIs like:

http://example.com/40-index.html

40 doesn't mean page 40. It actually means "the page whose first article is the 40th". So /40-index.html holds documents 40, 41, 42, 43, 44 and the next 5-items page is /45-index.html.

An immediate benefit is that if a blog owner changes the number of articles per page, it doesn't break anything. Previously indexed / bookmarked / cached URIs remain totally valid, although the number of articles per page might not be initially consistent.

However, unless explicitely configured the other way round, document 40 means the 40th starting from the most recent one. Publish a new document, and ex-document 40 now refers to another document, and /40-index.html is out of date.

Overblog pages uses ETags, that properly change whenever the content of a page changes. Still, even if the content hasn't changed, a useless round-trip with a 304 reply is required every time.

Things would have been much better if document identifiers were incremental, ie. new documents get a higher id than older ones.

00f.net's initial blog engine worked that way. Pages URIs were made of timestamps:

http://example.com/1298640310.html

The content of such a page is N documents whose timestamp is greater than or equal to 1298640310.

This scheme works pretty well, because pages can get easily cached and indexed. Pages that don't get any update can keep the same URI forever.

An article added to the freshest page automatically causes the URI to change, without invalidating the previous URI.

Timestamps were good enough for this blog, but any increasing and globally unique identifier would be a good fit.

The downside of non-monotonically increasing ids is that while it's easy to jump to older content ("next page"), it can be tricky to get links to previous pages (newer content) without falling into duplicate content or HTTP redirects. But for infinite scroll pagination style, it makes wonders.

Dealing with update and removal

Unfortunately, adding new documents is not the only change that is going to occur. Updates and deletions are also bound to happen, albeit less frequently.

How to deal with that?

Well, if a different version of the content is associated to the same page identifier, just bump the version up and include this version very number in the URI.

http://example.com/v2/5/

If a request to http://example.com/v1/5/ ever comes in, a permanent redirect to http://example.com/v2/5/ can be issued. In any case, document 5 (+ older ones) will be present on the page, and browser cache will get invalidated.

While finer versioning (per-page group) would be achievable, a global version number is way easier to deal with. And good enough, considering the fact that updates and deletions are way less frequent than newly added content.

If having a version number in the URI of the first page, ie. the one with the most recent documents, is a concern, this page might be handled differently by falling back to Last Modified + 304 replies.

Thoughts on PHP sessions

2011-01-19T00:00:00+01:00

The sessions mechanism is one of the oldest component in PHP. It was born in 1997 and it remains pretty much the same in PHP 5.3, minus a severe security flaw that has been fixed.

Still, it remains widely used, possibly overused and misused as today's web sites aren't exactly facing the same challenges as sites made 15 years ago.

A PHP session is like a super-global array. The content of this array is automatically serialized and saved as a temporary file or in a database. A key, known as a "session identifier" is sent to the client. Further requests can include this key (usually as a cookie) in order to have PHP load the file or query the database, and fill back the super-global array.

This is a very bare bone mechanism.

Here's the typical flow of execution of a PHP script using sessions.

session_start();
# PHP reads a cookie (like PHPSESSID)
# PHP reads a file whose name is based on the cookie value
# PHP unserializes the content, and stores it into $_SESSION
# PHP adds a PHPSESSID cookie to the forthcoming reply

tons_of_things();

# PHP serializes $_SESSION and stores it in the session file
# The output buffer is flushed

The content of $_SESSION can be written before the end of the script if session_commit() is explicitly called.

More or less randomly, an additional function is called. That one is responsible for removing expired session files. Instead of hurting arbitrary requests, some Linux distributions delegate the cleanup to a cron-driven shell script.

To lock or not to lock?

When session_start() is called, the default session handler opens or create a session file and immediately locks it for exclusive use (see ext/session/mod_files.c).

This file gets unlocked when session_commit() is called, either explicitly, or implicitly at the end of the script.

This locking has a significant impact: parallel requests on PHP scripts using sessions will not be processed in parallel, but sequentially.

If a document is being served to a user, another session-enabled request from the same user will block. session_start() will block until the previous document has been fully served.

Why locking is good

Handling things sequentially and not in parallel is an obvious way to avoid race conditions.

A lot of PHP applications don't pay attention to atomicity and integrity. Databases operations are made without any transactions. Referencial integrity is not enforced. Denormalization makes it worse.

Consider the following pseudo-code, inspired by a popular discussion board engine:

function grant_user_access_to_section() {
  add_user_to_section();
  update_user_status();  
  update_section_acls();
}

Without any transactions, there's an obvious and possibly dangerous race condition. A user could load another document in another window while the database is still in a half-assed state.

Session locking indirectly protects against this.

Another scenario to consider is this one:

The main page for an ecommerce web site starts to load and gets partially rendered by the browser. The script reads $_SESSION in order to display the current number of items in the basket,
Before the request is complete, the user clicks "add to shopping cart",
This triggers an AJAX request that adds the new item to the $_SESSION-powered basket,
The script for the main page completes.

Locking ensures that the script called through AJAX will block until the script for the main page completes.

Sure, this prevents race conditions. But from a user point of view, it sucks. It means that nothing will immediatly happen after the user clicks "add to shopping cart". The request will stall until session_start() unblocks. From a sysadmin point of view, it also sucks. Having full processes doing nothing but wait for a lock sucks. From a developer point of view, it also sucks. Serialization was acceptable when applications were designed for a single computer running a single core. Now, we're living in a massively parallel world and every time you use a global lock, God asks Cuteoverload to remove pictures of painfully cute kittens.

What would happen without locking?

The script for the main page loads the session file, stores the content into $_SESSION,
The script called through AJAX loads the session file, stores the content into $_SESSION,
The script called through AJAX adds an item to $_SESSION
The script called through AJAX saves $_SESSION as the session file,
The script for the main page saves $_SESSION as the session file.

A script is unaware of what another script is doing at the same time. So the script from the main page will overwrite the session file with the content it previously read, no matter what the other script did inbetween. The item won't be added to the shopping card.

But... is the lack of locking the real issue? Or is locking just a kludgy band-aid against bad code, a la magic_quotes?

Let's recap what's happening in the script for the main page:

The session file is loaded
$_SESSION is build, by unserializing the content of the session file
A single value (the number of items in the cart) is read from $_SESSION
The content of the file is overwritten with the content of $_SESSION.

Can't you spot anything wrong here, way more shocking that anything that has to do with locking?

First, $_SESSION is a single big bucket. Its whole content is read and written from/to the session file everytime. This is the worst possible ORM. Totally unrelated content is going to get packed in the same basket. A script whose purpose is to add a product to the basket can overwrite a captcha key, just because if both use the PHP session, they share the same storage space, which is completely reread/rewritten by every script every time.

Another shocking aspect is the fact that our main script was supposed to only read the session. Yet, at the end of the script, PHP will overwrite the session file. And guess what with? The content it just read, unmodified. The file should be touched, not overwritten. Best of all, the main script, that doesn't change anything to the session, blocks every other script waiting for the session.

Is locking the solution? No.

Why no lock is better

The HTTP protocol has been designed to be stateless. And it works wonderfully this way.

Serving totally different content according to unknown data stored in server-side sessions easily breaks the REST principles. If you don't care about the REST principles, consider that: relying on sessions is likely to ruin the cacheability of what you are serving. Serving the same content when asking for the same thing is the key to cacheability. And possibly what the HTTP protocol has been designed for.

Ditching locks means that requests can be handled in parallel, not sequentially.

A web site that cares a little about performance uses progressive loading. Fragments are rendered independently (and possibly reassembled as a single page through Varnish or Nginx for search engines, but we're disgressing).

If every fragment has to wait for the previous one to complete, the benefits in progressive loading can vanish. Parallelization, hence being lockless, is important.

No lock means no idle process, doing nothing but waiting for a lock being released. PHP is not event-driven. Blocked scripts can't be a good thing.

Going lockless also means that, in order to provide sessions that can be shared by multiple PHP servers, a bunch of options are available. Virtually any key/value data store can fit the bill: Redis, Membase, Kyoto Tycoon, you name it.

How to safely move from lockful sessions to lockless sessions?

Here's a simple rule: use a session as a cache for immutable data.

If your server-side scripts constantly need to know what the user name is, it might be acceptable to keep this value in the session.

Regarding the above example: use dedicated storage space in order to store the shopping cart. Just create a shopping cart entry for the current user in Redis, MongoDb, Riak, PostgreSQL, whatever. And push items there. You don't need to store the shopping cart in the session. You just need a key to retrieve the shopping cart. And this key may be kept in the session.

Respecting the REST principles makes this obvious. The URI for a shopping cart should be something like /cart/shopping-cart-id, not /cart for everybody, leading to a document that solely depends on the session data.

One URI is one resource. Having a single global array, holding mutable data from independent resources is total nonsense. If data have to be shared by different resources, they should be immutable. This is what a session is for.

Ditching PHP sessions

Are PHP sessions actually required after all?

If you have to store secret data that have to be hidden to clients, maybe. Although you need to double check that a session is really where you want to store them, ie. that these data will be constantly needed by a lot of scripts.

In almost every case, sessions are not needed at all.

How does a PHP session work? The client sends a cookie, a PHP script reads this cookie and fetches the session data, using the cookie value as a key.

Why not just keep the session data in the cookie?

This is a very common practice in the Ruby world, introduced a long time ago with Rails and Rack::Session::Cookie.

And this is fairly secure as long as the cookie also contains a HMAC for the data.

What are the benefits?

Fast!
No need to store any sessions,
No need to wait for two queries in every script,
One less point of failure,
Share-nothing: any client can connect to any PHP server, at any time,
Having sessions lasting 5 minutes or 5 years don't make any differences, server-side.

Downsides:

If the secret key is leaked, you're screwed.
Larger requests.

Cachecookie

Cachecookie is a simple way to use cookie-based sessions.

If you want to give it a shot, just download Cachecookie.

(I'm not convinced that it is worth pushing to Github, but if you really want to fork it, ping me).

Cachecookie lets you store a multiples values in a single cookie, signed with a HMAC.

Need to store values? Say the user name and his cart id?

CacheCookie::set('user_name', 'John', 86400);
CacheCookie::set('cart_id', 'deadbeef', 31536000);

The third argument is the life time. Yes. Cachecookie lets you store multiple values, with individual expiration dates, in a single cookie. Ain't it great?

Let's retrieve the user name:

$user_name = CacheCookie::get('user_name');

If no entry is found, all you get is NULL.

Of course, the signature of the cookie is verified before any operation.

Want to delete a value?

CacheCookie::delete('user_name');

The method returns TRUE if the key existed, FALSE if you attempted to delete something nonexistent.

Want to delete everything (in fact, the single cookie)?

CacheCookie::delete_all();

Have fun!