Frank DENIS random thoughts.

On the distribution of DNS TTLs

2012-05-10T00:00:00+02:00

I recently sampled 348,876,495 valid (actual records exist) A queries processed by OpenDNS servers. This represents 6,263,672 unique names.

And I then made a simple app that dumps the initial TTL (as reported by authoritative servers) for each of these unique names, in order to check what the TTL distribution looks like. This data was then processed with R.

Well, the TTL distribution looks very unbalanced, to say the least.

Do a lot of people publish records with a TTL that is longer than 30 days?

No, apparently, only a ridiculous amount of records happen to have a TTL larger than 30 days.

It's fun to see insane TTLs like 68 years (bbs.oouc.cn, beechglen.com, canadianangling.com, capitoltalk.com, durhambannerexchange.com, elimport.co.il, epictn.org, glutathioneforhealth.com, greeley.ca, gregsushinsky.com, ...), although these are probably configuration errors.

Let's zoom into more reasonnable TTLs, that are 1 week or less.

Wow. Apparently, even TTLs longer than 1 day are very rare. So let's shrink the window to TTLs that are no longer than 1 day.

A fair amount of records have been configured with a TTL that is exactly 1 day, but the vast majority seems to be below 4 hours.

Let's zoom in.

Ok, at this point, it's probably reasonnable to keep zooming in:

or with 10 segments:

This is still a very unbalanced distribution. 4 hours TTLs are common, 1 hour TTLs are more common, but the vast majority seems to be below 15 minutes.

TTLs below 1 hour represent the hot spot, so let's zoom in:

There's a fair amount of records with a 1 hour TTL, a high amount of records with a TTL below or equal to 10 minutes, and pretty much nothing in-between.

Let's see what TTLs below 10 minutes look like:

So, records within this interval are either below 5 minutes (with peaks at 1 minute and 2 minutes), or 10 minutes.

In summary:

TTL = 0	0.16 %
TTL <= 1 minute	9.84 %
TTL <= 2 minutes	16.32 %
TTL <= 5 minutes	39.88 %
TTL <= 1 hour	70.07 %
TTL <= 1 day	98.89 %

Looks like there are still ways to make the internet faster.

Akamai vs public DNS servers

2012-02-22T00:00:00+01:00

How a CDN works

"Downloads from Apple are excruciatingly slow". "Don't use Google DNS or OpenDNS, or else iTunes will be slow" "LOL, OpenDNS claims it makes the internet faster. Yeah, by telling Akamai you are in the US and getting 200K/sec downloads in Oz…"

Fact: these claims are true.

Apple uses Akamai, a popular CDN. The whole point of a CDN is to make your downloads faster: CDNs have servers at multiple locations, and when you request some content, CDNs are supposed to serve it from the closest (and hopefully fastest) location.

There are different ways to achieve this, but a very common one, that Akamai also relies on, is to leverage the DNS system.

When your client wants to resolve www.apple.com, Akamai DNS servers are going to give different replies according to the source IP address.

But what source IP address? Yours? Not always. What Akamai consider is actually the IP address of the DNS resolver hitting their servers. If you're using a local resolver, that's cool. Akamai will know your exact IP address, and hopefully redirect you to the closest server.

If you're using your ISP's resolver, this is the IP Akamai will consider in order to pick the server that will process your query. The result may be the same as if you had used a local resolver. Or not. If the DNS resolver is on a different subnet, Akamai can get confused and you can be redirected to a server that actually is way off base.

If you're using OpenDNS, Google DNS, Norton, Level 3, any other public DNS service, or any other remote DNS resolver (for example, through a VPN connection), Akamai will see the source IP address of your remote resolver, too. Nor yours.

And the server they will redirect you to would probably be an excellent choice if you were Google or OpenDNS. But the very same server can be a very poor choice for you.

From the same origin, packets can take a totally different route in order to reach Google, OpenDNS and Akamai. DNS queries sent to Google or OpenDNS from Paris to Amsterdam can be super fast. But then, downloads from Paris to an Akamai server in Amsterdam can be super slow. Just because the routes can be totally different.

What about other CDNs?

How come this problem occurs with Akamai but not with other CDNs?

Some other major CDNs don't have this problem because instead of considering the resolver IP address, they actually pay attention to the actual client network, in order to pick the closest/fastest server.

OpenDNS and Google DNS have been supporting the edns-client-subnet extension for a long time. This mechanism was designed by Google specifically to address this problem. And it works beautifully. CDNs can send a redirection to the best server no matter what resolver you use.

Unfortunately, Akamai still don't support this mechanism.

Make an exception

It's totally possible to use Google, OpenDNS or any other public DNS service and still avoid being redirected to a slow server by Akamai.

Just make an exception.

If you happen to run OSX, your operating system already provides a way to use specific DNS resolvers in order to resolve specific domains.

If you want queries to *.apple.com to go to your ISP's resolvers instead of Google, OpenDNS, or any other default resolver, just create a file named /etc/resolver/apple.com with the following content:

nameserver a.b.c.d
nameserver e.f.g.h

Replace a.b.c.d and e.f.g.h with your ISP's resolvers. You can have as many resolvers as you want, and as many exceptions as you want. Just create files as needed. One per domain name.

If you're running another operating system, you should be running a local DNS cache. The Windows DNS cache is a total joke and most Unix-like systems don't provide any cache at all.

Running a cache like Unbound is the best way to reduce latency due to DNS queries, even if queries are forwarded to Google or OpenDNS. In addition to caching queries, modern resolvers have nice features like the ability to automatically prefetch records before they expire. For CDNs and social networks using stupidily low TTLs, it can really make a difference.

Once you are running a cache, and it has been configured to forward queries to OpenDNS, Google, or anything else, you can configure exceptions.

With Unbound, it's as simple as adding two lines to the configuration file:

forward-zone:
    name: "apple.com"
    forward-addr: a.b.c.d
    forward-addr: e.f.g.h

The edns-client-subnet mechanism has been around for quite some time now, so all modern CDNs are eventually going to support it, and manually adding exceptions won't be required any more.

But as we speak, this is still required, at least for Akamai.

Fortunately, adding exceptions is easy. So you can totally work around CDNs that don't support a DNS mechanism that was designed for them, and still enjoy the benefits of fast and reliable DNS services in order to access content served by other services.

DNS records and TTL - how long does a second actually last?

2011-11-17T00:00:00+01:00

In a DNS zone, every record carries its own time-to-live, so that it can be cached, yet still changed if necessary.

This information is originally served by authoritative servers for the related zone. The TTL is represented as an integer number of seconds.

At first sight, the mechanism looks straightforward: if the www.example.com record has a TTL of 30, it's only valid up to 30 second, and caches must fetch it again if requested after this delay.

Chained caches

DNS caches can be chained: instead of directly querying authoritative servers, a cache can forward queries to a another server, and cache the result.

Having 3 or 4 chained caches is actually very common. Web browsers, operating systems and routers can cache and forward DNS a query. Eventually, this query will be sent to an upstream cache, like the ISP cache or a third-party service like OpenDNS. And these caches can also actually hide multiple chained caches.

In order to respect the original TTL, caches are modifying records as they forward them to clients. The response to a query that has been sitting in a cache for 10 second will be served with a TTL reduced by 10 second. That way, if the original TTL was 30 second, the whole chain is guaranteed to consider this record as expired as the same time: the original meaning of the TTL is retained no matter how many resolvers there are in the way.

Well, not exactly. A TTL is just a time interval, not an absolute date. Unlike a HTTP response, a DNS response doesn't contain any timestamp. Thus, requests processing and network latency are causing caches to keep a record longer than they actually should in order to respect the initial TTL.

A TTL being an integer value makes things even worse: a chain of N caches can introduce a N second bias.

In practice, this is rarely an issue: TTLs as served by authoritative servers are considered indicative, and not as something to depend on when accurate timing is required.

The decay of a TTL, as seen by different pieces of software

How does the TTL of a record served by a DNS cache decays over time?

Surprisingly, different implementations exhibit different behaviors.

For a record initially served with a TTL equal to N by authoritative servers:

Google DNS serves it with a TTL in the interval [0, N-1]
dnscache is serving it with a TTL in the interval [0, N]
Unbound serves it with a TTL in the interval [0, N]
Bind serves it with a TTL in the interval [1, N]
PowerDNS Recursor always serves it with a TTL of N

Hold on... Does it mean that how frequently an entry will actually be refreshed depends on what software resolvers are running?

Sadly, yes. Given a record with a TTL equal to N:

Bind and PowerDNS Recursor refresh it every N second if necessary
Unbound and dnscache are only refreshing it every N + 1 seconds at best.

TTL Zero

Although a TTL of zero can cause interoperability issues, most DNS caches are considering records with a TTL of zero as records that should not be cached.

This perfectly makes sense when the TTL of zero is the original TTL, as served by authoritative servers.

However, when a cache artificially changes the TTL to zero, it changes a record that had been designed for being cached to an uncachable record that contaminates the rest of the chain.

dnscache and a 1 second record

To illustrate this, a Linux box has been setup with a local DNS cache. Unbound has been chosen, but the last component of the chain actually makes little difference. Even web browsers caches have a very similar behavior.

Queries are forwarded to an upstream cache on the same LAN, running dnscache, and outgoing queries are recorded with ngrep. The same query, whose response has a TTL of 1, is made at a 10 queries per second rate.

The X axis represents outgoing queries, whereas the Y axis is the time elapsed since the previous query.

Even though a TTL of 1 is served by authoritative servers for this record, a large amount of responses are cached and served for 2 second. This is due to the local Unbound resolver.

However, there is also quite a lot of responses that haven't been cached at all. It happens when dnscache serves a response with a TTL of 0. Since it happens one third of the time, this is suboptimal and not on par with the intent of the authoritate record, which is served with non-zero TTL.

And although dnscache and Unbound are handling TTLs the same way, we can't expect their caches to be perfectly synchronized. When the local cache considers a record as expired and issues an outgoing query, the upstream server can consider it as not expired yet, just in the middle of the last second. What we get is a constant race between caches, causing jitter and outgoing queries sent after 1 second.

How different implementations can affect the number of outgoing queries

The following experiment has been made by observing outgoing queries when sending 10,000 queries for a record with a TTL of 2, at an average 10 qps rate, to a local cache, using Google, OpenDNS and Level 3 as upstream resolvers.

Here is the number of outgoing queries that were required to complete the 10,000 local queries using different services:

Service	Outgoing queries
Google DNS	1383
OpenDNS	632
Level 3	388

When using Google DNS (blue dots), queries are effectively never cached more than 2 second, even locally. This is due to the max TTL returned by Google DNS being the initial TTL minus one.

When Google DNS returns a TTL of zero, we observe the same jitter and the same slew of queries that couldn't be served from the local cache.

OpenDNS (orange dots) has the same behavior as dnscache and Unbound., with TTLs in the [0, N] interval. Our initial TTL with a value of 2 actually causes 3, 2, and 1 second delays between request, plus a sensitive amount of consecutive outgoing queries due a null TTL.

Level 3 (green dots) is running Bind, which returns a TTL in the [1, N] interval. Bind caches records one second less than dnscache and Unbound. Because our local resolver is Unbound, queries received with a TTL of N are actually cached N + 1 second. But as expected, the frequency of required outgoing queries very rarely exceeds 3 second.

What the correct behavior is, is out of scope of this article. All major implementations are probably correct.

But from a user perspective, with only 2 caches in the chain, and for a given record, the same set of queries can require up to 3.5 more outgoing queries to get resolved, depending on what software the remote cache is running.

With CDNs and popular web sites having records with a very low TTL (Facebook has a 30 second TTL, Skyrock has a 10 second TTL), the way a cache handles TTLs can have a sensible impact on performance. That said, some resolvers can be configured to pre-fetch records before they expire, effectively mitigating this problem.

Update: OpenDNS resolvers now behave like Bind: TTLs are now in the [1, N] interval.

The OSX cache

MacOS X provides a system-wide name cache, which is enabled by default.

Its behavior is quite surprising, though. It seems to enforce a minimum TTL of 12.5 seconds, while still requiring some outgoing queries delayed by the initial TTL value, and some consecutive ones. Resolving the 10,000 queries from the previous test took only an average of 232 outgoing queries.

Using Google DNS, OpenDNS and Level 3 as a remote resolver produce the same result, with the exception on Level 3 (Bind) avoiding frequent (less than 1 sec) consecutive queries during the time other return a TTL of zero.

Programmatically changing network configuration on OSX

2011-08-14T00:00:00+02:00

Programmatically changing DNS resolvers, IP addresses and proxies on OSX isn't rocket science but finding documentation on that is like looking for a needle in a haystack.

Changing the DNS configuration could have been as simple as updating /etc/resolv.conf, as OSX does provide an /etc/resolv.conf file. But unlike every other Unix variant out there, OSX doesn't actually read this file, as it is only there for compatibility with some legacy tools.

Meet configd

"The configd daemon is responsible for many configuration aspects of the local system. configd maintains data reflecting the desired and current state of the system, provides notifications to applications when this data changes, and hosts a number of configuration agents in the form of loadable bundles."

Think about it as a centralized key/value registry. Applications can watch for changes, add new key/value pairs and modify existing entries.

The scutil(8) command provides a command-line interface to this registry:

$ scutil
> list
  subKey [0] = Plugin:IPConfiguration
  subKey [1] = Plugin:InterfaceNamer
  subKey [2] = Setup:
  subKey [3] = Setup:/
  subKey [4] = Setup:/Network/BackToMyMac
  subKey [5] = Setup:/Network/Global/IPv4
  subKey [6] = Setup:/Network/HostNames
  subKey [7] = Setup:/Network/Interface/en0/AirPort
  subKey [8] = Setup:/Network/Service/0CE7A64C-9174-41AC-B634-41F4B6B5FE02
  subKey [9] = Setup:/Network/Service/0CE7A64C-9174-41AC-B634-41F4B6B5FE02/IPv4
  subKey [10] = Setup:/Network/Service/0CE7A64C-9174-41AC-B634-41F4B6B5FE02/IPv6
  subKey [11] = Setup:/Network/Service/0CE7A64C-9174-41AC-B634-41F4B6B5FE02/Interface
  subKey [12] = Setup:/Network/Service/0CE7A64C-9174-41AC-B634-41F4B6B5FE02/Proxies
...  
  subKey [32] = Setup:/Network/Service/527FC752-8B1D-4111-A205-4E81F757704E
  subKey [33] = Setup:/Network/Service/527FC752-8B1D-4111-A205-4E81F757704E/IPv4
  subKey [34] = Setup:/Network/Service/527FC752-8B1D-4111-A205-4E81F757704E/IPv6
  subKey [35] = Setup:/Network/Service/527FC752-8B1D-4111-A205-4E81F757704E/Interface
  subKey [36] = Setup:/Network/Service/527FC752-8B1D-4111-A205-4E81F757704E/Proxies
...
  subKey [58] = Setup:/System
  subKey [59] = State:/IOKit/LowBatteryWarning
  subKey [60] = State:/IOKit/Power/CPUPower
  subKey [61] = State:/IOKit/PowerAdapter
  subKey [62] = State:/IOKit/PowerManagement/Assertions
  subKey [63] = State:/IOKit/PowerManagement/CurrentSettings
  subKey [64] = State:/IOKit/PowerManagement/SystemLoad
  subKey [65] = State:/IOKit/PowerManagement/SystemLoad/Detailed
  subKey [66] = State:/IOKit/PowerSources/InternalBattery-0
  subKey [67] = State:/IOKit/SystemPowerCapabilities
  subKey [68] = State:/Network/BackToMyMac
  subKey [69] = State:/Network/Connectivity
  subKey [70] = State:/Network/Global/DNS
  subKey [71] = State:/Network/Global/IPv4
  subKey [72] = State:/Network/Global/Proxies
  subKey [73] = State:/Network/Interface
  subKey [74] = State:/Network/Interface/en0/AirPort
  subKey [75] = State:/Network/Interface/en0/CaptiveNetwork
  subKey [76] = State:/Network/Interface/en0/IPv4
  subKey [77] = State:/Network/Interface/en0/IPv6
  subKey [78] = State:/Network/Interface/en0/Link
  subKey [79] = State:/Network/Interface/lo0/IPv4
  subKey [80] = State:/Network/Interface/lo0/IPv6
  subKey [81] = State:/Network/Interface/p2p0/Link
  subKey [82] = State:/Network/Interface/utun0/IPv6
  subKey [83] = State:/Network/MulticastDNS
  subKey [84] = State:/Network/NetBIOS
  subKey [85] = State:/Network/PrivateDNS
  subKey [86] = State:/Network/Service/527FC752-8B1D-4111-A205-4E81F757704E/DHCP
  subKey [87] = State:/Network/Service/527FC752-8B1D-4111-A205-4E81F757704E/DNS
  subKey [88] = State:/Network/Service/527FC752-8B1D-4111-A205-4E81F757704E/IPv4
  subKey [89] = State:/Users/ConsoleUser
  subKey [90] = com.apple.DirectoryService.NotifyTypeStandard:DirectoryNodeAdded
  subKey [91] = com.apple.network.identification
  subKey [92] = com.apple.opendirectoryd.node:/Search
  
> get State:/Network/Global/DNS

> d.show
<dictionary> {
  ServerAddresses : <array> {
    0 : 208.67.220.220
    1 : 208.67.222.222
  }
}

As we can see, the registry holds most of the network settings, including DNS settings. The Setup:/* entries contain every available network configuration (as configured in the preferences pane), whereas State:/* entries include only what's currently active.

There are global DNS settings and configuration-specific DNS setings.

Programmatically accessing the registry

The overlooked DynamicStore API, which is part of the SystemConfiguration framework, let us access the configd registry.

The following code snippet opens the registry, looks for active DNS-related entries, and updates them with a new property list containing OpenDNS resolvers.

#include <SystemConfiguration/SystemConfiguration.h>

static bool setDNS(CFStringRef *resolvers, CFIndex resolvers_count)
{
    SCDynamicStoreRef ds = SCDynamicStoreCreate(NULL, CFSTR("setDNS"), NULL, NULL);
    
    CFArrayRef array = CFArrayCreate(NULL, (const void **) resolvers,
        resolvers_count, &kCFTypeArrayCallBacks);
    
    CFDictionaryRef dict = CFDictionaryCreate(NULL,
        (const void **) (CFStringRef []) { CFSTR("ServerAddresses") },
        (const void **) &array, 1, &kCFTypeDictionaryKeyCallBacks,
        &kCFTypeDictionaryValueCallBacks);    
    
    CFArrayRef list = SCDynamicStoreCopyKeyList(ds,
        CFSTR("State:/Network/(Service/.+|Global)/DNS"));
    
    CFIndex i = 0, j = CFArrayGetCount(list);
    if (j <= 0) {
        return FALSE;
    }
    bool ret = TRUE;
    while (i < j) {
        ret &= SCDynamicStoreSetValue(ds, CFArrayGetValueAtIndex(list, i), dict);
        i++;
    }
    return ret;
}

int main(int argc, const char * argv[])
{
    CFStringRef resolvers[] = {
        CFSTR("208.67.220.220"),
        CFSTR("208.67.222.222")
    };
    setDNS(resolvers, (CFIndex) (sizeof resolvers / sizeof resolvers[0]));
    
    return 0;
}

Of course, changing the registry requires root privileges, but that's all it takes.

As new entries can be seamlessly created, you can use the registry for centralizing configuration in your own apps, or just to save previous entries before altering them.

Changing network services

The technique describe above works fine. But the new settings aren't persistent. More often than once, this actually comes in handy, in order to apply temporary settings and to rest assured that once the user reboots his machine, network settings will always be back to a sane state.

Tweaking the system configuration database is not recomended by Apple because some daemons may not be aware of a configuration change. In addition, you may want to permanently apply the new settings.

Another way to tackle this problem is to actually alter the network services.

Under OSX, a network service is a configuration for a network interface. To apply a change globally, we need to step over all network services, check whether DNS (or whatever we want to change) is a support protocol, and make the related changes.

This achieves the same thing as what one can do from the Network preferences pane.

Almost.

After changing settings this way, you will notice that ths Network preferences pane properly reflects them. However, some apps won't have caught up with the changes. For example, the /etc/resolv.conf isn't automatically updated.

The trick here is to open the system configuration registry as described above, and then to close it without having done any actual change. Just "touching" the registry will trigger observers like configd so that they can do their own magic.

An empty DNS servers list or an empty IP/network address means that DHCP will be used in order to retrieve this information. But this doesn't happen automatically as the new settings are applied. Explicitly renewing the DHCP lease is required, by calling SCNetworkInterfaceForceConfigurationRefresh() on each interface.

@implementation DNSGlobalSettings

- (BOOL) DNSSupportForNetworkService: (SCNetworkServiceRef) networkService
{
    if (!SCNetworkServiceGetEnabled(networkService)) {
        return NO;
    }
    SCNetworkInterfaceRef interface = SCNetworkServiceGetInterface(networkService);
    if (!interface) {
        return NO;
    }
    NSArray *supportedProtocols = (__bridge_transfer NSArray *) SCNetworkInterfaceGetSupportedProtocolTypes(interface);
    if ([supportedProtocols indexOfObject: (__bridge NSString *) kSCNetworkProtocolTypeDNS] == NSNotFound) {
        return NO;
    }
    return YES;
}

- (BOOL) setResolversTo: (NSArray *) resolvers forNetworkService: (SCNetworkServiceRef) networkService
{
    if ([self DNSSupportForNetworkService: networkService] == NO) {
        return 0;
    }
    SCNetworkProtocolRef networkProtocol = SCNetworkServiceCopyProtocol(networkService, kSCNetworkProtocolTypeDNS);
    if (networkProtocol == nil || ! SCNetworkProtocolGetEnabled(networkProtocol)) {
        NSLog(@"disabled");
        return FALSE;
    }
    if (![resolvers isKindOfClass: [NSArray class]] || resolvers.count == 0) {
        resolvers = nil;
    }
    if (resolvers == nil) {
        NSLog(@"Setting this service's resolvers to DHCP");
    } else {
        NSLog(@"Setting this service's resolvers to [%@]", resolvers);
    }
    NSDictionary *DNSDict = (__bridge NSDictionary *) SCNetworkProtocolGetConfiguration(networkProtocol);
    NSMutableDictionary *newDNSDict;    
    if (DNSDict) {
        NSLog(@"This interface had an existing configuration");
        if (resolvers == nil) {
            SCNetworkProtocolRef networkProtocol = SCNetworkServiceCopyProtocol(networkService, kSCNetworkProtocolTypeDNS);
            if (networkProtocol == nil || ! SCNetworkProtocolGetEnabled(networkProtocol)) {
                return TRUE;
            }
            NSLog(@"Removing protocol from configuration");
            SCNetworkProtocolSetConfiguration(networkProtocol, NULL);
            return TRUE;
        }
        newDNSDict = [NSMutableDictionary dictionaryWithDictionary: DNSDict];
        if (resolvers == nil) {
            [newDNSDict removeObjectForKey: (__bridge NSString *) kSCPropNetDNSServerAddresses];
        } else {
            [newDNSDict setValue: resolvers forKey: (__bridge NSString *) kSCPropNetDNSServerAddresses];
        }
    } else {
        NSLog(@"This interface had no existing configuration");
        if (resolvers == nil) {            
            return TRUE;
        }
        newDNSDict = [NSMutableDictionary dictionaryWithObject: resolvers forKey: (__bridge NSString *) kSCPropNetDNSServerAddresses];
    }
    
    return SCNetworkProtocolSetConfiguration(networkProtocol, (__bridge CFDictionaryRef) newDNSDict);
}

- (BOOL) touchDynamicStore
{
    SCDynamicStoreRef ds = SCDynamicStoreCreate(NULL, CFSTR("myapp"), NULL, NULL);
    CFRelease(ds);
    NSLog(@"DynamicStore updated");
    
    return TRUE;
}

- (BOOL) forceDHCPUpdate
{
    NSLog(@"Forcing DHCP update");
    NSArray *interfaces = (__bridge_transfer NSArray *) SCNetworkInterfaceCopyAll();
    for (id interface_ in interfaces) {
        SCNetworkInterfaceRef interface = (__bridge SCNetworkInterfaceRef) interface_;
        SCNetworkInterfaceForceConfigurationRefresh(interface);
    }
    return TRUE;
}

- (BOOL) applyToNetworkServices: (BOOL (^)(SCNetworkServiceRef networkService)) cb andCommit: (BOOL) commit
{
    SCPreferencesRef preferences = SCPreferencesCreate(NULL, CFSTR("myapp"), NULL);
    SCPreferencesLock(preferences, TRUE);
    SCNetworkSetRef networkSet = SCNetworkSetCopyCurrent(preferences); 
    NSArray *networkSetServices = (__bridge_transfer NSArray *) SCNetworkSetCopyServices(networkSet);
    BOOL ret = TRUE;
    for (id networkService_ in networkSetServices) {
        SCNetworkServiceRef networkService = (__bridge SCNetworkServiceRef) networkService_;
        ret &= cb(networkService);
    }
    SCPreferencesUnlock(preferences);
    if (commit) {
        SCPreferencesCommitChanges(preferences);
        SCPreferencesApplyChanges(preferences);
    }
    CFRelease(networkSet);    
    CFRelease(preferences);
    if (commit) {
        [self touchDynamicStore];
    }
    return ret;
}

- (BOOL) setResolvers: (NSArray *) resolvers
{
    [self applyToNetworkServices: ^BOOL(SCNetworkServiceRef networkService) {
        return [self setResolversTo: resolvers forNetworkService: networkService];
    } andCommit: YES];   
    [[NSNotificationCenter defaultCenter] postNotificationName: @"CONFIGURATION_CHANGED" object: self userInfo: nil];

    return TRUE;
}

@end

Redis as a recommendation engine

2011-03-10T00:00:00+01:00

"You might be interested in red toilet paper, because you bought blue and green toilet paper, and people who also bought blue and green toilet paper tend to also buy red toilet paper".

Recommendation engines are now everywhere, from e-commerce to social networks.

Graphs databases like Neo4j are probably the best way to tackle the problem. But as an alternative, let's try building a recommendation engine on Redis.

Who bought what or who is following who could be stored in Redis sorted sets.

Let's take an example: a user buys A, and also B. We increment the score of the B item in the sorted set associated to key A. And we possibly also do it the other way round.

This is a trivial way to keep a list of what items have been bought, what are the related items and how many of them there are.

On a social network, the score associated to a relationship may reflect the level of trust.

Let's suppose we have the following relationships:

B -> C
D -> C
E -> F
A -> B
A -> D
A -> E

C is referenced by B and D. B and D are referenced by A. But C isn't referenced by A yet. This is presumably something to suggest.

F might also be a good candidate as a suggestion for A. However, since there's only a single path from A to F (through E), it's probably less relevant than C.

In order to limit the number of non-relevant results, we may want to add a threshold: nodes that have less possible paths than a cutoff value shouldn't be suggested.

How to do that with Redis, and only Redis?

Here's one possible way:

Use ZUNIONSTORE in order to build a temporary aggregate of all related items (scores are summed up), and remove the reference item from the set (cylic links).
Remove everything from the aggregate that is already directly related to the reference item. One way to achieve this on a sorted set is to use ZUNIONSTORE AGGREGATE MIN with a negative or null weight for the set we want to delete, followed by ZREMRANGEBYSCORE in order to actually remove everything from the reference set and everything below the cutoff score.
Sort and optionally trim the new set with ZREVRANGE et voila!

Maybe code speaks louder than words, so here we go for some code:

require "redis"

class Recommendation
  attr_reader :db
  
  def initialize
    @db = Redis.new  
  end
  
  def bought(id)
    Item.new(self, id)
  end
  
  def suggestions_for(id, options = { })
    Item.new(self, id).suggestions(options)
  end
  
  class Item
    def initialize(recommendation, id)
      @db, @id = recommendation.db, id
    end
    
    def self.key(id) "{rec}:items:#{id}" end
    
    def key() Item.key(@id) end
    
    def tmp_key(n = 1) "{rec}:_tmp:#{@id}:#{n}" end
    
    def and_also_bought(linked_id, weight = 1)
      @db.zincrby(key, weight, linked_id)
    end
    
    def suggestions(options = { })
      cutoff = options[:cutoff] || 1
      limit = options[:limit] || 50
      tmp, tmp2 = tmp_key, tmp_key(2)
      related_keys = @db.sort(key).collect { |item| Item.key(item) }
      return [ ] unless related_keys

      res = @db.multi do |m|
        m.zunionstore(tmp, related_keys, :aggregate => :sum)
        m.zrem(tmp, @id)
        m.zunionstore(tmp2, [ tmp, key ], :weights => [ 1, -1 ],
                                          :aggregate => :min)
        m.del(tmp)
        m.zremrangebyscore(tmp2, '-inf', cutoff)
        m.zrevrange(tmp2, 0, limit - 1)
        m.del(tmp2)
      end
      res[5]
    end
  end    
end

# Example

r = Recommendation.new

r.bought('b').and_also_bought('c')
r.bought('d').and_also_bought('c')
r.bought('e').and_also_bought('f')
r.bought('a').and_also_bought('b')
r.bought('a').and_also_bought('d')
r.bought('a').and_also_bought('e')

puts r.suggestions_for('a')