CD-GAIN: Content Delivery Using Graph-based Analysis of Interest Networks

Recent years have seen a sea change in the Internet traffic mix, with the Web moving from primarily text-based content towards rich-media such as video and audio streaming. This has imposed significant additional costs both on content providers as well as on Internet Service Providers (ISPs).

This project proposes a novel approach that saves costs by identifying and actively enhancing ISP-local content availability. The core idea is that requests for content items can be served from ISP-local copies, where one is available. ISP-local access creates a synergy, with content providers saving on streaming costs, ISPs decreasing their cross-border transit traffic, and users obtaining local copies, isolating them from effects beyond the ISP such as packet losses, route failures or congestion in the network core.

Interest-based social networks, which are increasingly available on many content provider sites, provide an ideal framework to engineer the availability of ISP-local copies. Using the interest-based network, communities of users who are interested in similar items can be identified within each ISP. Such communities can serve as repositories of ISP-local copies of items they are interested in, and can support different access rates by pushing additional copies of items among themselves as required. Because members of the community have a high affinity for items they are responsible for, they might have a local copy available, having accessed the item in the past. Alternately, they might be likely to access the item in the future, so if a copy is pushed to them, the expected overhead of the push could be balanced against the future access. Methods will be developed to identify high quality communities useful for sharing content, by weighting or ranking links based on shared content consumption, predicting additional ISP-local links that have not yet been self-identified by users, and adapting community detection methods to create multi-resolution communities with "core" and "peripheral" members, to support fewer or additional ISP-local copies, as required, to sustain different rates of simultaneous content access from the ISP's userbase.

The project will take a data-driven approach, using extensive real-world traces from leading content providers both to derive new patterns in the interest network that can be exploited for content delivery, and also to evaluate the benefits of proposed content delivery architectures under realistic workloads. In analysing the traces, the project will develop new characterisations of interest-based social networks using content consumption histories and chart ISP-level content availability from a content provider's viewpoint, both of which would be of independent academic and commercial value.

The project will directly benefit two sectors of the UK economy: The content provider industry and Internet Service Providers. In addition, "Big Data" experience gained by the Research Associate working on large real-world data creates a skill set useful in many scenarios.

1. The content provider industry will be a primary beneficiary of this project. The benefits are two fold: improved analytics, and lower operational costs for content delivery.
1.a - Analytics: By expanding our understanding of content consumption patterns, the project will enable new analytics to be developed, which can be used to increase relevance of content to consumers. Further, insights from the social network could also lead to new ways for enhancing user engagement and retention.
1.b - Lower cost content delivery: The ISP-local content reuse architecture, a main output of this project, will improve scalability of operations and decrease streaming bandwidth costs (and/or the carbon footprint of content delivery) for content providers by converting direct access from the content provider servers to peer-to-peer traffic between users.

2. The ISP industry will benefit indirectly, but massively, if the ISP-local content reuse architecture proposed in this project gets deployed by major content providers. In addition, our studies of ISP-local content availability and connectivity patterns of users could lead to new economic models for ISPs: For instance, studying the cross-ISP roaming behaviour of users (e.g. users having a mobile phone with connectivity via both 3G and fixed line ISPs, or a laptop connected to one ISP at work and another at home) may lead to new pricing models for such users, as they now have a choice of more than one ISP to access content from. As another example, a large potential for directly sharing content among users may indicate the need for new peering arrangements between consumer facing ISPs.

3. Lastly, the Research Associate working on the large data sets of the project will gain experience in handling "Big Data", which is becoming an increasingly valuable skill, and can be applied in various employment sectors.