Social Networking Data Discovery

Overview

The purpose of this document is to show how the search facility over the D4Science Infrastructure Social Networking data, primarily stored into a Cassandra Cluster, has been realized. Cassandra is an highly scalable and distributable database, used by a lot of companies around the world (eBay, Netflix, Instagram and many more). It offers highly availability by means of data sharding and replications.

The engine that enables the full-text search is ElasticSearch. ElasticSearch is an highly scalable, distributable, open source full text search and analytics engine based on the famous Apache-Lucene software library. It runs on one or more nodes and is reachable over http. It allows to organize documents in one or more indexes according to their schema. This schema can be defined in JSON format, even tought Elastic tries to automatically detect it.

The glue between Cassandra and ElasticSearch is a SmartExecutor plugin, namely the social-data-indexer plugin. In the following we are going to investigate which roles it have.

The main goal of the search facility is to let the users quickly search over this potentially huge amount of data, taking into account the data they are allowed to access. In fact, D4Science is a Research Infrastructure that offers many Virtual Research Environments (VREs). A user is allowed to see only the data of the VREs in which she is present.

The Social Networking Library

The gCube Social Networking Library is the bridge between gCube Applications and the social networking facilities. All information about the library can be retrieved here. As far as the search mechanism is concerned, the library is used to fetch data from the NoSql Cassandra cluster and to build up enhanced feeds. The concept of enhanced feed will be shown later. The library discovers the Cassandra Cluster in the infrastructure and offers a lot of methods, such as post creation/deletion, comment creation/deletion, notifications generation and so on.

Key features

TODO

Use cases

TODO

Design

TODO

Architecture

The engine underneath the full text search is an ElasticSearch Cluster. The ES software is an highly scalable, distributable, open source full text search and analytics engine based on the famous Apache-Lucene software library. ES runs on one or more nodes and is reachable over http. It allows to organize documents in one or more indexes according to their schema. This schema is defined in JSON format.

A SmartExecutor plugin, namely the SocialDataIndexer plugin has the role to fetch documents from the Cassandra nodes, organize them, and put them into the ElasticSearch cluster. Instead of pushing any new information sent to Cassandra when it is published, we decided to update the index by means of the plugin whose execution is scheduled pre

Usage/Examples

TODO

Deployment

TODO