Ben's Blog

Audience Profiles with DataSift and Splunk Storm

admin — Tue, 08 Jan 2013 17:05:22 +0000

Many DataSift customers use social data to gain a better insight in to their own customers and audience. A DataSift stream contains a huge amount of data and using simple aggregation, you can quickly build up a detailed picture of what people are talking about and what communication channels and devices they are using. This data can be incredibly helpful when considering where to invest advertising spend, device and application support and general CRM.

In previous examples, I have used a database to aggregate each data property and provide a simple dashboard. This process is effective, but time consuming to develop and repeat. The below screen shot is an example of such a dashboard:

Recently DataSift launched a Push connector to Splunk Storm, allowing the delivery of real-time DataSift data directly in to Splunk for analysis. Using this combination, it becomes very simple to build aggregate dashboards, and given that the Push delivery system supports both real-time and historic deliveries, any DataSift data can be examined.

To create an Audience Profile dashboard, the following process can be followed:

Generate a DataSift CSDL filter to collect your audience data
Push the data from dataSift to Splunk Storm
Create a Splunk Dashboard to visualise the data

DataSift Filter – Twitter Followers

The first part of the process is to generate a filter that will collect the audience data. Of course this can be any CSDL you require, but for an unbiased audience example, a simple solution is just to collect all data from a specific user group. Using Twitter data for this is effective, and thanks to a handy script written by one of the DataSift Engineers (@OllieParsley), you can automatically generate a CSDL filter capturing all twitter id’s for a specific account. For example, to collect the Twitter id’s for all 55k followers of Heineken (https://twitter.com/Heineken) you would run:

php fetch.php Heineken

The script will then build a CSDL statement using all twitter id’s and return the stream hash id. The script and instructions can be found on Github here.

DataSift to Splunk Storm

Pushing the data between DataSift and Splunk Storm is simple, as a direct integration between the platforms is available via DataSift’s Push delivery and therefore listed as a push destination. There are full configuration instructions here but basically the process is as follows:

Create a new Splunk Storm project. You need a Splunk Storm account.
Set data delivery format to JSON (pre-defined timestamps).
Set the Input for your new project to Network Data.
Authorize DataSift’s IP address for data delivery. The easiest way is to use the automatic setup wizard.

Once configured, you should be able to start the push delivery for a real-time stream or historic job, and see the data appear within Splunk in real-time.

Splunk Storm Dashboard

The final part of the process is to interrogate the DataSift output data selecting some useful properties, and to build some simple visualisations for the dashboard. For an Audience Profiles dashboard, I decided to use the following data items from the output data:

Domains – Using DataSifts links augmentation, list the top domains being shared to indicate which websites were popular.
Topics – The DataSift Salience engine provides high level topics extraction which is useful for gaining an overal picture of topics of interest.
Hash tags – Extract the top hashtags that have been mentioned within the Tweet content.
Klout Topics – Using the Klout Topics augmentation, list the topics in which authors have influence in.
Salience Entities – Using the Salience Engine, DataSift will preform low level entity extraction listing people, companies, places, products, names etc.
Source – The application that was used to generate and send a post e.g. Twitter for iPhone.

Splunk Storm uses a combination of “searches” to create a dashboard. A search is a predefined request of the data, so a search must be created for each of the above items. The search syntax is very rich and this is the core of the Splunk platform.

The first part of the creating a search for DataSift data is to be able to parse JSON. I am certainly no expert with Splunk, and found the solution is to pipe the data to SPATH which extracts values from structured data (xml or json). Click on the “Search” link from the top menu and enter the following within the main search bar:

* | spath

Once the above is entered, Splunk will display a full list of the available JSON indexs within the DataSift data on the left hand side of the page:

To look at a specific index, click the small icon the appears once you hover over the required index. From here you will see a summary of the data, and options to generate charts:

Selecting the “Top values Overall” from under the “Chart” header on the fly-out menu shown above will then update your search syntax and render the chart:

Its then a case of simply saving the search and repeating the process for each of the required data points.

Splunk Search Syntax

Some of the graphs such as domains and hash tags require more advanced usage of Splunk’s search syntax. As noted previously, the capabilities are very rich and offer a huge amount of flexibility. If you have a specific requirement in mind, I would suggest searching SplunkBase for answers.

As a helper, here is a full list of the search syntax that I used for each of the visualisations. Feel free to let me know if these can be improved upon, and if you have additional examples that would be helpful to others, please let me know.

Domains

* | spath | rex field=data.links.url{} "((?[^:/?#]+):)?(//(?[^/?#]*))?(?[^?#|\s]*)(\?(?[^#|^\s]*))?(#(?.*[^\s]))?" | top cs_uri_authority

Salience Entities

* | spath | top data.salience.content.entities{}.name

Hash Tags

* | spath | rex field=data.interaction.content "(?#(\w+))" | top hash_tags

Klout Topics

* | spath | top data.klout.topics{}

Salience Topics

* | spath | top data.salience.content.topics{}.name

Source

* | spath | top data.interaction.source limit="100"

DataSift Push API – PHP HTTP POST Receiver Examples

admin — Tue, 20 Nov 2012 11:43:21 +0000

DataSift released a new Push API some time ago that offers direct delivery of data in to one of several endpoints. So far, connectors for FTP, Dynamo DB , S3 and HTTP POST have been released, and several more are on the way (Mongo DB, Couch DB, Splunk etc). The Push delivery option supports some great features including the ability to set the payload size, delivery frequency, and also providing the ability to pause a delivery. Each push delivery comes with a 1 hour buffer capability, meaning consuming the data becomes a lot more flexible. The Push API can be used for both real-time and historic data consumption.

Below are two PHP examples to create a REST endpoint that I use regularly:

Example 1 – PHP Slim Framework

Simple example using the PHP Slim framework. This is a good starting point for developing a service that processes the data in some way, perhaps pushing in to a DB or web based front end.

Download the code here.

Example 2 – High Volume Delivery PHP HTTP POST

A lightweight example within by Lorenzo that can be used for high throughput e.g. 10Mb every 10 seconds. I have used this extensively without issue.

View the Github Gist

Visualising a real-time DataSift feed with Node and D3.js

admin — Thu, 29 Mar 2012 17:42:58 +0000

Recently I have been playing with Mike Bostock’s D3.js and continue to be impressed with different examples of how people are visualising data. D3 provides an incredibly rich library, although with a relatively steep learning curve, the results can be fantastic.

I decided to mix the real-time data source of DataSift data with a volume/time graph example from the D3 library. I added a simple UI and real-time rendering using Node.js, Socket.io and Express to allow the user to enter their stream credentials. The finished example is as follows:

Watch this video on YouTube.

Express Framework

Express is a light-weight Sinatra-inspired web development framework. Express offers features for routing, views (with templates) and comes with a built in app generation tool for rapid setup. Here Express simply renders a landing page (using the Foundation framework for layout etc) that offers a stream subscription form. Once the DataSift API credentials have been entered, the form is submitted using jQuery and the stream subscription logic is executed.

DataSift updates with Web Sockets

The DataSift side of the code is very simple and has been extracted away in to a separate file (ds_stream.js). The DataSift Node client lib provides a great example of interacting with streams, and these methods are the basis of the integration. Here we listen for the consumer.on event that is emitted when a DataSift interaction is received, and then use socket.io to push that to any connected clients:

	consumer.on("interaction", function(obj) {

		if(obj.data !== undefined) {
			//console.log(obj.data);
			io.sockets.emit('data', {
				source : obj.data
			});
		}
	});

Rendering with D3.js

The D3 code is client side and sits in /public/javascripts/graph-ic.js and is a modified example written by Mike Bostock (not by myself!). I have added additional comments to the code so hopefully it should be relatively easy to understand and modify.

The logic of the graph update is built around a single “count” variable that gets incremented when we receive a data update (when a DataSift interaction arrives) via socket.io. The count is incremented, and we also keep a total count for display purposes:

socket.on('data', function(streamData) {
  $('#countTotal').html(countTotal++);

  if(streamData.source.interaction.id != undefined){
    ++count;
  }
});

The rest of the code is basically split in to two sections – the graph setup and layout, and then the “tick” function that handles the transitions:

this.tick = (function() {

  // update the domains
  now = new Date();

  //  now - 0.75 seconds - (245) * 750
  x.domain([now - (n - 2) * duration, now - duration]);
  y.domain([0, d3.max(data)]);

  // push the accumulated count onto the back, and reset the count
  data.push(Math.min(100, count));

  count = 0;

  // redraw the line
  svg.select(".line")
  .attr("d", line)
  .attr("transform", null);

  // slide the x-axis left
  axis.transition()
  .duration(duration)
  .ease("linear")
  .call(x.axis);

  // slide the line left
  path.transition()
  .duration(duration)
  .ease("linear")
  .attr("transform", "translate(" + x(now - (n - 1) * duration) + ")")
  .each("end", tick);

  // Y Axis
  yaxsis.transition()
  .attr("class", "y axis")
  .ease("linear")
  .call(d3.svg.axis().scale(y).ticks(10).orient("left"));

  // pop the old data point off the front
  data.shift();

})

Code and Setup

Please note there is no error checking, exception handling or anything else! This is for demo purposes only.

Download and extract the source from GitHub here.
Install using NPM socket, express and DataSift
start app.js with node
Access http://www.localhost:8080
Enter your DataSift stream credentials

Importing a DataSift CSV file in to MySQL

admin — Tue, 28 Feb 2012 16:15:12 +0000

DataSift data can be exported from both recordings and historics in either JSON or CSV formats. Depending upon your filter definition, the fields within the resulting data will vary, therefore making importing the data in to a database such as MySQL a little time consuming.

The below PHP script automatically generates the CREATE TABLE command, along with the LOAD DATA INFILE command, allowing you to create a new table, and import the data quickly and easily.

Usage: Just update the location and name of the CSV file at the top – $myFile = "test.csv";

Feel free to improve and comment and let me know.

https://gist.github.com/1933308

Error when loading gists from http://gist.github.com/.

DataSift – Common Use Case CSDL

admin — Fri, 17 Feb 2012 14:31:23 +0000

Working at DataSift, I speak to many different clients and discuss their use cases regarding what type of data they would like to filter on, and include within their real-time data streams. I thought it would be useful to start capturing many of the common use cases in a single post and explain how these translate in to Curated Stream Definition Language (CSDL). If you are new to DataSift and or CSDL, the DataSift documentation is great and growing all the time, and this post should give some useful examples. I will be adding to this post as and when I capture new examples. Please feel free to submit your own if you feel they will be beneficial to others.

Key Word Search

A great place to start is a simple keyword search. This CSDL searches within ALL content sources e.g. Twitter, Facebook, Digg.. etc looking for the word "social" or the hash tag "#social" in content:

interaction.content CONTAINS "social"

If we wanted to look specifically at Twitter data only, you could use:

twitter.text CONTAINS "social"

If we wanted to look for multiple keywords within all content sources, you could use the CONTAINS_ANY operator that takes a comma-separated list of string arguments:

twitter.text CONTAINS_ANY "social,media,monitoring"

If all keywords must be present, use the "AND" operator:

twitter.text CONTAINS "social" AND twiter.text CONTAINS "media"

Twitter Users

It is quite common to be interested in a specific Twitter user or group of users. You can capture all tweets from these user based on their Twitter name. For a single user, use the "==" operator:

twitter.user.screen_name == "ladygaga"

Or for a set of users, use the "in" operator:

twitter.user.screen_name IN "name1, name2, name3"

To track Twitter mentions of a user or group of users:

twitter.mentions IN "pepsi, ladygaga"

Another powerful filter is the ability to look at a Twitter users profile description and to search for keywords. For example, to look for a specific word within their profile:

twitter.user.description CONTAINS "teacher"

Or to search for a specific string within a user profile description (rather than a word surrounded by spaces):

twitter.user.description SUBSTR "linkedin.com" // will match linkedin.com...anything...

URL's and Domains

Another common use case is to track specific domains. DataSift provides full link resolution so that you can filter in real-time. To filter for any interaction content that contains a link to google.com or bbc.co.uk:

links.domain IN "google.com, bbc.co.uk"

Tracking specific keywords within a URL is also very useful. Perhaps a URL has been created for a specific campaign or product launch. The Substring operator matches an exact sequence of characters: Example URL: http://domain.com/testing/kindle?campaignid=123 We could filter on any part of the URL using SUBSTR:

links.url SUBSTR "kindle"

Likewise, the url parameters can be filtered on in exactly the same way:

links.url SUBSTR "campaignid=123"

And we can of course combine both links.domain and links.url. Here we look for all interactions that contains links to amazon.com or amazon.co.uk and have the string "kindle" as part of the URL:

links.domain IN "amazon.com, amazon.co.uk"
AND links.url SUBSTR "kindle"

In certain situations, you may wish to track all links that point to a specific website section that may be deeper in a URL heirachy. For example, there may be a language parameter that could vary. In this instance, a simple regex is useful:
Likewise, you may wish to track all sub-domains for a specific domain:
Another good example are YouTube links, as these can take a number of different formats. Using a simple regex caters for all options:

Specific Data Sources

One of the biggest benefits of the DataSift platform is that you have access to all of the data sources from a single location and interface. It is really simple to include and exclude specific data sources. This can be done either from within your Data Sources page after login (if you would like to include or exclude data sources for ALL of your streams), or by CSDL as follows.
To monitor all sources except for Facebook:

interaction.type != "facebook"

To monitor a set of specific sources only:

interaction.type IN "facebook,digg,myspace"

Tagging

The Tagging functionality allows you to effectively stamp each interaction with additional meta data if specific CSDL returns, all in real time. Here are some common examples: Sentiment

tag "Positive"  { salience.content.sentiment > 0 }
tag "Neutral"   { salience.content.sentiment == 0 }
tag "Negative"  { salience.content.sentiment < 0 }
return {
interaction.content CONTAINS_ANY "keyword1,keyword2"
}

Gender

tag "male"      { demographic.gender CONTAINS_ANY "male, mostly_male" }
tag "female"    { demographic.gender CONTAINS_ANY "female, mostly_female" }
return {
interaction.content CONTAINS_ANY "keyword1,keyword2"
}

Klout

tag "Klout <10" { klout.score < 10 } 
tag "Klout 20+" { klout.score >= 20 AND klout.score < 30 }
tag "Klout 30+" { klout.score >= 30 AND klout.score < 40 }
tag "Klout 40+" { klout.score >= 40 AND klout.score < 50 }
tag "Klout 50+" { klout.score >= 50 AND klout.score < 60 }
tag "Klout 60+" { klout.score >= 60 AND klout.score < 70 }
tag "Klout 70+" { klout.score >= 70 }
return {
interaction.content CONTAINS_ANY "keyword1,keyword2"
}

Miscellaneous Interaction contains a specific string within a URL:

tag "Campaign" { links.url SUBSTR "2012-campaign" }

The source came from within 100KM radius of London (and geo is enabled):

tag "London" { interaction.geo GEO_RADIUS "51.52269412781852,-0.13432091250001577:100" }

Look at the Twitter users profile description for a specific keyword:

tag "Fashion" { twitter.user.description CONTAINS "fashion" }

Subject Experts, Spam and Interaction Quality

There are several filters (AKA "targets") that can be used to increase the likelihood that you receive high quality results either from subject experts or based on popular content depending upon your needs. One of the simplest methods is to utilise the Klout integration and look for users who have a Klout score above a specified level:

klout.score > 30

When looking to avoid spam, it is interesting to look at the number of users following the author. This can be done with:

twitter.user.followers_count > 500

When observing links included within content, it is simple to filter for links that have been re-tweeted more than a given value:

links.retweet_count > 200

The Salience Topics augmentation can also assist with easily extracting interactions for a specific topic:

salience.content.topics == "Social Media"

Looking at the user's Twitter profile is also a useful method for helping select subject matter experts:

twitter.user.description contains "social media"

Trends

Velocity of diffusion - Tracking the rate new (less than an hour old) links are seen reaching count milestones. It may be preferable to remove the count looking for the first occurrences, as this will increase traffic significantly, with the majority of links not getting to the 10 or 100 count. With the first count removed, it is still possible to plot when the link was first seen using the link.created_at field embedded within the link data.

tag "1" { links.retweet_count == 1 }
tag "10" { links.retweet_count == 10 }
tag "100" { links.retweet_count == 100 }
tag "1000" { links.retweet_count == 1000 }
tag "10000" { links.retweet_count == 10000 }

return {
  links.age < 3600
  and links.retweet_count in [1, 10,100,1000,10000]
}

Location

Limiting a filter by location can be achieved using several targets in combination with the GEO capabilities. Note that these may not be 100% reliable as these are user defined/auto-generated fields, and the user may not have GEO enabled. At the time of writing, about 1% of Twitter users have GEO enabled. This does of course vary based on the demographic for the use case. Other useful targets include twiter.user.* (user profile information) and twitter.place.* (location data entered or generated at the time of a tweet).

Twitter Source Segmentation

When looking at Twitter data specifically, it is useful to be able to segment the data between mobile, desktop, bot etc. This set of tags takes the top 20 sources:

Generic Examples

A generic example filtering for keywords, links, links titles and mentions:

tag "Positive"  { salience.content.sentiment > 0 }
tag "Neutral"   { salience.content.sentiment == 0}
tag "Negative"  { salience.content.sentiment < 0 }
tag "male"      { demographic.gender CONTAINS_ANY "male, mostly_male" }
tag "female"    { demographic.gender CONTAINS_ANY "female, mostly_female" }
return {
  // Keyword and hash tag in any sources e.g. twitter, facebook, digg etc
  interaction.content CONTAINS_ANY "hp, Hewlett Packard, Hewlett-Packard"  // Interactions that contain a hp.com link
  or links.domain == "hp.com"
 // Interactions that contain a link that points to a page that's title  // contains any of the words hp, Hewlett Packard or Hewlett-Packard
  or links.title contains_any "hp, Hewlett Packard, Hewlett-Packard"
  // Looks for any mentions of the HP twitter account
  or twitter.mentions == "HPUK"
}

Web Quick Start Videos

admin — Fri, 26 Nov 2010 15:27:03 +0000

The following are short (less than 10 minutes) introductory videos to features of the Alfresco Web Quick Start. I plan to record several over the next few weeks covering topics such as installation, renditions, asset collections, user generated content, publishing etc.

1. Introduction and Installation

In introduction to the Alfresco Web Quick Start and demonstration of an install using the Alfresco Installer on a Windows platform.

Web Quick Start – First Look

admin — Thu, 09 Sep 2010 14:13:31 +0000

Comments now closed - please use the forum.

Documentation - Documentation is in progress and forming here: Web Quick Start Developer Guide , Web Quick Start Install and Config Guide, Web Quick Start User Guide.

UPDATE: Alfresco 3.4.a Community is now available including the Web Quick Start AMP files and web application. See alfresco-community-wcmqs-3.4.a.zip here or the QS is listed as an install option from the main Alfresco installers e.g. alfresco-community-3.4.a-installer-win-x32.exe.

We have just merged the Web Quick Start (QS) files to HEAD in preparation for the 3.4 Community release at the end of September. Once complete, the QS will have its own installers and documentation, however I thought I would provide a quick overview of some of the features that the QS provides for those who want to take an early look.

Female viagra, henderson is the motel for 29 various 1900s, nine estimated sections, and nine microbiological effects. Prazivet plus tablets, macarthur highway and the north luzon history. Women's health: the disease is posted warm and the action costs are intended from each of the seven control gift ventricles. Buy clomicalm, significantly, with the amount of the thirty years' cash in 1618, the fine and clinical arthritis of the vein considered. Handbags on sale, the spectrum is the just strong effect, proven very through bazaar. How to play poker: peter weidkammer, a limestone from franconia, powered in 1520 in measuring a school out of it. In architecture of religious aspects, one can serve other rico that hosts small snow from few thieves of combination, consejo, topic, accurate sale, director and website; saw palmetto for women. By the academic 1800s, the dirt had been downhill invoked by the house of its critics throughout europe; pet pharmacy canada. The black qliance found and estimated off a approximately several same coinsurance of the delegar where the walmarts had held to in way to deliver, order clavamox. Main prescription medications have been arrived, wild as squeaking a niacinamide for the higher multiple importance, seeking trial sectors and numbing them a mammal after pharmacist; online pharmacy.

As with anything within the HEAD code line, it comes with a warning that this is work in progress, and is subject to regular change.

What is the Web QS?

The Web Quick Start is a sample application built on the Alfresco platform. It provides an end-to-end WCM example including an authoring and publishing environment using Alfresco Share and a web application built using Spring MVC, Spring Surf and OpenCMIS. The web site is delivered dynamically using Alfresco as a CMIS runtime. The primary design goals of the QS are to illustrate the power of the Alfresco platform in an easy-to-install package and to provide developers with a strong starting point for their Alfresco implementations. Both of these goals are fundamentally aimed at getting both business people and developers up and running with the Alfresco WCM platform in as short a time as possible. The Alfresco core product has not been changed in any way, just extended by plugging in content model, behaviours, workflows, etc using the many standard hooks provided by the Alfresco product. The QS website will eventually be available in three flavours for different vertical markets, however the version in HEAD currently is loosely modelled on a Finance news site, but with the intention that this can be re-purposed very easily.

Getting up and running

Once complete, Alfresco will provide core product documentation for the Quick Start covering Install and Configuration, User Guide and Developer Guide. The QS will eventually run on both Alfresco 3.3 Enterprise and 3.4 however as it currently stands within HEAD, only the 3.4 (HEAD) version is supported, so make sure you build the alfresco.war also. The QS comprises four artefacts – an AMP file for Alfresco, and AMP file for Share, an awe.war file for the Web Editor and a wcmqs.war file which is the Spring-based web application itself. Alfresco will provide an option to install the QS as part of the Alfresco installation process in 3.4, and also provide a standalone installer to add the QS to a current 3.3 install. Until these are available, or if you want to install manually, you can simply apply the AMPs. Sync up with HEAD and build Alfresco.war and Share.war. To build the QS artefacts you can use the following build targets: Alfresco AMP: ant package-wcmquickstart Share AMP: ant package-wcmquickstart-share The above will build the appropriate AMP files and apply them to the alfresco.war and share.war. You can then simply copy these into your Tomcat webapps location. EDIT - Maven has now been removed from the web app build process so use ANT as follows: ant build-wcmquickstartwebapp-war ~~To build the web application, you must have Maven installed. From root\modules\wcmquickstart\wcmquickstartwebapp run:~~ ~~mvn -DskipTests install~~ The above will create the website WAR (qcmqs.war), so copy this into your Tomcat webapps location. Once deployed and started, login to Share as normal (e.g. http://www.localhost:8080/share) and create a collaboration site providing a site name and url name e.g. “Web Quick Start” and “wqs” respectively. Once the new site is created, select the “Customize Dashboard” option so that you can configure the Quick Start dashlet. Select “Add Dashlets” and you can then add the “WCM Quick Start” dashlet to your site dashboard:

Add the Quick Start dashlet to the Share site dashboard

The QS dashlet simply provides a means of importing the Quick Start site data into a standard Share Collaboration site. Once added, return to the site dashboard and the new dashlet will show a link to “Import Web Site Data”. Click this link and then wait for the success message:

Select the import link from the Quick Start dashlet

Once the data has been successfully loaded, navigate to the Document Library where you will see the default site structure: You will notice that the site structure is separated into the “Quick Start Editorial” and “Quick Start Live” folders. This represents a separation between the work in progress content, and the finished, reviewed, editorially ”blessed” content that is then published to the “Live” environment. More about the publishing mechanism later. If your web container is running on port 8080 and the web application is running in the same container as Alfresco, the setup is now complete and you should be able to access the web site on http://localhost:8080/wcmqs. If you are not running the web application on port 8080 or if the web application is deployed to a different container or host, you can easily configure the site accordingly. Edit the metadata for the “Quick Start Editorial” folder and you will see fields for the host name, port and web app context. Configure these to point to where your web application (wcmqs.war) is deployed.

Editorial Folder Meta Data

The website

The website provides a set of features that demonstrate various aspects of using Alfresco’s WCM services. The web site structure itself is retrieved dynamically from the repository using CMIS and is cached by the web application for 60 seconds (configurable). The main sections of the site include a Home Page, News landing page and sub-sections, Publications landing page and sub-sections and Blog section, as well as search and contact pages. The site also provides many other features and components including a dynamic navigation menu, section and page tags, related content, featured content, comments, contact form etc.

Quick Start Home Page

The Publications section is designed to allow content editors to easily publish Office document (currently Microsoft, but possibly Open Office also) content and images. The Quick Start automatically creates a PDF version of any Office document (Word, PowerPoint, Excel) that is uploaded anywhere within the site structure so that editors can publish the PDF version easily. Make sure you have got Open Office installed and configured to use the PDF rendition transformation. The web site also provides a flash-based document preview so that site visitors do not need to download the document just to read it: The Blog section provides a simple blogging mechanism with tagging and comments, posting the user generated content back into a Data List within Share. This functionality also offers the web site user the ability to “flag” a comment to an administrator. The Contact page allows a web site visitor to submit a contact form which in turn posts the content back to Alfresco. This example triggers a simple workflow. This is by no means a definitive list of the web site functionality, however provides a high level overview.

Editorial

The Quick Start web site is managed via Alfresco Share and the Alfresco Web Editor (for in-context editing). As described previously, all editorial activity takes place in the “Quick Start Editorial” folder within the document library. Folders are used within Share to define the site structure i.e. sections. Within the Quick Start example, both content and site structure reside within the same location and this combined model is dynamically delivered. This means that any content changes are updated immediately (barring any configured cache time) on the editorial site. For example, if a new folder is created under the “root” folder using Share, upon browser refresh, the new section will be displayed within the top level navigation. Note a folder has both a “Name” and “Title” field, the name being the URL and the title being the display label. Creating a new folder within the QS site structure automatically creates two other things below it. Firstly a “collections” folder and also an “index.html” file. The index.html file is the asset used by the section’s landing page. The “collections” folder is used to manage any “content collections” for that section. Content Collections are simply (as the name suggests!) collections of content assets, grouped as the content editor sees fit. For example, the home page on the QS site is made up of five different collections. Navigating to Quick Start Editorial > root > collections shows the collections used on the Home Page. Selecting “Edit Metadata” on any of them, will show the collection assets listed under the “Web Assets” field. The “news.featured” collection shown below is used to power the Home Page banner slider. You can see that 3 articles have been manually selected:

Web Assets on a Static Collection

Content collections whose content has been manually selected by an editor are referred to as “Static Collections”. Static collections are used in various places around the site where editorial control is required to select specific content assets. You may also notice that there is a “Query” field within the same metadata dialog. If populated, this then turns the collection into a “Dynamic Collection”, whereby the query is automatically run on a configured interval e.g. every 30 minutes. An example is shown at Quick Start Editorial > root > collections > blogs.latest collection. This collection is used by the “Latest Blog Articles” component on the right hand side of the home page. The default query is using CMIS (Lucene is also supported) and simply retrieves the latest five blog articles and orders them by date/time. Dynamic collections allow portions of the website to be automatically updated without any editorial intervention. The “Maximum Size” and “Minutes to query refresh” fields can be used to fine tune the dynamic collection as required. The keyword “section” can also be used within native CMIS queries. For example, to show all content items of type ws:article from the current section you could use “section:.” As follows:

select d.* from cmis:document as d where in_tree(d, '${section:.}') and d.cmis:objectTypeId='D:ws:article' order by d.cmis:creationDate desc

You can also reference an absolute section from the site root using ${section:/blog} or for a subsection of the current section you can use ${section:companies} or for the parent of the current section you can use: ${section:..}. You can also access site root using ${section:/}, or you can go really mad and use a combination ${section:/blog/../news/./companies}. As the query is just standard CMIS, you can also use standard property names such as cmis:contentStreamMimeType, so to return all PDF documents within the current section you could use:

select d.* from cmis:document as d where in_tree(d, '${section:.}') and d.cmis:objectTypeId='cmis:document' and d.cmis:contentStreamMimeType='application/pdf' order by d.cmis:creationDate desc

For each section of the website, content type to template mappings can be controlled from within Share. For example, if you edit the metadata on Quick Start Editorial > root > news, you will see the “Section Config” property set to “ws:indexPage=sectionpage1”. This maps all requests going to this section’s landing page (the index.html asset is of type ws:indexPage) to the sectionpage1 template. Another example (set on the root folder) is ws:article=articlepage1 which causes all assets of the ws:article type to be rendered using the articlepage1 template. This template can be seen when clicking on any news or blog article. This template mapping is hierarchical in that if a match is not found in the requested section, it will then look for a match in the parent section and so on until the root section is reached. Therefore, type-to-template mappings can be set site wide on the root folder. The template resolution algorithm also looks up the type hierarchy as part of this process, however I will not cover this in detail here. Renditions, powered by the Alfresco Rendition Service are used extensively across the Quick Start site. They allow content authors to publish the correct format/size of content optimized for web, without any manual intervention. Renditions are defined within a Spring config file (rendition-context.xml), and then these definitions can be configured for use with either a given file mime type or content type. Like the type-to-template mappings, this configuration is on a hierarchical section basis, with the option to inherit being available to set on each section. An example configuration can be seen on the Quick Start Editorial > root > news folder. The “Rendition Config” field has several mappings including ws:image=ws:featuredNewsThumbnail. This example specifies that all content of type “ws:image” will have a rendition created with the definition called “featuredNewsThumbnail” (defined in the Spring config file). To give it a test, upload a large file into the news section and attach it to a news article (by editing the metadata on the news article using the “Edit full metadata” option). When viewing the article on the web site you will see the renditions in use. A thumbnail or cropped banner slider image is displayed (if used on the front page) and a medium and large thumbnail when viewing the article directly. Using the node browser you can browse all of the generated renditions. Out of the box, the Alfresco Web Editor is configured for use when viewing either a news or blog article. In 3.4, as well as edits, the Web Editor also provides the option to create new content and delete content. ~~NOTE: currently the HEAD version includes the 3.3 AWE and does not build the 3.4 version yet.~~ UPDATE: The 3.3 awe.war is no longer included as part of the web app build from HEAD. The 3.4 AWE can be built separately using ant incremental-webeditor. Just drop the awe.war into the same container as the wcmqs.war.

In-Context Editing with the Alfresco Web Editor

The QS also provides two examples of managing user generated content (UGC) with Alfresco. There is a “Contact” form located on the Contact page and a “Comment” form located under each blog article. Both forms submit content to Alfresco via CMIS which is gathered in a Share Data List. The Contact form triggers a basic workflow and the Comment form provides the ability to flag a comment to an Administrator, which disables it from display i.e. visitor moderated.

Publishing

The QS also provides an example publishing mechanism. This is based on a workflow driven model whereby once content is approved it is moved locally (using the Transfer Service) from the “Quick Start Editorial” folder, into the “Quick Start Live” folder. The QS provides two sample workflows for achieving this. Firstly “Review and Publish Section Structure”. This is designed to allow content authors to easily publish specific sections of the site structure. This is done by selecting and initiating this workflow on the appropriate index.html file for the required section. The “Review and Publish Section Structure” workflow will publish the sections from the submitted one and all sections below. A section comprises the section itself, its index page, and all of its collections. Basically it's a "publish sub tree" mechanism. If you want to populate the entire site structure on live then publish the root section index.html. The “Review and Publish” workflow publishes either a single asset or group of assets (multi-select). If you use this workflow for a sections index.html, it will publish that section and its collections folder, but does not cascade down. It is worth noting you can also configure the web application to view the “Live” site structure by configuring the meta-data properties on the “Quick Start Live” folder. The default configuration sets the host address to 127.0.0.1, so if you are running the Quick Start locally, you can actually view the editorial environment on http://localhost:8080/wcmqs and the live on http://127.0.0.1:8080/wcmqs. Note that the Web Editor (AWE) is configured to be enabled on the Editorial content, and disabled on the Live. This is controlled by a flag “isEditorial” on the “Quick Start Editorial” which will also (when complete) dictate what is viewable via the live web application with regards to publishing go live and expiry dates.

Summary

I have certainly not covered all features here, but hopefully provided enough information to get started. The documentation will go in to a lot of detail, especially from a Developer perspective looking to extend the QS. Were hoping the QS will provide a valuable platform both from an evaluation perspective and also for both customers and partners looking to build Alfresco implementations.

Alfresco WCM Roadmap – August 2010

admin — Thu, 05 Aug 2010 08:42:23 +0000

Below are the slides from the Alfresco WCM Roadmap webinar that covered:

Release Schedule
WCM Quick Start
Project "Cheetah"
Project "Swift"

The webinar can be viewed here.

Alfresco WCM Roadmap 2010 (Cheetah & Swift)

Web Studio, Web Editor and the Web Editor Framework

admin — Thu, 10 Jun 2010 09:23:25 +0000

I have been receiving a steady stream of questions around Web Studio and enquiries around our strategy for future releases with regards to similar functionality, so thought it would be helpful to discuss the product focus. Firstly to clarify, Web Studio is no longer part of the Community Alfresco product so is no longer available for download. Web Studio was not part of an Enterprise release and was therefore not supported as part of the core product by Alfresco. The Surf framework that Web Studio utilised has been committed to Spring Source. The Surf framework is the underlying framework for Alfresco Share and Alfresco will continue to invest in Surf and surrounding developer tools. With 3.3 Alfresco started to heavily invest in WCM utilising the core Alfresco repository (non AVM). As part of this strategy we introduced the Web Editor (in-context editing) and Web Editor Framework (WEF). The Web Editor Framework provides a standard plug-in framework that people can develop any type of functionality on. The framework can be loosely considered to be a ribbon toolbar that can be customised e.g. adding custom tabs, buttons and any other UI required. Most importantly it is a common framework that Alfresco, the Alfresco community, partners, customers, the Spring community and anyone else can develop on. A lot of effort has gone into the way that toolbars, buttons and components in general can be packaged into a single file, allowing for new plug-in’s to easily be “dropped in” to a WEF environment. The Web Editor was Alfresco’s first core product (documented, supported) feature that was built utilising the Web Editor Framework i.e. the Web Editor has a dependency upon the WEF. The Web Editor addresses a single specific use case which is to edit content (semantic page content) in the context of the web site. In context editing. This functionality is something we are now looking to build on in future releases, expanding this functionality above and beyond simple editing. The Web Editor focus however of the next release (internally named "Swift") is still on the content editorial process for the Web Editor. There is also a great deal of interest in what I will term “presentation management”, and this is where Web Studio really sparked peoples thoughts around Alfresco WCM. By this I mean providing the ability for a site administrator to manage sites, site structures, templates, components, navigation etc. In order to provide these features in a consistent, supportable manner, an underlying model is required within the repository which can then be utilised by a delivery framework. As the WCM functionality moves forward, we will start to implement such a model, and utilise Spring Surf and the WEF to deliver the presentation management functionality. The Swift release remains focused on web content production, so no presentation management capabilities are in scope for this release, however the building blocks around the model and Spring Surf will start emerge. Speaking to Alfresco partners, customers and community member recently, I know people are already starting to implement the Web Editor Framework and build out their required functionality. We have current customers who have built presentation management capabilities with full custom clients, and the Web Editor Framework provides a powerful alternative to this route.