Facebook Launches Zero, A Text-Only Mobile Site For Carriers

February 16, 2010 · Leave a Comment
Filed under: FaceBook Integration, Featured 

At the Mobile World Congress 2010, Facebook’s Chamath Palihapitiya just finished a 20-minute keynote, outlining the social networking juggernaut’s mobile strategy. We’ll have a video of the full presentation up soon.

There were no major announcements, but Palihapitiya did introduce something interesting that hasn’t been publicly announced yet: Facebook Zero.

When you visit the URL zero.facebook.com using your mobile’s browser, you’ll see a notice that your carrier does not yet support the service – obviously because it was just launched and operators have yet to sign up.

Palihapitiya only touched on it for a couple of seconds, but from what I gathered it’s basically a text-only version of the Facebook service that carriers can offer to their subscribers at no charge. If a user then decides to switch from text-only to multimedia (e.g. view photos from their friends), mobile operators can start charging them for ‘premium’ data service.

This system is apparently called zero-rated pages, and allows operators to use a trimmed down version of a web application as a sort of teaser, driving the adoption of certain mobile services or apps, and more data usage revenue down the line.




Presumably, Facebook will offer Facebook Zero to carriers for free, since it helps them make the social network as ubiquitous on mobile phones as possible.

We have an e-mail in with Facebook PR for more information.

Update: Facebook spokesperson Brandee Barker writes back:

“Zero” is a light-weight version of m.facebook.com that omits data intensive applications like Photos. It will launch in coming weeks and we are discussing it at MWC as an option to make Facebook on the mobile web available to everyone, anywhere and allow operators to encourage more mobile Internet usage.

Facebook- Flips The Switch On Its Improved Photo Uploader

February 6, 2010 · Leave a Comment
Filed under: Facebook, Featured, Technology News 

It’s no secret that Facebook Photos is massively popular, with the company boasting that it receives a whopping 2.5 billion photos uploaded every month. And as the site continues to grow — it just passed 400 million users — that number is only going to get more staggering. Unfortunately, the photo uploading experience hasn’t always been smooth; you’ve to wait for the photo uploads to finish, and the entire process was just sort of clunky. Today, the company announced that it’s rolling out an improved photo upload browser plugin.

The new plugin includes a new photo navigator that should make it easy to choose the photos you want. And once you’ve started the upload process, you can browse to another page — the plugin will keep uploading the files in the background.

This plugin has actually been available for the last few months in Facebook’s Prototypes section, and has even been rolled out to a very small number of users before now. Starting today it’s getting released to everyone, though the rollout will take a few weeks. Before now Facebook has offered an upload plugin based on Active X and a Java applet, but the company says this one was built entirely in-house.  One important thing to note: because this was built by Facebook, the company could potentially add more functionality in the future, including things that aren’t related to photo uploads.

Of course, there are plenty of other ways to upload photos to Facebook. The functionality has been integrated into iPhoto, you can download a Picasa Uploader, and there are plenty of unofficial apps available too

Facebook’s Project Titan: A Full Featured Webmail Product from FaceBook

February 5, 2010 · Leave a Comment
Filed under: Facebook, Featured, Technology News 

Facebook is completely rewriting their messaging product and is preparing to launch a fully featured webmail product in its place, according to a source with knowledge of the product. Internally it’s known as Project Titan. Or, unofficially and perhaps over-enthusiastically, the Gmail killer.

Facebook messaging has been the bane of users’ existence for years. My first public gripe was in 2008, when I said that urgent changes were needed. The biggest problem is simply deleting old emails. It takes so long that I have thousands of unread and read but not deleted messages in my inbox.

But Facebook messaging is also only indirectly linked to the email, which is still the standard way that people exchange digital messages when not on Facebook.

Facebook has occasionally dabbled with improvements to messaging, like adding the ability to search messages. But for the most part it has remained static. And not very useful.

Even MySpace moved away from their aging messaging platform to a true webmail service in 2008 (albeit one that lacked POP or IMAP support).

But now Facebook is getting itself back in the game. And if the details we’ve heard are accurate, Project Titan, or whatever it’s called when it launches, may be the kind of product people flock to.

First, our understanding is that there will be full POP/IMAP support, meaning users can access the account other than through Facebook itself. Your email account name will be your vanity url – vanityurl@facebook.com.

Email is all about identity. And Facebook is ahead of everyone else in the identity game via Facebook Connect. Facebook says more than 60 million people log in to 80,000 third party websites each month via Facebook Connect.

Tacking a real webmail product on top of those vanity URLs and Facebook connect is something even Google may shudder at. Gmail killer? I don’t think so. But a strong product move nonetheless.

CrunchBase Information

Facebook

Information provided by CrunchBase

Simple Steps to Publish On FaceBook Fan Page Using PHP

January 26, 2010 · 14 Comments
Filed under: FaceBook Integration, Featured, PHP 

After Done with my previous post http://blog.theunical.com/facebook-integration/5-steps-to-publish-on-a-facebook-wall-using-php/ that only publish’s on a Facebook wall.

So now we will see how to publish on a facebook Fan’s page.

First we need to follow the 5 steps that I described in my previous post Next just create a Fan Page in Facebook.

Then get the page ID that you can get by the Page URL.

Then replace the APIKey, Secret Key and Session Key with your keys.

Then replace $target_id with your page id. That is Fan’s Page id.

Here is the code in PHP to publish on a Facebook FAN Page

<?php
define('FB_APIKEY', 'APIKEYxx');
define('FB_SECRET', 'SECRETKEYxxxxxxxxxxxxxxxxxxx');
define('FB_SESSION', 'SESSIONKEYxxxxxxxxxxxxxxxxxxxx');

require_once('facebook-platform/php/facebook.php');

try {
$facebook = new Facebook(FB_APIKEY, FB_SECRET);
$facebook->api_client->session_key = FB_SESSION;
$fetch = array('friends' =>
array('pattern' => '.*',
'query' => "select uid2 from friend where uid1={$user}"));
echo $facebook->api_client->admin_setAppProperties(array('preload_fql' => json_encode($fetch)));

$message = 'From My App: publish steven on facebook Fan Page';

$target_id ="273641842409";

if( $facebook->api_client->stream_publish($message, $attachment,$action_links, $target_id))
echo "Added on FB Wall on Fan Page";
} catch(Exception $e) {
echo $e . "<br />";

}
?>

publish on facebook fan wall page using php
That’s it your are now done enjoy!!! publishing on fan’s page wall.

You can see my Fan Page here http://www.facebook.com/pages/Fan-Page/273641842409?v=wall Where I published using above example.

Facebook status update – PHP

December 1, 2009 · Leave a Comment
Filed under: FaceBook Integration, Featured 

To update a users status on Facebook, you should look at the Status.set API call.

For coding a googletalk bot in php there are some articles that should get you started:

Post Image on Facebook using java api to publish on Wall

November 8, 2009 · 6 Comments
Filed under: FaceBook Integration, Featured, JAVA 

First Follow the listed Steps from my Previous post http://blog.theunical.com/facebook-integration/facebook-java-api-example-to-publish-on-wall/ then in last program include the below code to publish a photo and also meta data on a facebook wall.

In the same way we can even publish MP3, Videos, Flash on facebook.

Publishing a Post Containing an Image, Action Links and Custom Metadata.

package com.unical;

import org.apache.commons.httpclient.HttpClient;
import org.apache.commons.httpclient.HttpState;
import org.apache.commons.httpclient.NameValuePair;
import org.apache.commons.httpclient.methods.GetMethod;
import org.apache.commons.httpclient.methods.PostMethod;
import org.apache.commons.httpclient.params.HttpClientParams;

import com.google.code.facebookapi.Attachment;
import com.google.code.facebookapi.AttachmentMediaImage;
import com.google.code.facebookapi.FacebookException;
import com.google.code.facebookapi.FacebookJsonRestClient;
import com.google.code.facebookapi.FeedFacebookPhoto;
import com.google.code.facebookapi.Permission;
import com.google.code.facebookapi.TemplatizedAction;

public class SendtoFacebook {

	public static void main (String a[]) throws FacebookException{ 

	SendtoFacebook sfb=new SendtoFacebook();
	sfb.send("From My App: publish steven on facebook");
	}
	public void send(String message)throws FacebookException{

		String FB_APP_API_KEY = new String("YOUR APIKEY");
		String FB_APP_SECRET = new String("YOUR SECRET");
		String FB_SESSION_KEY = new String("YOUR SESSION KEY");
		FacebookJsonRestClient facebook = new FacebookJsonRestClient( FB_APP_API_KEY, FB_APP_SECRET, FB_SESSION_KEY );

		//FacebookJsonRestClient facebookClient2 = (FacebookJsonRestClient)facebook.getFacebookRestClient();
		FacebookJsonRestClient facebookClient = (FacebookJsonRestClient)facebook;
		Attachment attachment = new Attachment();
		attachment.setCaption("Caption for attachment");
		attachment.setDescription("Description for attachment");
		attachment.setHref("http://blog.unical.com");
		attachment.setName("Hopefully this works");

		AttachmentMediaImage  attach_media = new AttachmentMediaImage( );
		attach_media.addImage("http://profile.ak.fbcdn.net/v229/868/86/q518210788_2085.jpg", "http://blog.theunical.com");

		attachment.setMedia(attach_media);

		facebookClient.stream_publish(message, attachment, null, null, null);

			System.out.println("successfully updated");
	}
}


Facebook Java api example to publish on Wall

November 8, 2009 · 79 Comments
Filed under: FaceBook Integration, Featured, JAVA 

According to facebook Wiki posting on a  wall is very confusing and very difficult to write a program to publish on a wall. And it Don’t even support or document any methods on how to publish on wall using JAVA, and thanks to Google for giving a JAVA library to post on facebook.

Download Library from http://code.google.com/p/facebookjavaapi/

Don’t confuse and don’t use old libraries only use facebook-java-api-3.0.1-bin.zip or greater.

I spent around 2 weeks reading many articles and many documents, Wiki on facebook but in vein and tried many ways and finally I got one workable model. Below are simple steps to publish on a Facebook wall using JAVA.

Step1: Login to Facebook

First login to Facebook and goto the url http://www.facebook.com/developers/ then

Click on “+ Set Up New Application” button to start creating the application as shown below.

start application

Provide the application name, click agree for facebook terms and click on “Create Application” button.

Fill in application Name

create app step1

Now Application is created for you. Just copy the Application API Key, Application Secret and Application ID details which will be used to write the application code.

application page

Now goto Canvas link and enter any url.

canvaspage

Next goto Connect URL link and enter the same url.

connecturl

Then click on Save. That’s it we have created a facebook application with an Iframe.

Now Copy Application API Key, Application Secret and Application ID details which will be used to write the application code.

Now download the php library from facebook “http://blog.theunical.com/wp-content/uploads/2010/05/facebook-platform.tar1.gz”. For now we will use this library for generating one time session key.

Step2: Generate One Time Token key

How to generate One Time Token key:

Run below URL to get temporary token for the particular user by logging into Facebook.

https://login.facebook.com/code_gen.php?api_key=API_KEY&v=1.0

Note: Please replace “API_KEY” with your application API key from above page.

Then we will get a temporary token key.

allowaccess

generatemylogin info

savemyinfo

This is one time token key that we can use to generate a permanent session key. Careful! Don’t try to execute this programs many times

Step3: Generate one time session key

Below is the sample JAVA code to generate one time session key.

/**
 * FacebookClient
 *
 */
public class FacebookClient {
 public static String API_KEY = “YOUR API KEY”;
 public static String SECRET = “YOUR SECRET”;
 public static void main(String args[]) {
 // Create the client instance
 FacebookRestClient client =       new FacebookRestClient(API_KEY, SECRET);
 client.setIsDesktop(true); // is this a desktop app

 try {
 String token = client.auth_createToken();
 // Build the authentication URL for the user to fill out
 String url = “http://www.facebook.com/login.php?api_key=”
 + API_KEY + “&v=1.0″
 + “&auth_token=” + token;
 // Open an external browser to login to your application
 Runtime.getRuntime().exec(“open ” + url); // OS X only!
 // Wait until the login process is completed
 System.out.println(“Use browser to login then press return”);
 System.in.read();

 // fetch session key
 String session = client.auth_getSession(token,true );
// obtain temp secret
String tempSecret = client.getSessionSecret();
// new facebook client object
client = new FacebookJaxbRestClient(API_KEY, tempSecret, sessionKey);

 System.out.println(“Session key is ” + session);

 // keep track of the logged in user id
 Long userId = client.users_getLoggedInUser();
 System.out.println(“Fetching friends for user ” + userId);

 // Get friends list
 client.friends_get();
 FriendsGetResponse response = (FriendsGetResponse) client.getResponsePOJO();
 List<Long> friends = response.getUid();

 // Go fetch the information for the user list of user ids
client.users_getInfo(friends, EnumSet.of(ProfileField.NAME));

UsersGetInfoResponse userResponse = (UsersGetInfoResponse) client.getRepsonsePOJO();
 // Print out the user information
 List<User> users = userResponse.getUser();
 for (User user : users) {
 System.out.println(user.getName());
 }
 } catch (FacebookException e) {
 // TODO Auto-generated catch block
 e.printStackTrace();
 } catch (IOException e) {
 // TODO Auto-generated catch block
 e.printStackTrace();
 }
 }
}

(OR)

For example, the full URL for logging in a user could be:

http://www.facebook.com/login.php?api_key=YOURAPIKEY&connect_display=popup&v=1.0&next=http://www.facebook.com/connect/login_success.html&cancel_url=http://www.facebook.com/connect/login_failure.html&fbconnect=true&return_session=true&session_key_only=true&req_perms=read_stream,publish_stream,offline_access

If the user is redirected to the URL specified by the next parameter, then Facebook grants your application a session. This session is appended to the URL as a JSON-decodable object of the form:

&session={“session_key”:”SESSIONKEY”, “uid”:USERID, “expires”:0 || UNIXTIME, “secret”:”SESSIONSECRET”}

In continuing with the above example, the redirect for a successful login would be:

http://www.facebook.com/connect/login_success.html?session=%7B%22session_key%22%3A%223.kxhAu6W0qo_bLGjmdWrgfw__.86400.1243443600-688626964%22%2C%22uid%22%3A%22688626964%22%2C%22expires%22%3A1243443600%2C%22secret%22%3A%220NVNMxpO6jVyDcVCvVv_PA__%22%2C%22sig%22%3A%22ac1c0c77c137567389defea70481b7aa%22%7D

If the user grants your application the offline_access extended permission, 0 gets returned for expires and the session never expires unless the user removes the application. In this case, you should store the session key so the user doesn’t have to log in the next time he or she launches your application.

Once the browser has been redirected successfully and you have your session information, you should automatically close the browser window.
Note: The above code will execute only once. We should note down the above session key, This will be used to auto login into the application.

Note: The above code will execute only once. We should note down the above session key, This will be used to auto login into the application.

If you don’t save these session key values you should generate other token key to create one time session key.

We can use the token only once.

Step4: Setting Permission for wall

Giving Permission to your application to publish on facebook wall. To give permission just replace the your API key in below URL and execute in browser.

http://www.facebook.com/login.php?api_key=APIKEYXxxxxxxxxxxxxxxxxx&connect_display=popup&v=1.0&next=http://www.facebook.com/connect/login_success.html&cancel_url=http://www.facebook.com/connect/login_failure.html&fbconnect=true&return_session=true&req_perms=read_stream,publish_stream,offline_access

Then you must see the below 3 Screens

If you don’t see the below then your call back URL or Canvas URL must be incorrect, please check again your application settings and execute the URL again.

allowpermission

allowpublishing

Then on final page, you will see a success message.  That’s it you have given permission for read_stream,publish_stream and offline_access.

Step5: Publishing the message on Facebook Wall

Java Code to create facebook object with onetime session key and publishing the message on Facebook Wall.

import org.apache.commons.httpclient.HttpClient;
import org.apache.commons.httpclient.HttpState;
import org.apache.commons.httpclient.NameValuePair;
import org.apache.commons.httpclient.methods.GetMethod;
import org.apache.commons.httpclient.methods.PostMethod;
import org.apache.commons.httpclient.params.HttpClientParams;

import com.google.code.facebookapi.Attachment;
import com.google.code.facebookapi.AttachmentMediaImage;
import com.google.code.facebookapi.FacebookException;
import com.google.code.facebookapi.FacebookJsonRestClient;
import com.google.code.facebookapi.FeedFacebookPhoto;
import com.google.code.facebookapi.Permission;
import com.google.code.facebookapi.TemplatizedAction;

public class SendtoFacebook {

	public static void main (String a[]) throws FacebookException{ 

	SendtoFacebook sfb=new SendtoFacebook();
	sfb.send("From My App: publish steven on facebook");
	}
	public void send(String message)throws FacebookException{

		String FB_APP_API_KEY = new String("YOUR_APIKEY");
		String FB_APP_SECRET = new String("YOUR_SECRET");
		String FB_SESSION_KEY = new String("YOUR_ONETIMESESSIONKEY");
		FacebookJsonRestClient facebook = new FacebookJsonRestClient( FB_APP_API_KEY, FB_APP_SECRET, FB_SESSION_KEY );

		//FacebookJsonRestClient facebookClient2 = (FacebookJsonRestClient)facebook.getFacebookRestClient();
		FacebookJsonRestClient facebookClient = (FacebookJsonRestClient)facebook;
facebookClient.stream_publish(message, null, null, null, null);

			System.out.println("successfully updated");

	}
}

Now you will see the post on wall Great.

published on facebook

So thrilling isn’t we can also publish photos videos on wall.

http://wiki.developers.facebook.com/index.php/Stream.publish

have a great day

– Steven

Steven Fan Page on Facebook

Please don’t copy this content to any site. This is fully protected by TheUnical Technologies

5 Steps to publish on a facebook wall using php

November 1, 2009 · 308 Comments
Filed under: FaceBook Integration, Featured, PHP 

Since Facebook has stopped supporting Java we need to use PHP for now to post on a wall.

According to facebook Wiki posting on a  wall is very confusing and very difficult to write a program to publish on a wall.

I spent around 2 weeks reading many articles and many documents, Wiki on facebook but in vein and tried many ways and finally I got one workable model. Below are 5 simple steps to publish on a Facebook wall.

Step1: Login to Facebook to Create a Facebook App

First login to Facebook and goto the url http://www.facebook.com/developers/ then

Click on “+ Set Up New Application” button to start creating the application as shown below.

Create New Facebook application

Provide the application name, click agree for facebook terms and click on “Create Application” button.

Fill in application Name

create app step1

Now Application is created for you. Just copy the Application API Key, Application Secret and Application ID details which will be used to write the application code.

Fill the Name and Description of your application in About Link.

Facebook Application About Page

Now Click on Website Link to get API Key

Facebook Application web Site

Now goto Facebook Integration link and enter any url. Here is where you will get Secret Key.

Enter the Canvas URL and page name. This will be the return URL of your Application.

Facebook Application Facebook Integration

That’s it You now just successfully create a Facebook Application, now lets work on program

Then click on Save. That’s it we have created a facebook application with an Iframe.

Now Copy Application API Key, Application Secret and Application ID details which will be used to write the application code.

Now download the php library from facebook “http://blog.theunical.com/wp-content/uploads/2010/05/facebook-platform.tar.gz

Step2: Generate One Time Token to generate Session key

How to generate One Time Session key:

Run below URL to get temporary token for the particular user by logging into Facebook.

https://login.facebook.com/code_gen.php?api_key=API_KEY&v=1.0

Note: Please replace “API_KEY” with your application API key from above page.

Then we will get a temporary token key.

allowaccess

generatemylogin info

savemyinfo

This is one time token key that we can use to generate a permanent session key. Careful! Don’t try to execute this programs many times

Step3: Generate one time session key (permanent session key)

Below is the sample PHP code to generate one time session key.

<?php

// FB_APIKEY is your facebook application api key

// FB_SECRET is your application secrete key

$FB_APIKEY="YOUR_API";

$FB_SECRET="YOUR_SECRET";

$fb = new FacebookRestClient($FB_APIKEY, $FB_SECRET);

$testtoken= "ONETIMETOKEN"; // Replace this value with your Token Value

$result = $fb->call_method('facebook.auth.getSession',

array('auth_token' => $testtoken, 'generate_session_secret' => true));

echo "<br /><pre>";

print_r($result);

echo $session_key = $result['session_key'];

?>

(OR)

For example, the full URL for logging in a user could be:

http://www.facebook.com/login.php?api_key=YOURAPIKEY&connect_display=popup&v=1.0&next=http://www.facebook.com/connect/login_success.html&cancel_url=http://www.facebook.com/connect/login_failure.html&fbconnect=true&return_session=true&session_key_only=true&req_perms=read_stream,publish_stream,offline_access

If the user is redirected to the URL specified by the next parameter, then Facebook grants your application a session. This session is appended to the URL as a JSON-decodable object of the form:

&session={“session_key”:”SESSIONKEY”, “uid”:USERID, “expires”:0 || UNIXTIME, “secret”:”SESSIONSECRET”}

In continuing with the above example, the redirect for a successful login would be:

http://www.facebook.com/connect/login_success.html?session=%7B%22session_key%22%3A%223.kxhAu6W0qo_bLGjmdWrgfw__.86400.1243443600-688626964%22%2C%22uid%22%3A%22688626964%22%2C%22expires%22%3A1243443600%2C%22secret%22%3A%220NVNMxpO6jVyDcVCvVv_PA__%22%2C%22sig%22%3A%22ac1c0c77c137567389defea70481b7aa%22%7D

If the user grants your application the offline_access extended permission, 0 gets returned for expires and the session never expires unless the user removes the application. In this case, you should store the session key so the user doesn’t have to log in the next time he or she launches your application.

Note: The above code will execute only once. We should note down the above session key, This will be used to auto login into the application.

If you don’t save these session key values you should generate other token key to create one time session key.

We can use the token only once.


Step4: Setting Permission for wall

Giving Permission to your application to publish on facebook wall. To give permission just replace the your API key in below URL and execute in browser.

http://www.facebook.com/login.php?api_key=APIKEYXxxxxxxxxxxxxxxxxx&connect_display=popup&v=1.0&next=http://www.facebook.com/connect/login_success.html&cancel_url=http://www.facebook.com/connect/login_failure.html&fbconnect=true&return_session=true&req_perms=read_stream,publish_stream,offline_access

Then you must see the below 3 Screens

If you don’t see the below then your call back URL or Canvas URL must be incorrect, please check again your application settings and execute the URL again.

allowpermission

allowpublishing

Then on final page, you will see a success message.  That’s it you have given permission for read_stream,publish_stream and offline_access.

Step5: Publishing the message on Facebook Wall


PHP Code to create facebook object with onetime session key and publishing the message on Facebook Wall.

<?php

define('FB_APIKEY', 'YOUR_APIKEY');

define('FB_SECRET', 'YOUR_SECRET');

define('FB_SESSION', 'YOUR_SESSION_key');

require_once('facebook-platform/php/facebook.php');

echo "post on wall";

try {

$facebook = new Facebook(FB_APIKEY, FB_SECRET);

$facebook->api_client->session_key = FB_SESSION;

$fetch = array('friends' =>

array('pattern' => '.*',

'query' => "select uid2 from friend where uid1={$user}"));

echo $facebook->api_client->admin_setAppProperties(array('preload_fql' => json_encode($fetch)));

$message = 'From My App: publish steven on facebook';

if( $facebook->api_client->stream_publish($message))

echo "Added on FB Wall";

} catch(Exception $e) {

echo $e . "<br />";

}

?>

Now you will see the post on wall Great.

published on facebook

So thrilling isn’t we can also publish photos videos on wall.

In my next post see how to publish images and videos to facebook is done

http://blog.theunical.com/facebook…/add-picture-on-facebook-wall-using-php/

http://blog.theunical.com/facebook…/post-video-on-facebook-wall-using-php/

See here how to publish on facebook page as a facebook user

http://wiki.developers.facebook.com/index.php/Stream.publish

have a great day

— Steven

Steven Fan Page on Facebook

Please don’t copy this content to any site. This is fully protected by TheUnical Technologies

Post Laster Updated on 13 March 2011

ASP.NET MVC Facebook Connect

September 27, 2009 · Leave a Comment
Filed under: Featured, MVC Frameworks 

We need someone to develop in C# ASP.NET MVC Framework a Facebook Connect logon routine. Using C# and ASP.NET MVC Framework, the canidate will need to develop a working sample web site which allows a user to sign in via Facebook Connect.


Bidmus – http://bidmus.net/

10 Ways To Complement the Enterprise RDBMS Using Hadoop

September 7, 2009 · Leave a Comment
Filed under: Featured, MySql, MySql 5.0, MySQL 5.1 

When it comes to data, the workhorse relational database has been the tool of choice for businesses for well over 20 years now. Challengers have come and gone but the trusty RDBMS is the foundation of almost all enterprise systems today. This includes almost all transactional and data warehousing systems. The RDBMS has earned its place as a proven model that, despite some quirks, is fundamental to the very integrity and operational success of IT systems around the world.

However, as I discussed earlier this year in my 10 Must-Know Topics for Software Architects in 2009, the relational database is finally showing some signs of age as data volumes and network speeds grow faster than the computer industry’s present compliance with Moore’s Law can keep pace with. The Web in particular is driving innovation in new ways of processing information as the data volumes of Internet-scale applications become prohibitive using traditional SQL database engines.

When it comes to database processing today, change is being driven by (at least) four factors:

  1. Speed. The seek times of physical storage is not keeping pace with improvements in network speeds.
  2. Scale. The difficulty of scaling the RDBMS out efficiently (i.e. clustering beyond a handful of servers is notoriously hard.)
  3. Integration. Today’s data processing tasks increasingly have to access and combine data from many different non-relational sources, often over a network.
  4. Volume. Data volumes have grown from tens of gigabytes in the 1990s to hundreds of terabytes and often petabytes in recent years.

Each of these factors is presently driving interest in alternatives that are significantly better at dealing with these requirements. I’ll be clear here: The relational database has proven to be incredibly versatile and is the right tool for the majority of business needs today. However, the edge cases for many large-scale business applications are moving out into areas where the RDBMS is often not the strongest option.One of the most discussed new alternatives at the moment is Hadoop, a popular open source implementation of MapReduce. MapReduce is a simple yet very powerful method for processing and analyzing extremely large data sets, even up to the multi-petabyte level. At its most basic, MapReduce is a process for combining data from multiple inputs (creating the “map”), and then reducing it using a supplied function that will distill and extract the desired results. It was originally invented by engineers at Google to deal with the building of production search indexes. The MapReduce technique has since spilled over into other disciplines that process vast quantities of information including science, industry, and systems management. For its part, Hadoop has become the leading implementation of MapReduce.

While there are many non-relational database approaches out there today (see my emerging IT and business topics post for a list), nothing currently matches Hadoop for the amount of attention it’s receiving or the concrete results that are being reported in recent case studies. A quick look at the list of organizations that have applications powered by Hadoop includes Yahoo! with over 25,000 nodes (including a single, massive 4,000 node cluster), Quantcast which says it has over 3,000 cores running Hadoop and currently processes over 1PB of data per day, and Adknowledge who uses Hadoop to process over 500 million clickstream events daily using up to 200 nodes.

10 Ways To Complement the Enterprise RDMS Using Hadoop

These datasets would previously have been very challenging and expensive to take on with traditional RDBMS and standard bulk load and ETL approaches, never mind efficiently combining multiple data sources or dealing with volumes of data that simply can’t reside on a single machine (or often even dozens). Hadoop deals with this by using a distributed file system (HDFS) that’s designed to deal coherently with datasets that can only reside across distributed server farms. HDFS is also fault resilient and so doesn’t impose the requirement of RAID drives on individual nodes in a Hadoop compute cluster, allowing the use of truly low cost commodity hardware.

So what does this specifically mean to enterprise users that would like to improve their data processing capabilities? Well, first there some catches to be aware of. Despite enormous strengths in distributed data processing and analysis, MapReduce is not good in some key areas that the RDMS is extremely strong in (and vice versa). The MapReduce approach tends to have high latency (i.e. not suitable for real-time transactions) compared to relational databases and is strongest at processing large volumes of write-once data where most of the dataset needs to be processed at one time. The RDBMS excels at point queries and updates, while MapReduce is best when data is written once and read many times.The story is the same with structured data, where the RDBMS and the rules of database normalization identified precise laws for preserving the integrity of structured data that have stood the test of time. MapReduce is designed for a less structured, more federated world where schemas may be used but data formats can be much looser and freeform.

RDBMS and Hadoop: Apples and Oranges?

Here is a comparison of the overall differences between the RDBMS and MapReduce-based systems such as Hadoop:

RDBMS MapReduce
Data size Gigabytes Petabytes
Access Interactive and batch Batch
Structure Fixed schema Unstructured schema
Language SQL Procedural (Java, C++, etc)
Integrity High Low
Scaling Nonlinear Linear
Updates Read and write Write once, read many times
Latency Low High

Figure 1: Comparing RDBMS to MapReduce

From this it’s clear that the MapReduce model cannot replace the traditional enterprise RDBMS. However, it can be a key enabler of a number of interesting scenarios that can considerably increase flexibility, turn-around times, and the ability to tackle problems that weren’t possible before.

With the latter the key is that SQL-based processing of data tends not to scale linearly after a certain ceiling, usually just a handful of nodes in a cluster. With MapReduce, you can consistently get performance gains by increasing the size of the cluster. In other words, double the size of Hadoop cluster and a job will run twice as fast, triple it and the same thing, etc.

Ten Ways To Improve the RDBMS with Hadoop

So Hadoop can complement the enterprise RDMS in a number of powerful ways. These include:

  1. Accelerating nightly batch business processes. Many organizations have production transaction systems that require nightly processing and have narrow windows to perform their calculations and analysis before the start of the next day. Since Hadoop can scale linearly, this can enable internal or external on-demand cloud farms to dynamically handle shrink performance windows and take on larger volume situations that an RDBMS just can’t easily deal with. This doesn’t elide the import/export challenges depending on the application but can certainly compress the windows between them.
  2. Storage of extremely high volumes of enterprise data. The Hadoop Distributed File System is a marvel in itself and can be used to hold extremely large data sets safely on commodity hardware long term that otherwise couldn’t stored or handled easily in a relational database. I am specifically talking about volumes of data that today’s RDBMS’s would still choke on, such as dozens or hundreds of petabytes, which is common in genetic, physics, satellite data, and other scientific, medical, and government applications.
  3. Creation of automatic, redundant backups. Hadoop can then keep the data that it processes, even after it it’s been imported into other enterprise systems. HDFS creates a natural, reliable, and easy-to-use backup environment for almost any amount of data at reasonable prices considering that it’s essentially a high-speed online data storage environment.
  4. Improve scalability of applications. Low cost commodity hardware can be used to power Hadoop since redundancy and fault resistance is built into the software, instead of using expensive enterprise hardware or software alternatives with proprietary solutions. Adding more capacity Hadoop is an affordable and very granular way to scale out instead of up. While there can be cost in converting existing applications to Hadoop but for new applications, it should be a standard option in the software selection decision tree. Note: Hadoop’s fault tolerance is acceptable, not best-of-breed, so check this against your application’s requirements.
  5. Use of Java for data processing instead of SQL. Hadoop is a Java platform and can be used by just about anyone fluent in the language (other language options are coming available soon via APIs.) While this won’t help shops that have plenty of database developers, Hadoop can be a boon to organizations that have strong Java environments with good architecture, development, and testing skills. And while yes, it’s possible to use languages such as Java and C++ to write stored procedures for an RDBMS, it’s not a widespread activity.
  6. Producing just-in-time feeds for dashboards and business intelligence. Hadoop excels at looking at enormous amounts of data and providing detailed analysis of business data that an RDBMS would often take too long or would be too expensive to carry out. Facebook, for example, uses Hadoop for daily and hourly summaries of its 150 million+ monthly visitors. The resulting information can be quickly transferred to BI, dashboards, or mashup platforms.
  7. Handling urgent, ad hoc requests for data. While certainly expensive enterprise data warehousing software can do this, Hadoop is a strong performer when it comes to quickly asking and getting answers to urgent questions involving extremely large datasets.
  8. Turning unstructured data into relational data. While ETL tools and bulk load applications work well with smaller datasets, few can approach the data volume and performance that Hadoop can, especially at a similar price/performance point. The ability to take mountains of inbound or existing business data, spread the work over a large distributed cloud, add structure, and import the result into an RDBMS makes Hadoop one of the most powerful database import tools around.
  9. Take on tasks that require massive parallelism. Hadoop has been known to scale out to thousands of nodes in production environments. Even better, It requires relatively little innate programing skill to achieve since parallelism is an intrinsic property of the platform. While you can do the same with SQL, it requires some skill and experience with the techniques. In other words, you have to know what you’re doing. For organizations that are experiencing ceilings with their current RDBMS, you can look at Hadoop to help break through them.
  10. Moving existing algorithms, code, frameworks, and components to a highly distributed computing environment. Done right — and there are challenges depending on what your legacy code wants to do — and Hadoop can be used as a way to migrate old, single core code into a highly distributed environment to provide efficient, parallel access to ultra-large datasets. Many organizations already have proven code that is tested and hardened and ready to use but is limited without an enabling framework. Hadoop adds the mature distributed computing layer than can transition these assets to a much larger and more powerful operating environment.

Hadoop is growing in leaps and bounds these days and there are additions or extensions to meet a wide variety of needs. Just a few examples include Pig, a “platform for analyzing large data sets that consists of a high-level language for expressing data analysis programs, coupled with infrastructure for evaluating these programs” that enables “embarassingly parallel” data analysis tasks. There is HBase, which actually enables random, read/write access to large datasets. Also worth mentioning is HadoopDB which attempt to merge the methods of MapReduce and the RDBMS.

Whichever the case, while Hadoop is still an emerging technology, what current makes it special is its proven ability to perform well in many, well-known production applications. This makes it likely that you’ll see it considered more and more often in next-generation enterprises. I’ll keep track of it here as I can, in the meantime, I recommend spending some time understanding how Hadoop works and getting a feel for what it can do in your enterprise when you need it.

Are you planning to use Hadoop in your organization? Why or why not?

« Previous Page