IBM supporting SPARK

Working with our Government clients worldwide I have seen great innovative Big Data projects.  As data volume and algorythms get more complex the need to manage those interactions have put a strain on the existing technologies.  Today IBM announced support for SPARK.  This is a very exciting announcement and I encourage you to learn more about what this means for the world of Analytics and Big Data.

To summarize the announcement IBM will be building Spark into the core of our analytics and commerce platforms.

Open sourcing our IBM SystemML machine learning technology and collaborating with Databricks to advance Spark’s machine learning capabilities.

IBM will be offering Spark as a service on IBM Bluemix to enable app developers to quickly load and model data.

Opening a Spark Technology Center in San Francisco for the Data Science and Developer community to be at the center of Spark innovation, collaborating closely with AMPLab and Databricks.

Educating at least 1 million data scientists and data engineers on Spark through partnerships with AMPLab, DataCamp, MetiStream, and Big Data University MOOC

This represents a commitment of more than 3,500 researchers and developers working on Spark-related projects at more than a dozen labs worldwide.
Follow @IBMBigData and @IBMAnalytics on Twitter

Watch the Livestream on IBM.com at IBM.com/spark on June 15th at 7 pm PDT. There will also be a replay. 

Visit the Spark Summit Website

Big Data Reality

When working with government clients it amazes me the broad description and ideas that are included in talking about big data.  The biggest hurdle to these types of projects is that IT or a program lead has an idea for data analysis but culturally the agency or program is not thinking about what Big Data could answer for them.

One area in a business case that is attempting to justify a Big Data or Analytics project that does not get a lot of attention is the organizational change management ( this is true for all IT projects I find ).  So how do you instill a culture where end users and executive

What I see are proof of concepts that try to pick an area that will then instil curiousity across the various business units.  So they can see what is possible.  Or if they have clear data sets then asking what else can we ask of them.

Another challenge I see is that most organizations have skill sets and knowledge as it applies to business intelligence tools or other analytics; however when you start to look at what a data scientist does and the algorithms they develop it becomes apparent that most organizations are lacking in the proper skills to implement a Big Data project.

Currently we are looking at helping government organizations by “hooking” them up with data scientist in order for them to play with data sets and try to answer some complicated questions: like workforce development or underground economics.  Predicting and understanding so that programs and legislation are more effective at their mandates.

Back to the Future

I know I have not posted in quite some time.  A few changes have occured in that my responsiblity at work as broadened to include more of the analytics offerings from IBM.  This move is exciting and has opened my eyes to the great things that governments are doing worldwide in the areas of Big Data and Analytics.

What I do see are government agencies still wrestling with Big Data.  What actually is it ? and what can I do with it ?  We are actively helping agencies see what is possible with Big Data by aligining Data Scientists with best practices so that agencies actually get to see the art of the possible.

The other observation I have made is that due to my ECM background I am sensitive to information lifecycle goverance (ILG) or compliance issues.   Some agencies when looking at Big Data just think they can throw data sets up into areas such as Hadoop and a lot of this information is unstructured still and without proper metadata managemnet these Big Data projects are putting the agency at risk for loss or improper storage of the information they are attempting to use.

I guess it comes down to fundamentals still.  Regardless of the innovative use of data etc.  The basics of information governance still rule supreme.

Stay tuned for my next post around Cyber Security and threat prediciton.