Connect – a Simba Technologies Blog

Amazon's new Big Data service - DynamoDB

Jim Ericson at Information Management writes about DynamoDB - Amazon's new Big Data service here: http://www.information-management.com/news/Amazon_big_data_cloud-10021812-1.html.  Worth a quick read.  DynamoDB builds on SimpleDB, and Ericson writes that "since it is a non-traditional database, DynamoDB will face a limited audience that the company hopes will grow as the service is approached and tested."  One thing about the so-called NoSQL Big Data databases is that they are simple to use and very powerful.  The challenge is to use these within the existing IT landscape.  Both Cloudera and Microsoft are promoting ODBC drivers as the way to bridge between the SQL and NoSQL worlds.  I think this is a good idea but one that needs more than the basic ODBC that these folks are touting.

Big Data in 2012 - User Friendly

I was reading an interesting post by Harlan Smith at Hitachi about Big Data Analytics: Trends to Watch For in 2012.  He has a nice overview of big data in the enterprise and he also nicely discusses user-friendly tools for Big Data.  There are a lot of SQL-based tools for viewing, reporting, and analyzing data which are very user friendly.  If the upcoming crop of connectors for accessing big data enable us to bridge these user friendly SQL-based tools to things like Hadoop, Cassandra, and Mongo, then Big Data will become mainstream much faster than if we need to build a whole new set of tools.  The key is bridging between the SQL and NoSQL worlds.

Understanding the Oracle-Cloudera Big Data offering...

The big news of the day is likely Oracle's announcement of the specifics of their Big Data Appliance. As usual, TheReg has done a great job running down the specs. What I want to focus in on is the connectors that were announced today. The details are still somewhat thin but here's what I understand to be their offerings:

1.  “Oracle Loader for Hadoop, which moved data from Oracle 11g R2 databases to Hadoop data stores.”

Sounds like SQOOP to me.

2.  “Oracle Data Integrator for Hadoop, a twist on the existing Data Integrator tool that can automatically generate MapReduce code to chew on data and bring data sets into view of Oracle databases.”

This must be Cloudera’s ODBC driver that Cloudera announced late last year together with Tableau. I wonder how they are addressing the HiveQL/SQL gap.

3.  “Direct Connection for HDFS, and this essentially makes a section of the HDFS file system holding mapped and reduced data to be viewed as an Oracle database table.”

This sounds like a great idea. If this connector bypasses Thrift, it offers the potential for the highest performance.

4.  “R Connector for Hadoop. With this, Oracle has indeed taken the open source R statistical analysis package and added optimized math libraries to link it to the various data stores in the Big Data Appliance stack. This connector is not based on the Hadoop-friendly R tools from Revolution Analytics.”

 A specialized driver to address R.

 

Specifics aside, what looks clear is that integration/connectivity with "the establishment" is important. So what do you think and/or need? If you're trying to fit Hadoop into your landscape, drop me a line and let me know.

 

 

Microsoft Isotope CTP ultra-slick...

The Christmas and New Year season is finally passed. Or, at least, the turkey is out of the way. Whether one has gotten over the holiday season is another matter.

If anyone paid attention, Microsoft released a CTP of Isotope just before Christmas (see here). The easiest way to explain Isotope is that  it is "Hadoop for Windows".

I had some time to try it out and I was pleasantly surprised by how slick the package is. For one, the CTP is hosted on Azure so the pre-requisite for evaluating it is absolutely minimum. And the Metro look is refreshing.

The most useful bit is the inclusion of an ODBC driver and a VSTO pane that provides integration with Excel. The VSTO pane provided functionality equivalent to Microsoft Query since Microsoft Query doesn't currently run. (Of course, HiveQL isn't SQL so there will be limitations down the line.) But I was able to poke the ODBC driver using other tools. On the whole, this CTP is ultra-slick at showing Hadoop's power and demonstrates how Microsoft can harness its vast repertoire of technologies to provide a compelling offering.

If you haven't signed up yet, do.

 

Tableau connects to SAP BW powered by HANA using MDX query language

This past week, SAP and Tableau announced that you can use Tableau to visualize your SAP BW data.  This is great news and speaks to the openness of SAP BW.  Some people were curious how this works.  It is simple actually.  SAP BW has supported the MDX query language since it was launched in the late 1990's.  There are three ways to send an MDX query to SAP BW:

1. OLE DB for OLAP (ODBO)

2. XML for Analysis (XMLA)

3. OLAP BAPI

Whichever "transport" you use, you will hit the same MDX query processor on BW.  The MDX query language is the same standard used by SAP BW, Microsoft Analysis Services, Teradata OLAP, Oracle Database OLAP Option, Kognitio Pablo, etc.

So, it was easy to connect Tableau to SAP BW because Tableau issues MDX queries and SAP BW supports MDX queries.    Also note that Tableau will work with SAP BW on HANA and SAP BW that runs without HANA.  You can read more and see a demo here: Demo of Tableau running on SAP BW.

Of course, the next question is, Tableau now works on Microsoft Analysis Services and SAP BW.  When will Tableau officially support Teradata OLAP, Oracle Database OLAP Option, Kognitio Pablo, etc?

Using Hive from ODBC

Hadoop and Hive are definitely on everyone's mind these days. Whenever I see the words 'big data', connecting to Hadoop and Hive are not far. There is a latent demand for ODBC connectivity to Hive. To recap, nearly every top vendor has announced and/or released some form of integration with Hadoop & Hive:

• IBM: http://www-01.ibm.com/software/data/bigdata/enterprise.html
• Microsoft: http://www.microsoft.com/bigdata
• Oracle: http://www.oracle.com/us/technologies/big-data/index.html
• Pentaho: http://www.pentaho.com/hadoop/

There is of course Hive's own ODBC driver. But it seems that many people are looking in particular for a Windows driver.

Looking further, Hive's HiveQL is both a subset and extension of SQL-92. Supporting the full SQL-92 would be especially useful when integrating Hive ODBC into legacy apps.

So what are your challenges with integrating Hive into your solution and/or landscape? Drop me a line.

 

Is it that easy? "Seven Steps to Accounting Nirvana with HANA"

I was reading a blog post on SDN by Gregory Misiorek entitled "Seven Steps to Accounting Nirvana with HANA".  Gregory show how easy it is to pull ECC data into HANA in order to analyze using an Excel Pivot Table.  If you are considering HANA, this is worth a look.  Gregory's conclusions are:

"Conclusion or the main benefits of doing HANA reporting in Excel Pivot:

1. No need to create custom reports at any intermediate step listed above.
2. Use simple pivot tool which comes with each excel installation "for free".
3. Leverage column storage of HANA with ease of use of Excel to make millions of auditable records available to end user.
4. End users will never ask you to write another report after seeing this but it's up to you if this is a benefit to you and to your customers."

My interest in all of this of course is the underlying technology that makes this possible.  Excel communicates with a dimensional data source using the MDX query language.  If you are using Excel 2007 or Excel 2010, you need a data source that support the MDX 2005 query language.  Products like Essbase or Mondrian only support MDX 1999.  Products like SAP HANA, SAP BW, Microsoft Analysis Services, Teradata OLAP Connector, or Oracle Database OLAP support MDX 2005.  Since HANA supports the MDX 2005 query language as well as OLE DB for OLAP (ODBO), once the data is in HANA, it is just a matter of point and click.  In Gregory's article, steps 1-6 are about getting the data into HANA.  Step 7 is the point and click part of analyzing the data in HANA.

Yes, it is that easy!  I will be presenting a session on how to analyze your HANA data in Excel at the Administration and Infrastructure 2012 Conference which runs Feb 28 to Mar 2, 2012 in Las Vegas.  Come by and let me show you how easy it is to use Excel for your ad hoc data analysis and reporting!

Interesting Gartner Report on SAP HANA

SAP is betting the company on their next generation HANA technology and it is interesting to watch as HANA evolves.  Gartner analysts Massimo Pezzini and Daniel Sholler recently published a report entitled SAP Throws Down the Next-Generation Architecture Gauntlet With HANA, which is definitely worth a read.  Some of their key findings which I found interesting were:

"SAP application infrastructure and architecture strategy, dubbed HANA Architecture, combines cloud and in-memory technologies in a progressive, and potentially disruptive, vision."

"The HANA Architecture is a work in progress, and will undergo several significant changes before it is completed."

"SAP's vision will force megavendor competitors to respond by clearly and openly articulating their cloud and in-memory computing strategies."

One thing that I still find missing around the whole discussion of the in-memory approach of HANA is why it is possible.  While it is great to hold all the data in memory so as to be able to quickly access it, the key concept that is making this possible is the fact they store data by columns rather than by rows.  Storing data by columns is a concept that has been around for a long time but was never widely implemented because traditional databases were OLTP and hence were optimized for writing data.  Today's analytical databases (OLAP) need to be optimized for reading data and hence the column store makes sense.  Optimizing a database for read access and storing data by column allows you to significantly compress the data.  When the data is compressed, it becomes much easier to pull it into memory - basically, you don't need as much memory.  This allows you to actually have a much more affordable in-memory system or if you have a lot of money, to have an in-memory system that holds a large amount of data.  Once the data is in memory, then your analytical engine can run much faster and presto, you have a speed of thought analytics/OLAP system.

Companies like Teradata and Vertica are also following strongly with the column store.  While Vertica made a lot of noise about the column store and was acquired by HP, Teradata has not been saying as much.  I have heard from people at Teradata like Jeff Carter and Rich Charucki that they have been able to engineer the column store technology into the existing Teradata engine so the user can choose to store data either row-oriented or column-oriented depending on how they want to access it.  This is a great concept and I look forward to hearing more in the future.  In the meantime, read the Gartner report on SAP HANA and consider how this will cause the industry to change.

Making MapReduce easy to use

Interesting article at Forbes by Dan Woods where he talks to Tasso Argyros from Teradata Aster Data and discussed how to make MapReduce easier to use for business.  A major challenge in the Big Data world today is that the technology is very specialized and hence businesses need people with specific skillsets to utilize things like MapReduce and Hadoop.  Companies like Teradata Aster Data are trying to bridge this by making Big Data usable from technology that businesses already use like SQL.  You can read the full article here: Can MapReduce be Made Easy.

ODBC, Business Intelligence, and Big Data

The recent announcements by Microsoft about Big Data are very good news for those of us who work in the OLAP and BI space.  Basically, it looks like Microsoft is going to allow you to do OLAP and BI from Analysis Services on top of Hadoop and Hive.  A very glod blog post by Andrew Burst is worth a read.  "Microsoft will also make available an ODBC driver for Hive, the data warehousing front-end for Hadoop developed by FaceBook. What’s the big deal about an ODBC driver?  The combination of that driver and Hive will allow PowerPivot and SQL Server Analysis Services (in its new “Tabular mode”) to connect to Hadoop and query it freely.  And that, in turn, will allow any Analysis Services front end, including PowerView (until last week known by its “Crescent” code name), to perform enterprise-quality analysis and data visualization on Hadoop data.  Not only is that useful, it’s even a bit radical."  You can read the full post here: Putting the 'BI' in Big Data

Top 12 Players in Big Data Analytics

Doug Henschen at Information Week has posted his list of the top 12 players in Big Data Analytics:

1. EMC GreenPlum

2. Hadoop and MapReduce

3. HP Vertica

4. IBM Netezza

5. Infobright

6. Kognitio

7. Microsoft

8. Oracle

9. ParAccel

10. Sybase

11. Teradata and Aster Data

12. 1010data

This is an interesting list in what it includes and what is missing.  I am curious that although Sybase IQ is on the list, there is no mention of SAP HANA.  Since HANA seems to be THE thing at SAP, you would figure there should be mention of it.  SAP HANA is very interesting because you see the potential to combine your transactional and your analytical database into one.

Also of interest is the fact that Doug mentions OLAP for both Kognitio and Teradata.  Both of these companies have in-database OLAP engine that allows you to build cubes on the fly and do the analytics in the database.  Traditional OLAP where you pull the data out into a separate MOLAP engine like Essbase or Analysis Services seems to be going away because why build a cube on a mid-tier server when your "Big Data" database can do all the cubing virtually and let you do analysis on all your data faster than you can extract the data into the mid-tier cube?

The list includes Hadoop and MapReduce but does not mention companies like HortonWorks.  Last week was interesting when Microsoft announced a partnership with HortonWorks for Hadoop.  It was interesting to see the CEO of HortonWorks on stage at a Microsoft SQL Server conference.  This is a partnership to watch.

You can read Doug's full article here: Big Data Analytics Goes Big Time

Using Excel on SAP HANA... the inside scoop from SAP's Thomas Zurek

Thomas Zurek, VP for R&D of SAP BW and analytic parts of SAP HANA, just posted a blog about Using Excel on HANA.  This is important information on SAP HANA.  Basically, SAP HANA supports queries using the MDX language.  You can send MDX queries to HANA using the ODBO Provider.  HANA does not yet support XMLA, but it would be natural to expect this sometime soon.  Just like SAP BW has supported the MDX query language from day one, so does HANA.  What is key to note is that in his many presentations, SAP CTO Vishal Sikka has shown various applications connecting to HANA.  In all the hype about new applications on HANA, what has often been overlooked is that Excel can connect directly to HANA without requiring anything else.  What Thomas Zurek points out is that you can use your Excel Pivot Tables to directly analyze your data in HANA.  The power of this solution should not be overlooked when considering how easy it is to do these analytics.  From your Excel desktop, you get the full power of HANA to crunch your data.

Microsoft announces support for ODBC on Linux

Yes, you read the title correctly.  Microsoft is about to release a Linux ODBC driver for SQL Server.  This was announced yesterday at the SQL Server PASS conference.  For those of us who are ODBC fans (like me), this is great news.  With the news that Microsoft will deprecate the OLE DB provider for SQL Server and now with the news that we will get an official Microsoft Linux ODBC driver for SQL Server, it just re-inforces the fact that when Microsoft and Simba developed the ODBC specification 20 years ago, we got it right.  ODBC is the most prevalent database connectivity protocol out there and combined with SQL is a must have for any data source that wants to interoperate today.

There is a good blog post by Brian Swan from Microsoft that you can read here: Microsoft Announces SQL Server ODBC Driver for Linux!

ODBC is the gateway to Big Data

I am at the Microsoft SQL Server PASS conference this week and the big news today was Microsoft's announcement that they are supporting Hadoop.  So, how is Microsoft supporting Big Data?  Through ODBC of course.  Specifically, Microsoft will introduce an ODBC driver for Hive (the Hadoop query engine) on Windows.  This should be released next year.  It is interesting because other Simba customers including Teradata Aster Data use Simba's technology to provide ODBC drivers.  As much as Big Data and the Cloud are interesting, at the end of the day, you still need to connect applications and data sources and you need to connect to the desktop.  For 20 years, ODBC has been a standard of connectivity and in the world of Big Data and Cloud, ODBC continues to be the most widely used data connectivity interface.  You can read more about Microsoft's ODBC for Big Data.

ODBC is the gateway to the Cloud

It's interesting that it is SQL Azure that is pushing Microsoft to invest more in their ODBC driver and deprecate their OLE DB provider.  We are seeing the same thing in the ODBC SDK business.  We continue to partner with companies who have a Cloud-based or hosted solution and while these are great solutions, their customers still want to connect products like Microsoft Excel to these cloud data sources.  There are really two solutions to do this.  The first is to export the data from the cloud to some sort of delimited text file and then import this text file into Excel.  This is a very cumbersome solution.  The second is to build an ODBC driver that will talk to the web services interface of the cloud data source and allow Excel to connect directly and dynamically.  More and more of the cloud vendors are seeing the power of such an ODBC for cloud or ODBC for web services solution.  We just recently had a customer in Europe that build an ODBC driver using the C# programming language in a week using our SimbaEngine ODBC SDK.  When they showed the solution to their management, they were amazed at how easy it was to build and yet how functional the solution was.

Follow up from Teradata Partners

A good recap of Teradata's announcements at the recent Teradata Partners conference can be found in this article by Timothy Prickett Morgan:

- Hardware compression

- 2690 Appliance

- Columnar

- Unity

 

Steve Jobs--Architect of Big Data

The passing of Steve Jobs the day after the iPhone 4S launch under the symbolic guidance of his successor closes off Steve's life in a fitting and tidy manner. The signifcant shifts in industry in the past two decades--computing, publishing, media, mobile--have been touched or, more frequently, been shaped by what Steve has built.

Yesterday night as I was reading and watching various tributes to Jobs, I thought back to a TDWI talk by Donald Farmer earlier this year. (A version of his talk is available here. ) Donald gave a very insightful perspective of our future that is shaped by four forces: search, mobile, apps, and social media. When you stop and think about that, you'll note what many considers Jobs' opus magnum--his iPhone--is the embodiment and intersection of these four. Factoids: two-third of mobile searches on Google originates from an iOS device. Sixteen billion songs downloaded over the ten years that iTunes has reigned works out to about 51 songs downloaded per second. Ground-breaking price-points recreated and redefined the media, gaming and application market. Facebook, Twitter etc needed the vehicle of Jobs' iPhone to make the leap from college curiosity to ubiquity.

And Jobs' influence extends into Big Data. Whether you consider the actual data sources that were fed by iOS directly or indirectly (such as Twitter and Facebook), short product life cycles which challenge supply chains (which Apple's product exemplifies), or just heightened expectations for how to interact and work with machines that compute, Steve is part of this fabric too. Steve and Apple are the ever-present force that pushed industry forward in ways that no one expected.

 

Strata September Playlists

 So I had a few moments yesterday to pull together playlists for the recent Strata event. For whatever reason, Oreilly has not posted much material on the actual Strata Conference (Sept 22 & 23). But the JumpStart (Sept 19) and Summit (20 & 21) are nearly completely online. These are sessions I attended that I felt are eye-opening, new ideas or just well-articulated points of view. Let me know what you think.

Strata JumpStart playlist

Strata Summit playlist

Strata Conference playlist

 

Data Science and Visualization at Strata...

 Data Science

So the second Strata conference has came and gone. (Actually, the event consists of three sub-conferences: JumpStart, Summit and Conference; but I digress...) The general tone of the event is excitment. The presenters--consisting of vendors and practitioneers--were understandably excited and proud about the self-named field of "Data Science". Recent intersecting trends have brought us to our current point in time. Commoditization of computing power, proliferation of sensors, and low cost manufacturing together are ushering in the age of Big Data. It is possible to record just about any data stream that you can imagine and social networks is the obvious prominent driver of much of these development. Oreilly did a great job in bringing together a varied group. In particular, I found Monica Rogati's session on her experience with LinkedIn's exploding user base and TJ Patil's session on Data Science particularly engrossing for their mix of narrative, technology, and war stories. Unfortunatley, only TJ's session is available on Oreilly's channel at YouTube (here).

 Visualization

I stopped by the Tableau booth to check out the capabilities of the free public version of Tableau (available at tableaupublic.com). It is limited to sourcing data from a textfile. I enquired casually about the possibility of connecting to NoSQL databases. Currently, there is no connectivity but there is apparently something in the works. In speaking with DataStax's Ben Coverston, he points out that Cassandra CQL is accessible via JDBC. Cassandra 1.1 is coming out RSN. I'm planning to look into how to hook up Tableau to Cassandra directly. Drop me a line if you're interested in this.

 

OLAP sessions at Teradata Partners conference

I will be at the Teradata Partners User Conference next week in San Diego and wanted to point out a couple of session that are worth attending if you are interested in OLAP analysis on your Teradata data.

The first session is: Seamless Access from Excel PivotTables to Teradata by Sam Tawfik.  The abstract for Sam's session is:

"Teradata OLAP Connector makes it easy for Excel PivotTables to access Teradata. This tool makes it easy for Excel users to navigate the dimensional data stored in Teradata without having to extract subsets of the data.

Extracting data for analysis creates security challenges, inconsistent view of the data, and delay in access to current business data.

This session also includes an Excel to Teradata demo."

The second session is: Introduction to Teradata Business Intelligence Optimizer by Shannon Terry who is Director of Customer Analytics at Nationwide.  The abstract for Shannon's session is:

"This presentation will show the practical application of the Teradata Business Intelligence Optimizer (TBIO) suite of tools in addressing a common problem in organizations: spreadsheet proliferation. Spreadsheets are powerful tools, but they can become dangerous when data is taken out of the data warehouse and improperly handled or manipulated. The TBIO tool suite allows you to embrace the reality of spreadsheet-based BI in your organization while creating a safe and structured environment in which spreadsheet users can leverage value in your warehouse."

Sam's session looks to be a presentation about Teradata OLAP and how you can connect from Excel.  Shannon's presentation is a customer presentation on how they are using Teradata OLAP with Excel.

Also, if you are at Partner, make sure you visit the arcplan booth on the exhibit floor.  The folks from arcplan will be demonstrating how their product allows you to do analysis on your Teradata OLAP data.  Lots to see.

Going to Oracle OpenWorld? Use Oracle OLAP?

Simba Technologies and Vlamis Software Solutions are throwing a special OpenWorld 2011 reception for current and prospective Oracle OLAP users.

Time:     Tuesday, October 4th, 2011

                  5pm – 7pm

Location: San Francisco (walking distance from Moscone Center)

                    Please RSVP for location.

To request an invitation, email events(at)simba.com.  Be sure to tell us which company you’re with and how you heard about the event.  We look forward to seeing you there!

Teradata announcement on Teradata Columnar

Here is Teradata's official announcement of Teradata Columnar.  Definitely worth reading.  I will be at the Teradata Partners conference the first week of October and will be looking to learn more there. http://www.teradata.com/News-Releases/2011/Teradata-Establishes-New-Standard-for-Columnar-Databases/ 

Strata 2011

Some of you may know I've been working on a few ideas in the last while. Of late, I've been talking to some of you and attending conferences/webinars. I'm in New York for O'Reilly's Strata this week. Apart from a couple of misdirections, the day has turned out rather well. Day 1 was the so-called Jumpstart day before the Summit and Conference. By that, the organizer meant for day 1 to provide a preamble to the topic at hand: Big Data. I had rather low expectations for day 1 mainly because I couldn't see how much any one speaker could fit into 35 minutes. After all, we managed to fit 14 sessions, two coffee breaks and one lunch into one nine hour day.

But, like a good hors d'oeurve, the day has whetted my appetize for what's to come in the remainder of the week. The wide ranging topics spanned people (e.g., recruiting data scientist and Google's Project Oxygen), methodology (Simon Wardley's "disturbance in the flow", Jennifer Zeszut's agile marketing), process (Michael Hugos on Starbucks' supply chain) and policy (Michael Nelson on organizational transparency). What stood out was that nearly all speakers were presenting their hypothesis with sound data and/or analysis. (This really shouldn't surprise since the conference is about using and working with data.)

It should be a good week.

 

My adventures at SAP TechEd and Sybase TechWave

I was only briefly at SAP TechEd/Sybase TechWave this week.

The biggest thing for me was showing a demo of an MDX Provider for Sybase IQ.  We have worked jointly with Sybase to build this Proof of Concept and we got some good interest from Sybase partners and customers.  If you are interested in getting more information, please follow up with me or Andrew Neugebauer at Sybase.

Of course, the biggest thing at TechEd again was SAP HANA.  Vishal Sikka, SAP's CTO had a keynote where he unveiled two new applications for HANA - SAP Smart Meter Analytics and SAP COPA Accelerator.  You can read more here.  SAP HANA is very interesting because it "delivers single database with native support for row and columnar data stores; a data calculation engine; SQL and MDX interfaces and a unified information modeling design environment."

Vectorsoft ODBC Driver Performance Optimization

Vectorsoft has been a long-time Simba customer. They have an interesting product called CONZEPT 16 - an integrated database development environment and system. They have just released a new version (5.6) of their ODBC driver based on the SimbaEngine SDK 8 for CONZEPT 16. Not only are they getting the immediate benefits of the support for ODBC 3.52, Unicode and 64-bit,  but they are also taking advantage of improved performance and the advanced performance features in the new generation of SimbaEngine. After implementing pass-down of filters and joins, they were excited to share a blog with us (http://blog.conzept16.de/2011/08/ODBC-Treiber-Performanceoptimierung, note in German). According to Robert’s testing, the new driver has vastly outperformed the old driver which is based on SimbaEngine SDK 7. The performance boost is especially dramatic (orders of magnitude) for queries with filters on non-indexed columns and with joins. This totally makes sense since operations (or functions) are “shipped” closer to the data store rather than the data. The operations are not limited to the above mentioned, they can also be aggregations, “select from” queries and etc. What’s powerful about this feature is it allows the driver specific glue code (DSII) to dynamically select operations to pass down and automatically leave the rest to the SimbaEngine – a truly collaboratively approach to SQL processing.