Connect – a Simba Technologies Blog

How do I connect my data to Tableau? - With an ODBC driver of course!

Working in the database technology business, we run into a lot of software companies who have some sort of data or database engine. A lot of these software companies are very focused on building the next generation of powerful database engine. They often come to Simba asking how they can connect with front end visualization and analytics products like Microsoft Excel, SAP BusinessObjects Crystal Reports, SAP Lumira (formerly Visual Intelligence), MicroStrategy, and of course Tableau.  Tableau is today one of the hottest companies in data visualization.  All of the Big Data vendors want to demonstrate that Tableau connects to their unique data engine.  So how do you connect your data engine to products like Tableau?  The answer is simple, build an ODBC driver.  Of course, building an ODBC driver is not simple.  Although the ODBC standard has been around for many years and is relatively easy to follow, to be able to properly connect with all these wonderful products, you must understand data connectivity.  This is what Simba does.  Our SimbaEngine ODBC SDK is proven - we have been doing this for more than 2 decades.  SimbaEngine ODBC SDK is recognized by companies like Tableau as a premier solution that will work.  So, if you build your ODBC driver using the SimbaEngine ODBC SDK, you know that Tableau will work with your data engine.  In fact, you know that Excel, SAP, Tableau, etc will *ALL* work with your data engine.

SAP HANA and Intel Hadoop - a powerful enterprise combination for Big Data

Last week at the SAP Sapphire conference, I was able to get some more insight into what SAP is doing around Big Data. They have been closely working with Intel to integrate Intel's Hadoop distribution with SAP HANA. If you are an SAP enterprise customer - and there are a lot of SAP enterprise customers - this could be a very powerful Big Data story going forward.

Basically, what SAP is doing is allowing you to use HANA as the hub for all your data. You can then use SAP Data Services to ETL your Hadoop data into HANA. Even more powerful, is the fact you can leave your data in Hadoop and use HANA to front it all and so have the full power of the SAP BI Suite against all your data whether it is in HANA or in Hadoop. Since HANA fronts all the data, the SAP BI Suite only needs to integrate with HANA. When HANA gets a query for data that exists in your Hadoop cluster, it will directly issue a query to Intel's Hive ODBC driver which will then push the query into the Hadoop cluster.  All this is done from the SAP HANA Studio.  This looks similar to what Microsoft does with Polybase but seems more geared towards the enterprise.

You can read more here:

1. Intel press release about SAP Sapphire and HANA

2. SAP Demo of HANA Hadoop integration

Platform agnostic OLAP reporting from SAP BusinessObjects

Simba has been a leader in OLAP and MDX data connectivity for the last decade and a half. Most of the OLAP providers out there that support the MDX query language have been built using the SimbaProvider SDK - these include SAP BW, SAP HANA, Oracle, Teradata, Kognitio, and Exasol.

One question that often comes up is what BI or analytics tool can I use that will support multiple OLAP data sources?  First up is always Microsoft Excel Pivot Tables.  Many analysts know how to use Excel Pivot Tables and if the OLAP data source supports the OLE DB for OLAP (ODBO) protocol and the MDX query language, then this is a no-brainer.

Since all of the major BI vendors have been bought up by larger companies, it has always been a challenge to find a serious BI tool that support lots of different OLAP data sources. Most recently, Tableau is an up and comer but they are still limited in their OLAP support largely to Microsoft, SAP, and Teradata.  Last week at the SAP Sapphire conference, I attended a session entitled "What's new with Analysis Office and OLAP". This session covered the new features in the SAP BusinessObjects Analysis for Office and Analysis for OLAP products.  There were two specific slides regarding OLAP connectivity here. One slide covered all the different OLAP data sources that SAP supports and another slide showed that you could have simultaneous connection to multiple data providers. Also, unlike Excel, the SAP BusinessObjects Analysis product is part of a complete BI Suite and so is that much more powerful in any business enterprise.  The different OLAP back ends that SAP supports are:

- SAP NetWeaver BW

- SAP HANA

- Microsoft SSAS

- Teradata

- Oracle OLAP and Oracle Essbase

- SAP EPM providers: SAP BusinessObjects Planning and Consolidation, SAP BusinessObjects Profitability and Cost Management, SAP BusinessObjects Financial Consolidation

BigQuery at Google I/O 2013...

Google IO has came and gone. It was a drink-from-the-fountain experience for this IO newbie. But apparently, day one's 3.5h keynote was similarly challenging for the seasoned. (Both Engadget and TheRegister have great live-blog of the keynote if you want to relive the marathon at your own pace. Engadget's even included pictures!)

BigQuery fits into IO's "Google Cloud Platform" track.

As usual, everything from the 2013 IO is available via the Google Developer channel.

For your convenience, here is my #IO13 playlist for the useful BigQuery-related sessions I found. I may have missed a session given the amount of material presented so do let me know if you saw something worth compiling here.

1. Cloud Platform Track Kickoff - Good way to understand how BigQuery fits into Google's Cloud portfolio.
2. Importing Large Data Sets into Google Cloud Storage - A dive into gsutil.
3. Keys to the Kingdom: Design Patterns for Using OAuth in the Cloud - How to use OAuth for BigQuery and other Google services.
4. Big Data Mashups: Enabling Next Generation Analytics Using BigQuery - Shutterfly shares of their experience in using BigQuery to drive their analytics.
5. Google Analytics and AdSense Data Analysis in BigQuery - A preview of what's to come for AdSense and Analytics Premium customers.
6. Here Be BigQuery: Building Social Gaming Infrastructure on the Google Cloud Platform - A second case study by Gamesys on their success in utilizing BigQuery for social game analytics.
7. All the Ships in the World: Visualizing Data with Google Cloud and Maps - 
8. Behind the Data Sensing Lab: Gathering, Processing, and Analyzing Data at Scale using the Google Cloud Platform - Google took the idea of gathering sensor data from Strata and bumped up the number of sensors. Great example of how to work with large stream.

 

Is Oracle losing database customers to DataStax and Cassandra

An interesting post by Alex Williams at techcrunch from this past March about "Oracle Is Bleeding At The Hands Of Database Rivals" is worth a quick read.

Most of the new Big Data vendors like DataStax use ODBC drivers from Simba.  The reason they use our drivers is that besides being fully standards compliant (actually, Simba co-developed the ODBC specification with Microsoft back in the day), Simba's ODBC drivers have our SQL Connector technology which allows us to bridge between SQL and things like HiveQL.  Therefore, if you are using a product like Tableau, SAP Lumira (fka Visual Intelligence), Excel, or Crystal Reports, everything just works.

So, why is Alex's article of interest?  Well, we have so many partners in the Big Data space, that I am always curious how they are doing against the traditional database vendors like Oracle and Teradata.  Alex's article helps give some insight into how seriously enterprises are looking at the new Big Data vendors like DataStax, Cloudera, Google, HortonWorks, MapR, Splunk, etc.

Teradata Intelligent Memory - in-memory performance at a fraction of the cost

Yesterday, Teradata announced Teradata Intelligent Memory - part of the Teradata Unified Data Architecture. I previously wrote that Teradata found the following when studying data usage:

1. 43% of I/O uses 1% of cyclinders

2. 85% of I/O uses 10% of cyclinders

3. 94% of I/O uses 20% of cyclinders

Teradata has now extended their analysis and their technology to determine what data should be in-memory and so they are optimizing the memory of modern database systems. Therefore, unlike others who want to put everything in-memory, Teradata is using smart algorithms to determine what should be in-memory and hence doing the price/performance trade-off with a kick - significantly lower price because not everything has to be in-memory but almost similar performance. This is big news because as data volumes grow, cost becomes an issue and Teradata is able to keep costs down while maximizing performance.  You can read more about it here: Teradata Intelligent Memory.

SQL drives up Hadoop usability

Darryl Taft at eWeek wrote a very good article Hadoop Drives Down Costs, Drives Up Usability With SQL Convergence. The gist of the article is that Hadoop has a lot of potential and adding SQL capability allows people to use Hadoop with today's mainstream BI and reporting tools. After all, if your data source supports SQL, even products like Excel can connect directly. While adding SQL to Hadoop is very important, the requirement for ODBC or JDBC should also not be forgotten. SQL is the query language. ODBC and JDBC are the bridges that take the SQL from the BI tool to Hadoop and bring the results back. Not all ODBC and JDBC drivers are created equal. You need good drivers that support all BI tools and have a long track record of success. Most of the major Hadoop vendors deliver ODBC drivers that come from Simba.  So, when you use a Hadoop distribution from DataStax, HortonWorks, Intel, or MapR, you know you are getting a good ODBC quality driver that will work well with products like Excel, SAP BusinessObjects or Tableau.  Simba's ODBC drivers support all major Hadoop distributions including Apache and Cloudera so if you want a driver that connects to any Hadoop distribution, Simba is definitely the way to go.

Google IO 2013

Google IO 2013 is less than two weeks away now and there's a lot of excitement as usual about what's just about to pop out from the Chocolate Factory.

This will be the first IO I'm attending live. Beneath the froth of the devices both mainstream and otherwise (glasses!), there is a track for Google's cloud which include Google Compute Engine and Google BigQuery. There's also G+ community which has been helpful for orienting yourself to this mega-event .

There is a handful of sessions on BigQuery which I'll report back on. Watch for it.

 

April Hive User Group meeting

The Hive User Group held its semi-regular meetings last week at Hortonwork's office.

This was a jam-packed evening as the Stinger initiative has started to deliver some of its firstfruits. Gunther's session on Hive 0.11 was the longest talk of the night and generated some of the liveliest discussion in the room.

A Webex recording of this meeting has been made available by Hortonworks. I offer the time marks as a public service for anyone wanting to review this :

1. +0:04:41: "Merge of HCatalog and Hive" by Alan Gate
2. +0:13:30: "The Hive Security Situation" by Carl Steinbach
   Carl proposed a generalized AccessServer as part of HiveServer2's future.
   
    
3. +0:47:15: "New cool stuff in Hive 0.11" by Gunther Hagleitner
   Lotsa details on what's just around the corner in 0.11. The vectorization (or arguably less confusing columnization) feature is showing great promise.
   
   
    
4. +1:33:30: "Optimizing Hive Queries" by Owen O'Malley
   Owen shared a lot of the internals of the ORC file format including the great compression.
   
   
    
5. +2:10:28: "The Hive Marketplace" by George Chow
   This was open source collaboration in action. I shared our performance benchmarks of HiveServer2 from late last year. I had previously shared this privately with a few individuals including Carl. Carl was gracious to step up and explain his findings--that this was possibly due to a poor implementation of union in Thrift. (Update: refer to JIRA 3746.) I think we're enroute to a solution real soon now.
   
   

 

SQL for the Hadoop NoSQL world - Cloudera Impala is now GA

Now that Cloudera's Impala is GA, I expect we will see the SQL battle heat up in the Hadoop/NoSQL world. GigaOM did an interview with Cloudera CEO Mike Olson where they compare what they have against the other major Hadoop distros - HortonWorks, MapR, and EMC.  While I think it is still early days, this is worth a read to get an idea of where things are going... http://gigaom.com/2013/04/30/with-impala-now-ga-clouderas-ceo-sizes-up-the-sql-on-hadoop-market/

Some reasons to choose MDX for analytic solutions

With the increasing use of self-service BI tools, data exploration tools, and data discovery tools, end users have more options to analyze data coming directly from any data source. The other day I was asked why people should choose MDX (and constructing a cube) for analysis when there are so many other options. Here are some scenarios where MDX is advantageous, especially in Excel but also in other tools. This is a good starting point for evaluating MDX as an analytic solution.

Capability	MDX (ODBO/XMLA)	ODBC/JDBC (usually SQL)
Large data volumes	Data stays on the server and the server calculates the summary values.	In Excel, the data is imported and stored in the workbook. Excel does the summary calculations, at performance expense.
Ease of use for BI client users	Point and click interface to connect to model, get summary values, slice and dice data.	End user must know SQL very well to join tables and perform anything other than basic summary analysis.
Reusable data model	One data model can be built  and used with many different BI clients (Excel, Tableau, SAP BusinessObjects, Cognos, Microstrategy, etc). MDX calculations in the model can be reused.	The calculation logic must be repeated in every BI client connecting to the server.
Analytic capabilities	MDX language has shortcuts for common analytic capabilities such as time-based calculations. Top 10 and other sets can be shown automatically. Can drill to details after viewing summary information.	Optimized for table and row operations, not for analytics. Calculations that are easy in MDX can be hard in SQL.

This is data science

Hadoop is often taken to be synonmous with Big Data, the solution to an organization's decision-making ills, the cure for cancer, the caramilk secret and more. But this is, of course, not the case. Hadoop merely provides a foundation for working with much of the data that was previously unreachable or unuseable.

Similarly, the term data science is much bandied about. 

So I attended the inaugural talk at Data Science Vancouver by Don Turnbull about this new discipline data science. The talk wasn't recorded but you can follow Don's exposition thru his slides. My 5-second summary is that data science is essentially a renaissance of science that pairs new techniques with (dare I say it) old-school scientific method. As Don puts it, it is science afterall.

Don concluded with a great reading list for someone either getting into or wanting/needing to deepen their understanding of this field. For your convenience, I've reassemble the list here with Amazon links. (I didn't know that Issac Asimov's Foundation is not available in ebook format. The other books in the series are... just not the first. Weird)

Fiction

1. Foundation, by Isaac Asimov 

Statistics

1. The Control Revolution, by James R. Beniger 
2. Against the Gods: The Remarkable Story of Risk, by Peter L. Bernstein 
3. Data Mining: Practical Machine Learning Tools and Techniques, by Ian H. Witten and Eibe Frank (and Mark Hall?) 
4. The Lady Tasting Tea: How Statistics Revolutionized Science in the Twentieth Century, by David Salsburg 
5. The Laws of the Web: Patterns in the Ecology of Information, by Bernardo A. Huberman 
6. The Rise of Statistical Thinking, 1820-1900, by Theodore M. Porter 
7. When Information Came of Age: Technologies of Knowledge in the Age of Reason and Revolution, 1700-1850, by Daniel R. Headrick 
8. Men of Mathematics, by E. T. Bell 

Knowledge Discovery and Data Mining (KDD)

1. Advances in Knowledge Discovery and Data Mining, edited by Usama M. Fayyad, Gregory Piatetsky-Shapiro, Padhraic Smyth and Ramasamy Uthurusamy 
2. Machine Learning, by Tom M. Mitchell 
3. Readings in Information Visualization: Using Vision to Think, by Stuart K. Card, Jock D. Mackinlay and Ben Schneiderman 

 

 

What Would Google Do, part N...

WWGD. "What Would Google Do?" I've seen this refrain enough in the last few weeks but I didn't know that there's a book by that name. Fittingly, a couple of clicks in Google leads you to a nice summary of the book.

But today, I want to talk about Dremel and the post-MapReduce world. To put things in perspective, the following diagram (full disclosure: I grabbed it from Doug Cutting's October Strata NYC keynote) shows the lineage of some of today's Big Data technologies.

Everyone's likely familar with the first three rows. But Dremel, F1 and Spanner are new on the scene.

Let's consider Dremel first. (Refer here for Google's whitepaper.) Dremel is used extensively within Google. What is notable is that Google decided to commercialize it for external use via its BigQuery service. In typical Google fashion, it has been steady firming up for last couple of years. Google just announced several important new features (including big JOIN) for BigQuery last week. Previously, JOIN were only supported if all but one of the tables were "small" (small being at composed 8MB compresses). The new big JOIN feature lifts this restriction. Concurrently, Tableau is just on the verge of shipping support for BigQuery in it upcoming 8.0 release. And just in time, Tableau 8.0 which includes BigQuery as a first-class datasource just shipped. BigQuery is hitting it in strides. 

So what is the open source community's response to Dremel? In the diagram above, Impala is highlighted in red because the diagram is from Cloudera who led the Impala effort. But if you casted your eyes wider, there's another project--Apache Drill--that purports to carry the flame of Google Dremel:

In fact, there's more... there's also OpenDremel. The good news is that OpenDremel and Drill decided to join force a little while back. So now there's only one project--Drill--which should simplify things for the future.

Drill is making progress. In fact, there was a design review just last week of its SQL parser. If you want to a summary of Drill's vision and architecture, Ted Dunning has been touring and making the case. The exciting point I see for Drill is that it anticipates many areas of extensibility including query language and file formats.

One vision that I foresee is implementing a language that is well established for analytics such as MDX on Drill. To give you a flavour for what this could look like, Simba has implemented an MDX/SQL translation on top of Impala to demonstrate the viability and performance of such a system. In the case of Drill, we can potentially do one better with tighter integration and, hopefully, better performance and functionality.

If you're sharp, you'll note that there two other Google projects--F1 and Spanner--that as yet have no open source equivalence. Based on material I've seen to date (including an excellent presentation in February at Strata Santa Clara), the hallmark of Spanner and F1 is Paxos/TrueTime--a radical new algorithm for achieving consistency using atomic clocks. Despite what Cloudera's slide above show, what I have seen of Impala to date is that it doesn't contain anything along that line. It's safe to say that F1 and Spanner is another step in the evolution of the database that Google has up its sleeve.

 

Update on what HortonWorks and Microsoft are doing with Hadoop on Windows

Microsoft and HortonWorks have announced Hadoop on Windows and Mark Smith at Ventana Research wrote a nice blog post giving an overview of Hadoop on Windows.  Some key points:

1. Microsoft HDInsight Server and Windows Azure HDInsight Service are built on Hortonworks Hadoop Data Platform and make Hadoop readily available on Microsoft Windows. This strategic alliance helps IT organizations bring the power of Hadoop to Microsoft platforms. The Beta is now available.

2. Part of Hortonworks approach is that new advancements are contributed to the open source community where other developers and organization can contribute and help or use it once finalized and ready for distribution. This is a much different approach than others in the market who source Hadoop from the community and make proprietary extensions to it or embed it in their software and sell the license to the customer.

3. Hortonworks worked with Simba to provide a Hive ODBC connector to support SQL-92 access from business intelligence tools.  Microsoft Excel can easily direct access to Hadoop through SQL which opens up further support for a large number of organizations.

 

Oracle OLAP webinar series videos available

On February 5 we delivered our most popular webinar series to date: Store once, read anywhere - using Oracle OLAP data in multiple BI clients. The videos are now available here:

• SAP BusinessObjects Analysis, edition for OLAP with Simba XMLA for Oracle
• IBM Cognos with Simba XMLA for Oracle
• Microsoft Excel with Simba XMLA for Oracle

One of the highlights of the webinar series is when Ed Seibold at Rovi Corporation talks about his experience with Simba’s MDX Provider for Oracle OLAP product and how it has changed the way Rovi works in BI. Launch the Excel video and scroll to about 12:30 to view his segment of the broadcast. Also, Deepa Sankar from SAP joined us to talk about SAP’s investment in vendor agnostic, platform-independent BI solution. Launch the SAP video and scroll to about 14:30 to view Deepa’s segment.

I have to thank Dan Vlamis and the team at Vlamis Software Solutions for helping us deliver such great content.

Day 2, Strata Santa Clara.

Day two of Strata Santa Clara turned out to be a day of sober second thought. Actually, there was sign of that in day 1 with Tim’s session on “why [we] can’t escape SQL”. But Kate Crawford’s keynote in the morning “Algorithmic Illusions: Hidden Biases of Big Data” is another articulate counterpoint to the boundless optimism surrounding Big Data. I am not a curmudgeon and I indeed applaud Oreilly’s bravery to consider and run these sessions. When I see that there is finally some dissenting voice about the state of Hadoop, then truly one can see that Hadoop has grown up.

The highlight for me has to be Google’s presentation on F1. I see F1 as a reasoned response to the BigTable-inspired technologies that are in bloom today. F1 is a *gasp* SQL RDBMS with ACID properties which nonetheless scales globally (literally). Even better, the slides are up already so have a look to see the future.

I provided a small scale defense of why SQL and query language matters at Intel’s booth in the mid-afternoon. I’ll share that with you once Intel post-production is complete. To summarize, the trends that I outlined yesterday regarding SQL hints at the future. Both Drill and Stinger/Tez have allowances for new query languages beyond SQL (MDX?). I believe this trend for extensibility shows a necessary maturing of the technology. Substituting query languages with APIs works in limiting cases; for integration and continuity with existing tools and constituencies, a query language—at least SQL—is necessary.

Day 1, Strata Santa Clara.

 

Day 1 of Strata Santa Clara continued the frentic pace of day 0. In light of Stinger and Hawq, I was particularly keen to see the SQLization trend play out. The day started with Rajat Taneja, CTO of EA, sharing how Hadoop and Big Data has changed EA approaches their entire portfolio of games. He summarized it as follows:

1. Data is Only a means to an End
2. Focus on the Usage and Action of the Data First
3. The Right Data at the Right Time is More Valuable than Bigger Data Sets
4. Use Data to Create Amazing Customer Experiences
5. Single TRUE identity is the crucial glue that connects the power of data insight to the CONSUMER.

Right after, EMC’s Scott Yara stepped up but not to crow about Monday’s Pivotal HD but rather to recognize influential “data people”. But it was the third keynote that I thought was bold and ambitious: Stitch Fix pairs machine algorithm with human curation to deliver (literally, shipped to a customer/subscriber) apparel recommendations. The theme for the keynote is that people matter.

I began the day with Berkeley’s Spark/Shark. I had first seen at last year’s Hadoop Summit. Judging from the attendance which was the highest that I saw for the day, the demand and interest for interactive/low-latency SQL is huge. 

I followed up with Tomer Shiran’s introduction to Apache Drill. Tomer and Jacques and the team (now comprising 6-7 companies) are making progress and are looking at a Q2 alpha and Q3 beta. Drill is of great interest to me because of its extensible front and back-end architecture that allows for additional query languages and operators. The questions from the audience suggest that SQL on a scaled-out architecture is definitely on top of mind.

EMC’s session on their Pivotal HD distribution, right after lunch, was no snoozer. The benchmarks number that they shared between Hawq & Hive and Hawq & Impala are amazing. In the former, accelerations of 19X to 648X. In the latter, accelerations was still from 9X to 69X. Of course, this isn’t too surprising given that they are leveraging Greenplum’s well-developed engine and optimizer. Hawq is a game-changer for the “SQL on HDFS” market because of EMC’s position in the storage market and the strength of Hawq.

Hortonwork’s Alan Gates followed with a survey of the various tools in the Hadoop ecosystem. He touched lightly on the new Tez project (pronounced “taze”). But, more importantly, I also got some longstanding misunderstanding about HBase and HCatalog cleared up. Thanks Alan!

Finally, it was the session that I had been waiting all day for: Tim O’Brien’s “The Future of Relational (or Why You Can’t Escape SQL)”. Tim did a breathless re-cap of the last 40 or so years of database technology. Starting with the pre-SQL world, moving onto the early SQL market, the mature SQL market, the rise of NoSQL in the mid-2000s, to today’s diverse Big Data technologies and to what I’ll offered as post-BigData NewSQL world of Impala/Drill/Spanner. From his experience as a database developer, he offers several choices statements about the need/motivation for many a team who choose NoSQL. (You can refer to theregister’s article for those…) But the core point that he offers is that SQL and relational are not joined at the hip. Relational for data modeling may be out. But SQL remains applicable and valuable: expressiveness, a corp of practitioners with a vast body of knowledge, and widely available and deployed tools. A final curious reason he offered is suggestively summarized as “What Would Google Do”: Google itself has come full-circle back to SQL with Spanner after inventing BigTable/MapReduce.

I ended the day with Citus Data’s Carl Steinbach. Citus took Postgres and tacked on an implementation of SQL/MED that they have prosaically named “foreign data wrapper”. Carl, being a Hive guy, described a foreign data wrapper as a Hive SerDe with extensions for predicate push-down and metrics. Works for me. In the end, it delivers huge improvement even on non-HDFS data.

 Summarizing day 1, the themes surrounding SQL are:

1. A proven and understood query language—SQL—is not an option: it is table stakes for Big Data to enter any enterprise.
2. But the query language must span all workloads. It must not only provide for bulk/ETL scenario where latency is immaterial but also interactive workload where latency is king.
3. SQL-like isn’t good enough. SQL compliance is no longer an ideal but necessary: to enable the full gamut of tools to run unmodified; to enable analysts to write unrestricted SQL.

Day 0, Strata Santa Clara.

The run-up to any conference can be rather feverish. This year's Strata is turing out to be one such.

Hortonworks finally announced their Stinger initiative last Friday (after Sanjay had leaked it earlier at a TheHive event). As I heard it from Alan Gates, it is a program of enhancements to bring Hive's query performance into "human use" realm. 

Then, just before my flight down to San Francisco, EMC Greenplum announced their Pivotal HD ("Hawq") distribution. As usual, El Reg has a good summary of the details while Gigaom has some backstory. In short, the SQLization of NoSQL/BigData is real. The trend that Cloudera very publically kicked off at last October's Strata with Impala is now in full bloom.

Running down the list of vendors, just about everyone now has something in the burner now:

Contributor/Author	Technology	Note
Google	F1	"Spanner". SQL. Transactions.
Cloudera	Impala	HiveQL; will be open sourced. Currently in beta release as part of CDH4; GA expected in CQ1.
MapR	Drill	SQL-2003; in progress; in development as an Apache incubator project.
Hortonworks	Stinger initiative	Program of enhancements to existing projects (including Hive) and also proposes two new projects: Tez and Knox. Expected in March.
EMC Greenplum	Pivotal HD “Hawq”	SQL-20XX. Source status unclear currently.
Salesforce.com	Phoenix	Contribution to Apache. No defined release vehicle yet.

Synchronicity-ly, there's a session Wednesday afternoon by Tom O'Brien aptly titled "The Future of Relational (or Why You Can't Escape SQL)". I'll be checking that out to take pulse on this trend.

On a broader note, Intel has just threw their hat into the Hadoop distro ring by announcing their IDH. (You can read more about the extensive Asian connections here.) I'll be giving a tech talk Thursday 2PM at Intel's booth (#101) about Hadoop's query language scene. Stop by to say hi if you are interested in this fast moving  scene.

But onto day 0 events: the tutorials. Jonathan Hsieh did a great job with his morning tutorial on HBase for the app developers. He gave great examples and tips for one pondering the question of "Whither HBase". With O'reilly's move to putting all real content behind the paywall, the session won't be readily available but Jonathan has promised to post his slides. I'll post the link when that shows up.

For those of us who enjoyed Ryan Boyd & Michael Manoochehri's shows on the Google Developers channel, the dynamic duo joined up with Julia Ferraioli to present a tutorial to takes one thru the end-to-end process of collecting, persisting, processing, and visualizing data using collective might of Google App Engine, Compute Engine, BigQuery. Aside from the rough edges of getting all the relevant tools downloaded, installed and configured, the process went smoother than expected. Having worked with BigQuery previously, I'm familiar with that piece of the picture. I still have to finish the assignment (thanks Julia for keeping the group up!) due to a Cygwin problem during class but it brings out the kid in me to see everything run from soup to nuts.

And that's all for day 0! The formal Strata hasn't even started. 

 

Hadoop query language trend

In talking with analysts and partners in the past week leading up to this week's Strata, it's safe to say that one trned is clearly in focus: the utility and urgency for a query language for Hadoop. Rick van der Lans and Gigaom have done an excellent job surveying the happenings here so I'll refer you to their posts (Rick's "The SQL-fication of NoSQL Continues" and Gigaom).

Simba's been also working on this query language question. If you look down the page, you'll note that Cathy has a writeup about our "MDX on Impala"; drop us a line to let us know what you think.

This work with Impala is inspired by what we've been doing for some of our customers. But others have been thinking about this query language situation too.

Notably, the Apache Drill project is looking to provide a pluggable query language front-end. It's possible to imagine Drill with direct support for MDX. That is, no translation from MDX to SQL; just direct execution plans built from MDX statements.

Strata is going to be awesome this year with the market coming on full steam behind SQL. I'll be reporting on Strata as it plays out.

It's gonna be exciting...

 

Demo: Microsoft Excel PivotTables on Cloudera Impala via Simba MDX Provider

Last Thursday we sent the Impala user group a demonstration of some Big Data analytic technology in early development here at Simba. You can watch the video here: Microsoft Excel PivotTables on Cloudera Impala via Simba MDX Provider. In this video, I give a short demonstration of how to use core PivotTable functionality in Excel 2010. The video shows how to establish connectivity, build a pivot, sort, format, and slice.

What the video doesn’t show or describe is the technical implementation under the covers. Conceptually, this implementation is very similar to the Teradata OLAP Connector. The architecture is described in Simba’s case study on Teradata’s implementation, the architectural diagram is on page two. Simba’s MDX Provider is an ODBO provider installed on the same machine as Excel. Simba also has a tool for building cube definitions, which we call schemas. These schemas are saved in XML. Simba’s schema maps MDX metadata constructs to Impala table structures. When an ODBO compliant tool such as Excel issues an MDX query, Simba’s MDX Provider maps the MDX query to HiveQL, sends the HiveQL to Impala, collects the results and returns them to the end user.

The most important technical concept is that there is no intermediate server or cube structure that caches data, all queries go direct from Simba’s MDX Provider to the Cloudera Impala server in real time.

Because Simba already has an MDX engine that translates MDX to SQL, it was not overly difficult to adapt the engine to issue HiveQL. HiveQL supports almost all SQL constructs. For this reason, this demonstration provider supports the breadth of MDX functionality, including calculated members and measures in the cube (not shown). For those of you in the know about MDX, the breadth of MDX query support was shown when using slicers. The slicer issues some pretty complicated MDX queries under the covers, so it’s a pretty good representation of what you can do.

As the technology is in early development, it is not generally available for early testing. We put this demo out there so that we could get some feedback and validation of the design. You can leave your comments here or in the Impala user group thread. Specifically, we’d love to hear feedback on:

• The types of data sets you’d like to analyze and the types of analysis you’d like to perform
• The MDX functionality you’d like to use
• Excel functionality you’d like to use
• Other BI clients you’d like to see supported by such a provider (Tableau?)

We are pretty excited about putting the world’s most popular BI tool on top of Big Data and using this as our first step towards a rich Big Data analytics story. Simba has a booth at Strata Santa Clara so if you’re at the conference, you can stop by and talk with us about Excel, Big Data, BI, and analytics and how you plan to put all these technologies into place in your organizations. Your stories and input are vital so that we can build out tools you can use.

The Microsoft BI/Excel story so far...

The PASS BA conference is coming up in April and PASS ran their 24h of PASS event last week to whet our appetite.

The slides and recordings are now posted and there's a good number of stories on Excel, Power View and Big Data. Here's a list of notable sessions to catch up on the latest Microsoft happenings:

1. PowerPivot for Excel 2013 in Action
2. Analyzing Your Data with Power View
3. Visualizing Data with Power View
4. Big Data on the Microsoft Platform
5. What is Big Data?
6. Why? Where? Spatial Analysis in Excel 2013 Marketing Forecasting with Excel

 

 

Webinar series: Using Oracle OLAP data in multiple BI clients

Join us for our webinar series on connecting to Oracle OLAP on Tuesday, February 5. You may have already read about this on Dan Vlamis’s blog. Vlamis and Simba will co-present.

We will show the following BI clients:

SAP BusinessObjects Analysis, edition for OLAP with Simba XMLA for Oracle
9:00 a.m PST (11:00 a.m. CST or 12:00 p.m. EST)

IBM Cognos with Simba XMLA for Oracle
9:30 a.m PST (11:30 a.m. CST or 12:30 p.m. EST)

Microsoft Excel with Simba XMLA for Oracle
10:00 a.m PST (12:00 p.m. CST or 1:00 p.m. EST)

You will hear how some major customers in manufacturing, telecoms, and entertainment industries implemented analytic solutions using Oracle OLAP and Simba's Solutions for Oracle. The webinars will include live demos and customer case studies.

• See how Oracle OLAP cubes enable you to do new, powerful types of analysis in your familiar BI applications
• See how ad-hoc querying and data analysis can be performed directly on your Oracle OLAP cubes
• See how users can interactively build reports, drill into details and slice and dice data
• See how connectivity can be established without the need to install any software on the client machine

Our customers have used Simba's solutions to capitalize on their existing investments in their front-end BI clients and their Oracle databases. Because the back-end is built on the secure and scalable Oracle database, and the front-end is built on users' existing tools, both IT and business analysts are happy.

Register at: https://attendee.gotowebinar.com/register/5075922165628167424.

For more information about these and other Oracle OLAP solutions, see www.vlamis.com/simba and www.simba.com/oracle.

New Microsoft ODBC Drivers for SQL Server

On their blog, Microsoft has introduced the new ODBC Drivers for SQL Server. "Microsoft is adopting ODBC as the de-facto standard for native access to SQL Server and Windows Azure SQL Database." Microsoft seems to now be aligning the versions numbers and features of their Windows and Linux ODBC drivers as they also move to totally deprecate OLE DB. This is good to see because ODBC has now become the de-facto standard for both SQL and noSQL databases. You can also learn more about using ODBC to connect to noSQL databases by reading Simba brings the power of SQL to Cassandra, Hadoop, BigQuery, and MongoDB.

Big Data use cases...

Big Data ws undeniably top-of-mind for 2012 and will continue to in 2013. The market is growing but also maturing rapidly.

What caught my eyes last week was a session by Anand Venugopal ("AV") from Impetus. He presented use cases from their customer projects in the last four years. Drawing from these, AV categorized Big Data projects/initiatives into 3 types:

1. Batch mode Multi-source Data Analytics (<30%)
2. Real Time or Near Real Time Analytics (30%)
3. Social Media and Natural Language Processing (remainder, at ~40%)

The percentage numbers were particularly revealing. The three types are close but  natural language processing was the lead among the three. 

There's a lot more material in the session that's worth studying especially if you are keen to learn and understand the whats and where they are being used.

 

TDWI's report on SQL's staying power

A recent report from TDWI talks about the staying power of the SQL query language. "According to Philip Russom, research director for data management with TDWI, most organizations are actually increasing their use of SQL in spite of increased adoption of NoSQL databases such as Mongo, Cassandra, and Couchbase." "[A]lmost all tools for analytics, reporting, data integration, and data modeling support or generate SQL code that can be co-opted for analytics." This is exactly what Simba is seeing as we work with leading Big Data companies like DataStax, HortonWorks, and MapR. Users can gain plenty of advantages using databases such as Cassandra and Hadoop but they want to continue to use their existing BI, reporting, and analytic tools. Therefore, having ODBC/SQL interfaces to products like Cassandra, Hadoop, and MongoDB makes sense.  In fact, Simba has seen a major uptick in our OEM business of licensing SQL, ODBC, and JDBC technology to Big Data companies who want to provide their own "standard SQL" interfaces to their great new Big Data engine.  You can read the complete article here: SQL's Staying Power