Dashboards as a Platform

A look back …

feel free to skip this part if you don’t care about my life story

I originally started this blog as a way to share MicroStrategy hacks and knowledge that I had accumulated over the years.  It was a subject I felt passionate about as I was very active on the forums, I presented many times at the conferences and wrote over a hundred and fifty posts on this site.  I even setup a help line and answered over a thousand questions.  I did it because I really liked the engineering challenge of figuring out MicroStrategy.  It was the perfect mix of Data Warehousing, SQL, Visualization and puzzle solving to me.  I also got quite a rush by making it do things that it didn’t know it could do and each one of those hacks got crazier and crazier.  It finally culminated in a huge presentation at MicroStrategy World 2013 where I felt that we raised the bar so high, we broke it in half.

Eventually though, I grew pretty weary of the games.  I was spending a tremendous amount of effort to make MicroStrategy do things that I thought it should be able to do out of the box.  I was frustrated when the company was focused on other endeavors that I didn’t think were as important as the fundamental things I was stretching to add.  That’s when I ran into Tableau.

I had a similar, but shorter lived romance with Tableau.  They are a truly amazing company lead by incredible people, topped only by their unbelievable community.  I was swept away by how they were everything that MicroStrategy wasn’t and they provided exactly the things I was looking for.  I was the first in my team to experiment with it and find success.  Others followed suit and seemingly overnight our MicroStrategy team had converted to Tableau for 100% of our development.  Their platform wasn’t quite as hackable as MicroStrategy’s, but I still managed to find some ways.  Their community was infinitely more mature so I never really got into Tableau blogging, except when it came to the hacks.  Just like with MicroStrategy, it culminated in a crazy presentation at the Tableau Conference 2014 (some 19 months after the MicroStrategy one) where we wowed the crowd with our wacky hacks and crazy things we’d done.  But just like with MicroStrategy, I felt weary of it again.  As my needs got more advanced, I was getting easily frustrated at the hoops I had to jump through and even though as a company they were actually moving to address the things I felt were the most important, it just wasn’t fast enough.

I realized a few things in both cases

• Vendor software develops for the lowest common denominator.  They have to make features that are widely applicable to everyone which in turn means they fully address no one.  We turn to blogs, books and conference sessions to figure out how to bend it to do what we want it to do.
• Even if they are aligned with your needs, they can’t move even remotely fast enough.  Features are measured in years, not days or weeks like the real world needs.
• Features needed to be developed to do as little as possible, lest they make the UI too complicated for the average person.  This is something I saw both sides of.  MicroStrategy grew to incredible complexity where you needed weeks of training to be able to operate it.  It could do a lot of things, but it required a massive ramp up curve.  Tableau was super easy to learn and you could build powerful things quickly, but if you went off of the predefined rails it got needlessly complex very fast.  MicroStrategy couldn’t make their interface easy enough to use to unlock what it could do, and Tableau couldn’t make their product do enough things while keeping the target audience of non-technical users.
• The “hacks” presentation at each respective conference was the death knell.  I completely stopped using both platforms within a month after each presentation.  Both showed the lengths I was willing to go to in order to get the platforms to do what I needed, but in the end it wasn’t fulfilling.  When I showed users what I had done, it was very anticlimactic.  “Look, responsive and formatted tooltips in MicroStrategy charts” and “Tableau dashboards that don’t refresh in your face” are the things that make people say, “ugh, finally” not, “whoa, wow”.  It just wasn’t impactful.

So where did I go?

I’ve had this idea for several years now: Photoshop for Dashboards.  I wanted to paint the data that I saw on the screen and not build schemas or calculated fields or other scaffolding.  Those things were in my way.  I just wanted to shape the data in ETL to feed a dashboard exactly the minimum it needs to consume.  Ultimate speed with the smallest possible footprint.  I built prototypes as early as 2011 (back when I was doing all of the Flash widgets in MicroStrategy) and shopped the idea around internally for a few years to various people and teams looking to get some traction but I never did.  After TC14 I had some renewed motivation and finally formulated what I thought the missing concept was: Dashboards as a Platform.

I kept thinking about Dashboards as a single tool because that’s what vendors have always made.  But then I realized that it doesn’t need to be a tool.  I don’t have to solve every problem via a UI.  I could make a hybrid that had a UI for convenience but would directly accept code augmentations.  Instead of SDKs being a limited afterthought, it would be the main event.  With this clarity, I found that there was already a very similar internal tool that already had everything I needed.  They were mostly focused on operations and not enterprise reporting, but they were interested in my use case and turned out to be amazing partners.  I’ve been working closely with them ever sense.

Dashboards as a Platform

The concept is simple: the platform is the framework of connecting to data and displaying it visually.  The displays are individual widgets which are self contained visualization objects.  Dashboards are then a layout of widgets.  The power comes from the widgets themselves.  I can make widgets that look any way I want, rendering with Highcharts, d3, custom libraries, all mixed together.  If I make an enhancement to the Line Chart and add some better formatting or spacing or colors or richer tooltips, every single dashboard in the system gets the change because I’m modifying the actual engine.  In Vendor tools, each dashboard is a silo and if I decide I want some new functionality, I have to edit them all individually.

Widgets can also be specialized.  A common design pattern I have today is what I call a Delta Table:

For a time series based data set, it shows the current value along with the % change from a week/month/year ago.  Making that table in Vendor tools is certainly possible and not terribly difficult, but it’s tiring.  I have to create several of them on every single dashboard.  It’s more common than Line Charts at my company.  I want to paint dashboards, I don’t want to spend my whole time creating metrics and measures with tedious formulas and calculation levels.  The source is just a simple date/dim/metric source which is effectively select * from table.  So I built a custom Delta Table widget and built the ability to slice off the first row, calculate the time series deltas (based on actual dates, not X rows ago) and format it just the way I liked it.  The result is that I can build one of these tables on a dashboard in about 10 seconds.  When I want new crazy features like, “I’d like to calculate y/y as of last week”, or “I’d like to have a column of sparklines in the table”, I can add an option to it and now I can do it with a checkbox.  For Line Charts, I use year over year a lot.  Instead of figuring out the calculations to do that in a vendor tool, I just added a checkbox “Render y/y” and in 1 second the graph is done with a simple time series input.  A dashboard with a dozen or so widgets that would take me several hours to build in Tableau now takes me a few minutes to build.

You may think, “well sure, if you’re going to hand code it then that’s cheating”, but that’s not really the case.  I have contributed a bunch of code to it, but the vast majority of people who use the platform don’t.  It’s effectively designed as an open source platform internally.  While all of the code to our internal tools are internally open source, it’s designed for user widget contributions.  People will make feature requests for existing enhancements or entirely new widgets, and someone will build them.  Sometimes the changes are minor enough that they just take a line or two of code and 5 minutes to add.  The end result is that users now have a tool that builds their dashboards as opposed to a tool to try to build their dashboards in.  The difference may sound subtle, but we’re now in a position where either you can build exactly the dashboard you want in a couple of minutes or you (or someone else) can build that feature in a few hours / days of work (and then you can build your dashboard in a couple of minutes).

Other advantages are incredible synergies with other systems.  There are wild integrations we’ve done with other internal systems that are easy to do in this environment but would be impossible for vendors.  We can also connect to databases that vendors can’t support, like internal proprietary ones.  Some of these are insanely fast or not even SQL based, but trivial to connect to from a platform like this.

Realizing the Dream

If you’re sold and you want it, the answer is sorry, it (probably) won’t be open sourced.  It’s too ingrained in our infrastructure to be able to outright share it.  I also don’t expect a vendor is going to offer such a solution any time soon because I think it’s a tough sell to the market .. for now.  I think another transition has to happen first, but this post is already too long so I’ll talk about that next time.