Evolving BI

It’s been almost a year since I’ve written a new post.  The main reason for that is that things have been changing very fast for me personally and professionally.  I’ve always had a particular approach and belief to BI, but it’s really started to take shape and become clear over the last few months.  Feeling rejuvenated after the break, I’m ready again to share my adventures, and I’m starting it off big with thoughts on the current state of BI as I’ve observed and experienced it, and where we collectively need to go.  I’ve said most of these words to many people at meetups, conferences or tech talks, and the responses have generally been positive.  I’ve been trying to crystalize them so that I could share them more broadly and I’ll probably come back to this post and refine it over time.  It’s a long read, but bear with me and have an open mind.  Hopefully some part of it will inspire you.

Be a partner organization, not a service organization
This is the single, most important part.  If you only read one section, read this one.

A company takes on the culture of its business, and for the majority of those companies, that’s not technology.  That means that in most organizations, BI isn’t a core competency that the business/product team wants to build, so by precedence, the job is outsourced to centralized IT departments.  Historically this made sense because the skills you needed to be successful at BI at scale were technical skills like Data Warehousing, SQL, mastery of complex tools, and time consuming, heads down development cycles.  This put us somewhere in between a helpdesk and infrastructure where we were called upon for specific projects that the business deemed worthwhile and appropriate for us to do.  We lived project to project, task to task, roadmap to roadmap.

But it doesn’t have to be this way.  Instead of positioning yourself as a service organization, try to position yourself as a partner organization.  Be part of the business.  Think about it as one big product team (even if your company’s product happens to be a service), where each seat at the table is a different role that contributes to common goals.  The definition of these roles will be different depending on what it is your company does, but I think you can find the parallel to your company.  Generally speaking, seats at the table are occupied by a Product Manager, Designer, Engineer and Analyst.  When they need some tool built, be it a report, dashboard or dataset, they call on you, the BI Team in IT with a specific request.  They decide what your contribution will be and they decide when and what to tell you.  Their underestimation of the effort on your part is what leads to fire drills and their lack of understanding of exactly what it is you’re providing leads to incomplete requirements.  So why not pull up a chair and join the table?  You’re providing a role on the team just as important as the others, so speak for yourself, set your own timelines, propose your own solutions on what could be helpful and work WITH the business, not FOR the business.

Own your data, it’s not just numbers, be its champion

But you probably support a large portion of the company, if not the entire place.  So how can you be in all places at once?  Segment your team vertically, not horizontally.  A service-oriented BI Team has a pool or resources where it’s next man up.  A partner-oriented BI Team tries to build Subject Matter Experts of their own who live and breathe the data they’re working with just as much as those responsible for consuming it.  Become a consumer of your own data, and it will cease to be numbers on a screen and begin to tell you stories, give you ideas, and no one will need to tell you the next thing to build because it will be plain to you.  Take pride in your products, check dashboards in the morning and start to question not just the accuracy of the numbers, but what the numbers actually mean.  Ask an analyst why a particular market looks like an outlier, ask a product manager why we continue to invest in a feature that’s clearly failing.  In the same way every other role at the table thinks about the company’s product and the dashboard is just a window into its soul, you too should use it for this purpose.

Declare what you’re optimizing for

But it still takes so long to build things!  There is still so much planning and understanding, that we can’t possibly keep up with the speed of the business!  It’s easy to ask for things, but it takes a lot to build them!

This is where you’re shooting yourself in the foot.  If a product team can put out products faster than you can build reports on those products, you’re doing something wrong, even when compensating for the differences in scale (obviously there are more people working on products than on reporting in your organization).  It could be that you need to ramp up more resources, or it could just be that you’re not optimizing for the right things.

Determine what you’re optimizing for.  If you’re in a finance organization, you’re probably optimizing for accuracy because every penny is pretty critical.  Even if you’re not, but you’re working with data that has to be SOX compliant, you’re probably in the same boat.  You need to be air tight in your methodologies, documentation and processes.  Chances are your business units are also bound by the same chains, so previous statements still apply here.

But most teams aren’t supporting such things.  Most BI teams support activities that are about understanding product direction and making decisions on where to take those products.  Those projects are directionally important, but not decimal important.  The more important factor here is speed.  If you can’t keep up with the speed of the product team, you risk not being relevant, and if you’re not relevant, then what value are you really providing to the company?  In these cases, the value provided is one of convenience, where you’re really just saving someone else a little bit of time.  But we have skills that others don’t such as skills in scalability, design and functionality.

Ruthlessly evaluate every process for how it contributes or hinders what you’re optimizing.  If speed is what you’re optimizing for (and it should be), then kill processes that are getting in its way.

Here are some examples for how you can optimize for speed:

* Relax change management.  Sometimes we get a little caught up in CYA and we end up in a state of “process lock”.  We are locking because we’re too afraid of making a mistake so our protection is more review committees for RCA, more forms, more documentation, more QA environments, less access to features for developers.  What’s the worst that happens if these things all went completely away?  Sure, mistakes will happen, but you’re also empowered to fix them faster.

* Trust your team.  It’s natural to hold tight control over the core aspects of the system (administration, configuration, schema, object creation) but it’s fine to relinquish this control and trust that your team has the competency and responsibility to do the right thing with that trust.  When the responsibility is on us, we take greater care, build greater things, and feel a sense of accomplishment when we’ve made something because we were able to put our mark on a project without having to gain the approval of others every step of the way for small things that ultimately don’t make a difference.

* Take Risks.  Don’t take limitations at face value.  Give people the opportunity to be a hacker.  If the software can’t do something, think of a way to make it happen.  More often than not, someone has a “crazy idea that might just work”.  Evaluate if it’s worth the risk (and maintenance) of doing something crazy, because I’ve often found that it is.  Don’t worry if no one else does it that way, that just means you’re blazing trails.  If something breaks, then fix it.  If something fails, then throw it out.  You learn more from doing than worrying.

* Iterate, don’t spec.  One thing that will cause disconnect and guarantee a slow down in progress is to continually meet, discuss, document, repeat.  Communication is key, but have something to communicate with.  Not words or drawings, but something people can actually touch.  We’ve all had a project where we built exactly what someone was asking for and then got upset when they decided to change everything.  The reason is that no one knows exactly what they want when they’ve never seen it before.  But put something in their hands, and watch where they get stuck.  Build something fast, ship incremental features even if it’s only 2 of the 5 things they wanted, and build on quick wins.  You’ll evolve something along the way that is exactly what everyone needs as opposed to what they thought they wanted 3 months ago when the project started.  When you optimize for speed, this type of style will come naturally.

* Best practices are what’s best for your practice.  Don’t let anyone tell you that you’re not supposed to do it that way.  Best practices are suggestions and ideas, not laws.  Every environment is different, every team is staffed with different numbers and talents, and every team’s goals are different.  When you find what you’re optimizing for, no “best practice” is safe.  Try new things, no matter how crazy, and test them out.  If you get burned or it hurts more than it helps, then ditch it.  Eventually, you’ll land on processes that work for you and your team, but don’t forget that these *will* change over time.  If your organization is growing (or shrinking) quickly, the dynamics are changing and so should you and how you do things.  Never accept that things are the way they are because that’s the way they’ve always been.

When we build walls with process, we aren’t protecting ourselves but perpetuating the blame game.  If something does go wrong despite your best efforts to plan, the general line of thought is that the process is “clearly broken”.  We needed another step.  Someone didn’t follow the process.  It was the technology’s fault (a bug).  But when there is no process, when these walls are torn down, we are the only ones left.  Now, we are all accountable.  We take extra care with our actions because we know that there is nothing to fall back on except the support of our team.  There’s no one to blame but ourselves, and we end up putting more care, more attention to detail, and take ownership of our data.  It ceases to become a number on the screen or a form in the pile or a check on a box, but it’s our contribution, our impact on the company.

What’s worked for me, a case study

What follows aren’t all my ideas but are more of a function of a culture that fosters likeminded individuals.  This is not a recommendation for what you should do, only what worked for us.  It serves as an example of the types of risks and tradeoffs you could make in your environment.

My current company was a long time MicroStrategy shop, and this blog was a beacon that eventually connected us (side note: let that be a lesson in the power of blogs.  I’m not the smartest or best, I just took notes and wrote them down on the internet).  When the team was small, everything was fine.  We were moving along at normal speed and doing things the way they’re supposed to be done.  We had some good wins, won over some departments, and as demand grew, our team grew.  After a certain point, we were having trouble keeping up and decided to evaluate some radical changes.  We were optimizing for speed above all else.

* Kill Dev & QA.  We had a small, but talented MicroStrategy team and change management was causing too much of a delay.  Tickets, migrations, packages .. it was too much trouble, so we said, “Who needs this?” and gave every developer full architect access to Production.  As outrageous as this sounds, we’ve never once had an issue in Production that could have been prevented had we only built it in Dev and tested it in QA first.  We were just as likely to have had an out of sync environment or corruption during migration, so why not just enable us to make fixes on the fly?  We created copies of dashboards to act as hot backups and kept daily metadata backups in case we ever had to roll back in a catastrophe (we never did).

* Ditch the Schema.  This one will be hard to swallow for some, but I believe the value of the schema is overrated.  It was taking us too long to build out scores of objects, wire them all up and debug their odd behavior, so we went with freeform SQL.  We were all pretty technical and fluent in SQL, and we could write a query for the dataset that a report or dashboard needed way faster than building the supporting objects.  In cases where there were product reasons that required schema objects (like if we needed to use certain kinds of prompts or custom groups or whatever), we’d put the SQL in a view and build a few schema objects in a silo that was fast and manageable.  Yes, there were “single version of the truth” issues from time to time, but nothing major.  It was well worth the speed trade off, and we were very well equipped to fix those issues when we found them very fast.

* Evaluate the Tools.  The way we were using MicroStrategy was no longer playing to its strengths.  We weren’t using the SQL Engine, we weren’t doing migrations or multiple environments or much in the way of its infrastructure tools.  Our development style was focused on fast, iterative, interactive dashboards, and none of those things are MicroStrategy’s top traits.  As such, we evaluated other tools and added Tableau to our tool library, where those are their top 3 strengths.  MicroStrategy still had its place for very complex dashboards (see the jQuery stuff we’ve done) and it’s still unmatched in daily email functionality, but the majority of our day to day work was now happening in Tableau.  Since we were already hand writing the SQL anyway, this was a pretty natural transition.  For each project, we then just chose the best tool for the job.  Huge data, complex layout, email alerts: MicroStrategy.  Exploratory, interactive, I need it today: Tableau.

* Became a Partner Organization.  This was a tough one (and a transition we’re still going through), but we now have a seat at the table with product teams.  Due to the speed at which we can develop, we can have near real-time development cycles with the product managers, analysts and engineers, which brings us inline with their contribution speed.  If someone needs a metric, they know they come to us, similar to how when someone finds a bug they go to the engineer, when someone has a theory they go to the analyst and when someone has an idea they go to the product manager.  It was a steep hill to climb to get there, but in some cases we’ve achieved it and have our sights set on achieving it everywhere.  We now contribute to the product in a deep way; a level of evolvement we’ve never experienced before.

We’re no longer a service organization, we’re now a partner organization.
I’m no longer a MicroStrategy blogger, I’m now a BI blogger: pushing what’s possible.

I’ll cover more than just technical MicroStrategy tips and tricks going forward, though there will surely still be some of those.  Some posts will be philosophical (like this one), some will include tricks on Tableau since I’m using that a lot now.  Others will include some of the fun software I’ve written to help support these tools, but all of it will be in the spirit of finding solutions to tough problems that otherwise looked unsolvable.