It’s been almost a year since I’ve written a new post. The main reason for that is that things have been changing very fast for me personally and professionally. I’ve always had a particular approach and belief to BI, but it’s really started to take shape and become clear over the last few months. Feeling rejuvenated after the break, I’m ready again to share my adventures, and I’m starting it off big with thoughts on the current state of BI as I’ve observed and experienced it, and where we collectively need to go. I’ve said most of these words to many people at meetups, conferences or tech talks, and the responses have generally been positive. I’ve been trying to crystalize them so that I could share them more broadly and I’ll probably come back to this post and refine it over time. It’s a long read, but bear with me and have an open mind. Hopefully some part of it will inspire you.
Be a partner organization, not a service organization
This is the single, most important part. If you only read one section, read this one.
A company takes on the culture of its business, and for the majority of those companies, that’s not technology. That means that in most organizations, BI isn’t a core competency that the business/product team wants to build, so by precedence, the job is outsourced to centralized IT departments. Historically this made sense because the skills you needed to be successful at BI at scale were technical skills like Data Warehousing, SQL, mastery of complex tools, and time consuming, heads down development cycles. This put us somewhere in between a helpdesk and infrastructure where we were called upon for specific projects that the business deemed worthwhile and appropriate for us to do. We lived project to project, task to task, roadmap to roadmap.
But it doesn’t have to be this way. Instead of positioning yourself as a service organization, try to position yourself as a partner organization. Be part of the business. Think about it as one big product team (even if your company’s product happens to be a service), where each seat at the table is a different role that contributes to common goals. The definition of these roles will be different depending on what it is your company does, but I think you can find the parallel to your company. Generally speaking, seats at the table are occupied by a Product Manager, Designer, Engineer and Analyst. When they need some tool built, be it a report, dashboard or dataset, they call on you, the BI Team in IT with a specific request. They decide what your contribution will be and they decide when and what to tell you. Their underestimation of the effort on your part is what leads to fire drills and their lack of understanding of exactly what it is you’re providing leads to incomplete requirements. So why not pull up a chair and join the table? You’re providing a role on the team just as important as the others, so speak for yourself, set your own timelines, propose your own solutions on what could be helpful and work WITH the business, not FOR the business.
Own your data, it’s not just numbers, be its champion
But you probably support a large portion of the company, if not the entire place. So how can you be in all places at once? Segment your team vertically, not horizontally. A service-oriented BI Team has a pool or resources where it’s next man up. A partner-oriented BI Team tries to build Subject Matter Experts of their own who live and breathe the data they’re working with just as much as those responsible for consuming it. Become a consumer of your own data, and it will cease to be numbers on a screen and begin to tell you stories, give you ideas, and no one will need to tell you the next thing to build because it will be plain to you. Take pride in your products, check dashboards in the morning and start to question not just the accuracy of the numbers, but what the numbers actually mean. Ask an analyst why a particular market looks like an outlier, ask a product manager why we continue to invest in a feature that’s clearly failing. In the same way every other role at the table thinks about the company’s product and the dashboard is just a window into its soul, you too should use it for this purpose.
Declare what you’re optimizing for
But it still takes so long to build things! There is still so much planning and understanding, that we can’t possibly keep up with the speed of the business! It’s easy to ask for things, but it takes a lot to build them!
This is where you’re shooting yourself in the foot. If a product team can put out products faster than you can build reports on those products, you’re doing something wrong, even when compensating for the differences in scale (obviously there are more people working on products than on reporting in your organization). It could be that you need to ramp up more resources, or it could just be that you’re not optimizing for the right things.
Determine what you’re optimizing for. If you’re in a finance organization, you’re probably optimizing for accuracy because every penny is pretty critical. Even if you’re not, but you’re working with data that has to be SOX compliant, you’re probably in the same boat. You need to be air tight in your methodologies, documentation and processes. Chances are your business units are also bound by the same chains, so previous statements still apply here.
But most teams aren’t supporting such things. Most BI teams support activities that are about understanding product direction and making decisions on where to take those products. Those projects are directionally important, but not decimal important. The more important factor here is speed. If you can’t keep up with the speed of the product team, you risk not being relevant, and if you’re not relevant, then what value are you really providing to the company? In these cases, the value provided is one of convenience, where you’re really just saving someone else a little bit of time. But we have skills that others don’t such as skills in scalability, design and functionality.
Ruthlessly evaluate every process for how it contributes or hinders what you’re optimizing. If speed is what you’re optimizing for (and it should be), then kill processes that are getting in its way.
Here are some examples for how you can optimize for speed:
* Relax change management. Sometimes we get a little caught up in CYA and we end up in a state of “process lock”. We are locking because we’re too afraid of making a mistake so our protection is more review committees for RCA, more forms, more documentation, more QA environments, less access to features for developers. What’s the worst that happens if these things all went completely away? Sure, mistakes will happen, but you’re also empowered to fix them faster.
* Trust your team. It’s natural to hold tight control over the core aspects of the system (administration, configuration, schema, object creation) but it’s fine to relinquish this control and trust that your team has the competency and responsibility to do the right thing with that trust. When the responsibility is on us, we take greater care, build greater things, and feel a sense of accomplishment when we’ve made something because we were able to put our mark on a project without having to gain the approval of others every step of the way for small things that ultimately don’t make a difference.
* Take Risks. Don’t take limitations at face value. Give people the opportunity to be a hacker. If the software can’t do something, think of a way to make it happen. More often than not, someone has a “crazy idea that might just work”. Evaluate if it’s worth the risk (and maintenance) of doing something crazy, because I’ve often found that it is. Don’t worry if no one else does it that way, that just means you’re blazing trails. If something breaks, then fix it. If something fails, then throw it out. You learn more from doing than worrying.
* Iterate, don’t spec. One thing that will cause disconnect and guarantee a slow down in progress is to continually meet, discuss, document, repeat. Communication is key, but have something to communicate with. Not words or drawings, but something people can actually touch. We’ve all had a project where we built exactly what someone was asking for and then got upset when they decided to change everything. The reason is that no one knows exactly what they want when they’ve never seen it before. But put something in their hands, and watch where they get stuck. Build something fast, ship incremental features even if it’s only 2 of the 5 things they wanted, and build on quick wins. You’ll evolve something along the way that is exactly what everyone needs as opposed to what they thought they wanted 3 months ago when the project started. When you optimize for speed, this type of style will come naturally.
* Best practices are what’s best for your practice. Don’t let anyone tell you that you’re not supposed to do it that way. Best practices are suggestions and ideas, not laws. Every environment is different, every team is staffed with different numbers and talents, and every team’s goals are different. When you find what you’re optimizing for, no “best practice” is safe. Try new things, no matter how crazy, and test them out. If you get burned or it hurts more than it helps, then ditch it. Eventually, you’ll land on processes that work for you and your team, but don’t forget that these *will* change over time. If your organization is growing (or shrinking) quickly, the dynamics are changing and so should you and how you do things. Never accept that things are the way they are because that’s the way they’ve always been.
When we build walls with process, we aren’t protecting ourselves but perpetuating the blame game. If something does go wrong despite your best efforts to plan, the general line of thought is that the process is “clearly broken”. We needed another step. Someone didn’t follow the process. It was the technology’s fault (a bug). But when there is no process, when these walls are torn down, we are the only ones left. Now, we are all accountable. We take extra care with our actions because we know that there is nothing to fall back on except the support of our team. There’s no one to blame but ourselves, and we end up putting more care, more attention to detail, and take ownership of our data. It ceases to become a number on the screen or a form in the pile or a check on a box, but it’s our contribution, our impact on the company.
What’s worked for me, a case study
What follows aren’t all my ideas but are more of a function of a culture that fosters likeminded individuals. This is not a recommendation for what you should do, only what worked for us. It serves as an example of the types of risks and tradeoffs you could make in your environment.
My current company was a long time MicroStrategy shop, and this blog was a beacon that eventually connected us (side note: let that be a lesson in the power of blogs. I’m not the smartest or best, I just took notes and wrote them down on the internet). When the team was small, everything was fine. We were moving along at normal speed and doing things the way they’re supposed to be done. We had some good wins, won over some departments, and as demand grew, our team grew. After a certain point, we were having trouble keeping up and decided to evaluate some radical changes. We were optimizing for speed above all else.
* Kill Dev & QA. We had a small, but talented MicroStrategy team and change management was causing too much of a delay. Tickets, migrations, packages .. it was too much trouble, so we said, “Who needs this?” and gave every developer full architect access to Production. As outrageous as this sounds, we’ve never once had an issue in Production that could have been prevented had we only built it in Dev and tested it in QA first. We were just as likely to have had an out of sync environment or corruption during migration, so why not just enable us to make fixes on the fly? We created copies of dashboards to act as hot backups and kept daily metadata backups in case we ever had to roll back in a catastrophe (we never did).
* Ditch the Schema. This one will be hard to swallow for some, but I believe the value of the schema is overrated. It was taking us too long to build out scores of objects, wire them all up and debug their odd behavior, so we went with freeform SQL. We were all pretty technical and fluent in SQL, and we could write a query for the dataset that a report or dashboard needed way faster than building the supporting objects. In cases where there were product reasons that required schema objects (like if we needed to use certain kinds of prompts or custom groups or whatever), we’d put the SQL in a view and build a few schema objects in a silo that was fast and manageable. Yes, there were “single version of the truth” issues from time to time, but nothing major. It was well worth the speed trade off, and we were very well equipped to fix those issues when we found them very fast.
* Evaluate the Tools. The way we were using MicroStrategy was no longer playing to its strengths. We weren’t using the SQL Engine, we weren’t doing migrations or multiple environments or much in the way of its infrastructure tools. Our development style was focused on fast, iterative, interactive dashboards, and none of those things are MicroStrategy’s top traits. As such, we evaluated other tools and added Tableau to our tool library, where those are their top 3 strengths. MicroStrategy still had its place for very complex dashboards (see the jQuery stuff we’ve done) and it’s still unmatched in daily email functionality, but the majority of our day to day work was now happening in Tableau. Since we were already hand writing the SQL anyway, this was a pretty natural transition. For each project, we then just chose the best tool for the job. Huge data, complex layout, email alerts: MicroStrategy. Exploratory, interactive, I need it today: Tableau.
* Became a Partner Organization. This was a tough one (and a transition we’re still going through), but we now have a seat at the table with product teams. Due to the speed at which we can develop, we can have near real-time development cycles with the product managers, analysts and engineers, which brings us inline with their contribution speed. If someone needs a metric, they know they come to us, similar to how when someone finds a bug they go to the engineer, when someone has a theory they go to the analyst and when someone has an idea they go to the product manager. It was a steep hill to climb to get there, but in some cases we’ve achieved it and have our sights set on achieving it everywhere. We now contribute to the product in a deep way; a level of evolvement we’ve never experienced before.
We’re no longer a service organization, we’re now a partner organization.
I’m no longer a MicroStrategy blogger, I’m now a BI blogger: pushing what’s possible.
I’ll cover more than just technical MicroStrategy tips and tricks going forward, though there will surely still be some of those. Some posts will be philosophical (like this one), some will include tricks on Tableau since I’m using that a lot now. Others will include some of the fun software I’ve written to help support these tools, but all of it will be in the spirit of finding solutions to tough problems that otherwise looked unsolvable.
Oops. In moving the blog to a new host, I had it setup to require an account to leave comments, but didn’t allow people to create accounts. Fixed now, and no accounts required 🙂
Cool and very relevant to today’s marketplace. Always insightful. Welcome back.
Great insights and thought leadership, Bryan. I am glad you decided to start blogging again and look forward to your future posts.
Great stuff Bryan. The team I’m on is in the midst of a pivot as well, although ours is from a delivery focus to a self-service focus. Have you guys done much in that way, or do you have others that focus on self-service these days?
Our version of self service is hand query the data which doesn’t require much support on our part, so we’re a little different. Otherwise, I’m not really as big on self service as a lot of people are, at least not with the solutions currently out there. I think self service is a little more unstructured than just handing someone a cube to slice, and there are a few products trying to do it (internally and externally) that I’m really excited about.
Can you share what products you mean?
Does the new post mean that “Ask Bryan” is back?
I can’t at the moment share information about either, but the moment I can, you can bet I will!
I don’t think I’ll ever be able to bring that level of support back. It was a pretty big time commitment, which is something I just don’t have any more. In addition to being busier at work in general than I was in those days, I also have 2 kids now that take up any remaining free time. 😉
Great advice. Speed matters. Nobody wants perfection at the cost of insight.
So glad you’re updating your blog again. I enjoyed reading the information under “optimizing for speed” and liked that you incorporated what worked well in your environment. Hope to see more of this in the future. Will keep a watchful eye on your future blogs.
Its great to see that your blog has returned as it was always a helpful resource when I too was just a MicroStrategy guy. I was also always very interested by all the cool stuff that you were getting up to at Facebook, so much so that I came very close to inquiring about job opportunities.
Today however I’m more involved in data warehouse development for one of the biggest mobile games developers and have more dealings with QlikView rather than MSTR or Tableau. That said, I’ll still be interested to hear your continuing thoughts about the state of BI and the different tools that are out there.
Welcome back Bryan. I used to visit every month in last year to see, if you posted anything. Now i am very happy, when you have came back with Bryan BI Blog. I like your web blog design, its colorful and well organized.
It is not the right place but as I do not know the other way to let you know – so…I use this one.
Some links between the blog entries do not work any more e.g. http://www.bryanbrandow.com/2012/02/automatic-export-via-url-api.html is pointing to many other articles that show “Error 404. Page not found!”
Thanks for the report. I moved from Blogger to WordPress, and I was able to match the URLs pretty close, but some may still be broken because WordPress ignores small words like “in” while Blogger included them. That said, I see the issue you’re pointing out is that the links in the articles that imported are using the original WordPress style and not my modified style. I’ll see if there’s a way I can fix that.
Glad your back! Your blog is refreshing and a great BI resource!
Thanks for your hard work,
Glad to see you are back. Hope to see more of you in the future.
Bryan, great article. I have thought of BI in a similar context almost since I was exposed to it. I would like to add a couple thoughts alongside yours.
I see the concept of BI occupying two different, but related spaces.
From one perspective, BI is a platform. It allows BI software vendors to sell solutions based around integration solutions. This is clearly a pain point for many companies as they outgrow their homegrown, or context dependent, reporting solutions. BI software vendors step in with a platform that allows companies to integrate disparate data feeds into a single solution.
From the perspective of business users, BI is (or should be) a set of answers. It is the actual Intelligence. Businesses want to see opportunities, based on analyses of data within their organizations. Frequently, they lack even basic visibility into simple operations of specific departments, let alone across the enterprise.
As BI practitioners, we are required to operate within these often competing perspectives. On one hand, we have to architect, build and maintain the infrastructure. On the other, we have to deliver analytic insights. Friction arises when the infrastructure effort outpaces the ability to deliver business answers. As the BI platform expands, it requires more maintenance. Concentrating the ability to create schema into a few hands exacerbates this problem. As a result, development cycles become longer. This result is guaranteed in a rigid BI environment with strictly controlled processes and limited permissions for developers.
There are many negative outcomes. Business users may begin to view BI in a negative light. Savvy users will eventually find a way to circumvent the standard development process, either through back channels or by writing their own SQL. Often the loudest voices will get their requests granted, even if they should be a lower priority. I have witnessed these and many other others first hand.
In my opinion, the worst outcome is that BI resources are increasingly devoted to infrastructure tasks at the sacrifice of delivering meaningful results. The technical becomes prioritized over the analytic. The greatest value added by BI is the ability to provide new insights that can profoundly influence business by opening up unforeseen opportunities. The infrastructure is the means, not the end goal.
My main take away from your article is an advocacy for a shift toward the analytic end of the BI spectrum. It is something I wholeheartedly endorse.
Glad you are back man! kkkkkk
Glad you are back and congratulations to have babies and be a father 🙂
Very interesting case study, Bryan, and a painful one for a schema enthusiast like me but totally understandable in your case.
I’ve always thought that the main strength of Microstrategy was the schema and the SQL generation out of it, and I still think it doesn’t have match. But it’s true that to unleash all its potential you need to be in the following case:
– Your needs are more of self service / data exploration and less of brebuilt dashboards
– You are fine with the fact that that exploration needs to happen online
– Your users are fine with the basic grids – graphs interface which has already more than 10 years. Visual Insights is arriving to replace it there but not yet, and besides still Visual Insights is a bit Fordian (as long as you like your car in black – or white you’re fine)
– You have a fast database behind. And / or, you have plenty of dynamic sourcing cubes AND no metrics that do count distinct of transactional fields like number of orders or customers. If you have those kind of metrics, you can’t leverage fully cubes (or aggregated tables for that matter) in a self service, drag-any-attribute-you-want environment.
– And most important: you have a really really good architect capable of creating a single big schema able to service many departments / users, keeping it out-of-the-box pure so the dynamic sourcing cubes don’t get disabled, and resisting the temptation of splitting it into different projects when things get tough, or creating report-specific attributes and facts that require to keep the schema siloed and that tend to kill the self-service.
If you tick all the boxes then I think MSTR schema really doesn’t have match, it is really much better than other tools, and the bigger is the scale of your user base the more you enjoy that economy of scale. The moment you don’t tick all the boxes and you start to descend into the arena of one-report one-cube, or Freeform SQL cubes / reports, you lose economy of scale and then you put yourself in a position in which other tools are serious contenders, we all know that. Tableau and Qlikview are awesome on the front end providing that somebody builds a cube first for you. But in my opinion then you aren’t in a situation of self service of non SQL savvy users, but rather in a situation of “ask me for a report / cube and I’ll do it really fast”. Gartner says we go towards data analytics and self service, but I guess all depends on the definition each of us has of self service.
I agree BI needs to be a business solutions partner, but do not agree that live/production changes without change management fits all IT scenarios; for example, health care data is highly regulated and requires access auditing. It is possible that untracked production changes would lead to the display of personal health information in a BI dashboard and result in a legal action. A legal action has the potential to close a business. My second example is a recent production support issue I helped resolve; our warehouse was suddenly not able to execute the SQL of a MSTR dashboard, which had been executing successfully for the prior 4 days. Our change management process made it easy to determine what was not the problem and helped us know what to focus our troubleshooting efforts on. Without change management tracking, troubleshooting could have taken us down many ‘rabbit holes’ before finally isolating the real issue. I value your technical experience in BI, but think the BI future you are envisioning is limited to less regulated data.
For sure, totally agree. I’m not prescribing any solutions, only saying that you should challenge the status quo and consider what your environment really needs, and not what others tell you it should be. Clearly there are environments with sensitive data that will require high levels of scrutiny and logging (even in my current company, even though my team operates in the way I’ve described, other BI teams still operate in a way you’ve described because of the nature of their data).
The scenarios you described weren’t my target audience, but I did caveat them in the “Declare what you’re optimizing for” section (you’d be more interested in accuracy and accountability than speed and iteration) and in my case study example, I stated, “This is not a recommendation for what you should do, only what worked for us.”
Still, you make an important distinction that is important for others to understand that all environments have their own considerations and priorities, and no “best practices” is a universal fit. Thanks for the feedback!
I’ve worked in the BI field for just over 20 years (14 years with MicroStrategy) and had up until about 4 years ago had my stake firmly planted in the standard methodology camp. However I have found that “Best Practices” such as thickly layered change management, rigid warehouse architecture and adherence to the MicroStrategy logical model are not in fact always the best practice. Often this classic approach creates a situation where the BI environment cannot respond quickly to the changing needs of the business. Standards and controls certainly have their place when it comes to customer facing products as well as data covered by regulations like SOX and HIPPA. However, I find for internal facing applications a more flexible approach is best.
Corporations invest billions in technology focused human capital. Unfortunately, many of these brilliant people are utilized as data/report request “Order Takers” and not consultants who collaborate with business to solve problems. So I don’t think adopting a more flexible view of how to deliver BI is a “Cool” idea but is an untapped resource for competitive advantage in business.
i am having a different scenerio currently we are using MSTR Vi designed graphs in normal Dashboards and its working till the previous versions but with the latest update to 9.4.120 MSTR Ipad app is crashing but it used to work till the previous version we are using linking to link between dashbaords.
so could you please let me know the solution i got the crsh log from DSSerror log for the Mobile ASPx and log files and the app crash log from the ipad even
Date/Time: 2014-04-21 02:39:08.948 -0700
OS Version: iOS 7.1 (11D167)
Report Version: 104
Exception Type: EXC_BAD_ACCESS (SIGSEGV)
Exception Subtype: KERN_INVALID_ADDRESS at 0x464bbcd0
Triggered by Thread: 0
could u please let me know the issue
Bryan – what a terrific post. I forwarded this to my entire team. One comment. I usually don’t like to use the word “requirement”. Requirement almost implies that the stakeholders know what they want and what the solution should look like. I like the word “question” better. When you’re answering a question you have carte blanche on shape and delivery of the solution. And you have more of a say in driving behavioral change.
Hi Bryan, this is my first time here as I just stumbled upon your blog from the Andy Kriebel’s one. You have litterarly opened my mind in what BI should be : dedicated to the business needs and not (as too often) an IT service, answering business requirements when they shout loud enough (as Peter McNally brilliantly mentioned above).
I’m a BI consultant working on one of these heavy platform (IBM Cognos) and I see every day how BI department split up with business users, how they struggle to satisfy them or worse, how they have decided to not answer they needs because they do not fit enough in their infrastructure…
Thanks a lot for your insight, hope to see more from you in the future.
Wow, thanks 🙂
How did you go about doing this successfully: Segment your team vertically, not horizontally. ?
The Engineering teams we support were already segmented this way, so it just involved creating smaller pods within our team to directly align with them. It started out as just 1 viz / 1 etl person for each, but each of those hires were very strong. As the teams and scope grew, those first 2 people became team leads and began to have junior people report to them to expand their reach and provide coaching opportunities to the juniors. Now, our team is made up of 5 of these pods where we each are pretty much our own micro BI teams. We still have team meetings and cross functional meetings (for example, all Viz people meet biweekly to share the things we’ve built and tricks we’ve learned), but operate independently. This gives each pod laser focus, 1 customer to answer to, and the ability to become subject matter experts themselves.
About fell off my chair regarding the FFSQL comment because that’s what I find myself doing these days. When I first got into MSTR, “FFSQL” was a five letter word to only use as a last resort, but I have since changed my way of thinking over the last couple of years too. I would rather write SQL than spend time dorking with the VLDB properties (among other things) to get the joins working when using regular MSTR objects. Not sure if we’ll ever get to the point of developing directly in PROD, but I’m sure there is a happy medium somewhere that doesn’t require the standard migration path (DEV/SIT-UAT-PROD).
how can we implement Inter process communication in IOS
we are having a client which is having his own product and wants to integrate to our app so what are the possible ways to integrate the apps.
The best article so far. I am working on an Agile Project which skips the entire process of UAT, Requirement Gathering.
The developers are not allowed to meet and/or interact with the business users/stakeholders for fear of break in Corporate Food Chain. So we develop and wait till we can add additional capabilities with each iterations. But, some of the change requests are pertaining to schematics, which has the tendency to shakeup the foundation of existing application objects every now and then. The result is a bull whip effect on the penultimate deadline of the deliverable.
What would be the best approach to achieve QC/QA initiatives under such circumstances?
The only thing I could think of, is, adopting your point on “Own your data, it’s not just numbers, be its champion”.
Brian, Good article but I have a follow up question for you. With ditching the MSTR schema for freeform SQL or Tableau, how have you found the effort to maintain the SQL in multiple places? We are a MSTR shop and we also have Tableau. We have started to find maintenance & impact analysis for freeform SQL & Tableau to be less than ideal. Just curious if you have found similar things or maybe ideas on how you have gone around them.
Thanks again for the thoughts!
We never had any problems with it. If anything, it was a lot easier than maintaining the MicroStrategy schema which could introduce unexpected issues in various reports by making small changes. One of the choices we made was to optimize for dashboard performance, and to do this we pretty much staged 1 table per dashboard. This way, we could explicitly structure the table to be the exact input format that would be the easiest for the reporting tool to consume while keeping local changes safe from affecting other reports. This allowed us to develop very fast because we didn’t have to do tons of testing when making changes, it made maintenance easier because we could just go into a dashboard/table and make whatever changes we needed right to the SQL, and it made the dashboards really fast because they were mostly select * / filter and didn’t have complex joins (or any joins at all for that matter).
I know it sounds crazy and messy, but it’s worked really well for us.