Moving Cubes

This is a hack, so proceed at your own risk!  I haven’t tested this or used it in a production environment, so things may break/crash/catch on fire.

Say that you have built a really nice dashboard, deployed it to production, and everyone loves it.  This dashboard is based on a cube that takes an hour to load.  Now you’ve made modifications to the dashboard and cube in Dev and you’ve just migrated it to Prod.  However, the cube instance is invalid because the data doesn’t match effectively taking your beautiful dashboard offline for an hour while the cube refreshes.  If only there was a way to move the cube’s data from Dev to Prod …

Intelligent Cube File Storage
When an Intelligent Cube is loaded, it is simultaneously stored on disk and in memory.  This is so that the IServer can have the option of unloading old cubes if it runs out of memory or reloading cubes quickly if the server crashes or restarts.

The location that files are stored is configured in your project settings.  From there, they are in a folder that contains the server definition name (not necessarily the server name you connect to, but whatever was defined at the time of configuration!) and some random letters and numbers.  Inside this folder are two critical components of the cube, the .cube file and the _info.cube file.  The first is the actual data and the second is metadata for the cube.  There are other .idx files, but it appears that you don’t need them (see the opening two sentences about warnings).  These files share the same ID in the name and that ID is generated at the time the cube is loaded.  If you delete and reload the cube, it will change, but if you simply unload and reload the cube, it won’t change.

Exploiting the File Storage
First, the requirements:

  1. You have to have the same project in both locations, ie, a project that you can migrate between.
  2. The cube has to already exist in both places and be loaded in both places.  There doesn’t seem to be any way to get the IServer to load a cube cold from storage, only hot (I’ll explain that in a second).
Step 1 – Open the Cube Monitor, right click on the cube you want to move, and choose Quick View.  One of the fields is File Name which gives you the full location and file name of the cube.  Locate the .cube and _info.cube files from source (ie, Dev) and target (ie, Prod).  They will have different IDs.
Step 2 – Copy the files from source to target, but rename the source file to match the file name of the target.  Effectively, you’re overwriting the target files with the source files (and renaming).
Step 3 – In the Cube Monitor in the target (ie, Prod), right click on the cube and choose Unload from memory quickly followed by Load in memory.  It’s done like magic!
Wait, What?
So, basically what happens is when you unload a cube from memory, it doesn’t write anything to disk.  When you load it into memory, it then reads those files.  This means that it’s safe for us to swap out the files underneath the system without affecting what’s in memory.  In fact, you can delete those files and the IServer won’t notice until it restarts.  Once we’ve swapped the files, it’s them a simple matter of telling the IServer to reload them via the Unload/Reload commands.  After this is all finished, you only had about a 1 second outage instead of 1 hour, or however long it takes to load your cube.
Fail Over Usage
Even if you don’t need this hack for migrations, it’s actually really useful for fail overs.  Say that you have a disaster recovery site with a passive IServer node that you’ll turn on in the event your main data center goes down.  If you store your cubes on a SAN that is accessible/redundant to the recovery site, then a simple script could allow you to instantly load every cube in your system from disk instead of needing to reload them from the database.
This hack probably also works for report caches, but I don’t personally use those anymore.  I don’t see why it wouldn’t, since cache storage works the same way as cubes.
Even if you don’t find this particularly useful (I suspect I’ll use it very rarely, but handy when I do), it’s still something kind of neat!

You may also like...

13 Responses

  1. Alex Belyaev says:

    Thanks for an idea, I’ll try to do so for testing.

  2. Scott Weaver says:

    I had tried this before without success and just moved on. I just did a quick test and encountered the following error:

    2013-05-04 12:51:14.244-05:00 [HOST:*****][PID:1264][THR:2340][Object Server][Error][0x800411D3] The project or configuration has not been loaded in the Content Server; it cannot be accessed
    2013-05-04 12:51:14.244-05:00 [HOST:*****][PID:1264][THR:2340][COM][Error][0x800645E4] Cube instance 82C969524B4C36D3A799DE825DF092DA cannot be loaded into memory due to error -2147216941.

    I’m very encouraged that someone was able to get this to work, however. We have an actual need to do something like this on a nightly basis.

    • Bryan says:

      We didn’t encounter any errors, but found that it must first be loaded in the target server so that it was in the cube monitor. Also, we had to move the .cube and _info.cube files together (it didn’t seem to matter what we did with the .idx’s). We got it to work in both Windows and Linux environments.

    • Scott Weaver says:

      I tried it again and received the same error. I remembered what it was that caused me to give up on this idea the first time I tried it; When I looked at the _info.cube file of the source file in a text editor the path and filename were stored in the file. Is this the case with the cubes you are moving as well? Maybe I’m missing something. It certainly wouldn’t be the first time 🙂 The steps I followed are:
      1. ensuring that both the source and target cubes are loaded.
      2. copying the .cube and _info.cube files to a location on the target.
      3. renaming the copied files to match the filenames of the target cubes.
      4. copying the renamed files into the target directory
      5. unloading the target cubes.
      6. loading the target cubes (this is where I get the error.)

  3. Scott Weaver says:

    This comment has been removed by the author.

  4. Len OToole says:

    Thanks Bryan. I’ve had a need for this multiple times over the last few months. I knew that it MUST be possible, just didn’t have the time to dig deeper. Not surprised that you’d be digging around in this fun stuff. I’ll give it a go and see how it works on 931.

  5. GAURAV says:

    Hi Bryan. I too tried the above trick but got the same error as the one received by Scott. 🙁

  6. Vikas says:

    Hi All,

    I am novice in MSTR and need to desgin cube for my client.
    My database size is neary about 50- 70 GB and cube generated out from this DB would would be more than 256 MB size or more than that. Right now Cube has filter on it and in filter i have specifed user id to fetch data only for that user id.
    Now we are planning to extend this solution to other user and hence need to fetch data corrosponding to all user ids in cube.
    1st Approach: I can fetch all user ids data in cube and corrosponding to each user id login, show filtered cube data to user.
    Cons of 1st approach: It makes cube of very large size more than 2 – 4 GB size and hence can break system.Secondly i am not sure weather this approach will work and if yes how to do it practically.

    2nd Approach: I can create seprate individual cube for each user.
    Cons of 2nd approach: Tere are mor ethan 100 user , creating separate cube fro each user would not be considered a practical solution .

    My question is how to design such cube ?

    Thanks in Advance,

  7. Bruno says:

    Hi Bryan,

    Wondering if you remember, while testing this in Linux, have you tried to use symbolic links instead of copying the files around?


    • bryan says:

      I didn’t try that (and never did use this in production), but why would you want to do that?

      • Bruno says:

        Our business is working in a way where they require the same user to have multiple profiles with conflicting rules in terms of data security (long story…). One of the options we are looking at includes having multiple similar projects, with the users having different security filters applied on each of them.

        Using links to a shared location instead of having multiple files around to use on the different projects might be a cleaner implementation but our current environment is all Windows based and I couldn’t get this to work with shortcuts (probably because the .lnk extension that is added to the file).

        This is not going to be a decisive factor that will make us move everything to Linux but it was just something else to consider.

        Thanks for sharing your test results.

  8. Joel says:

    As some of the commenters have noticed – this won’t work if you have different Project GUIDs in Dev and Test/Prod. The reason is that the cube instance includes some data structures that reference the project and when the GUID is different it will cause issues trying to load the cube.

    But since 9.4.1 Hotfix 5 or so MicroStrategy has included a tool to help make this simpler. Here’s what it looks like on Linux:
    [mstradm@vm bin]$ ./mstrcacheparser
    This utility parses cube, cache, and inbox info files, and also merges cube index files.

    Usage is as follows:
    mstrcacheparser -c:cache_directory -u:cube_directory -r:rwd_directory -i:inbox_directory -o:out_file -v:inbox_version -m:cache_options

    cache_directory is the full path to the report cache files
    cube_directory is the full path to the Intelligent Cube files
    rwd_directory is the full path to the document cache files
    inbox_directory is the full path to the inbox files

    If there is no option “-o”, parsed result will go to output.txt

    -v:preorion indicates those inbox files were created by pre-9.0.0 Intelligence Server, otherwise they were created by Intelligence Server version 9.0.0 or newer. This option is required only when parsing pre-9.0.0 inbox files.
    -m:lkup parses lkup index files only
    -m:pool parses pool index files only
    -m:info parses individual info files only
    -m:merge merges cube index files from multiple directories into one. -u must specify input directories separated by a semicolon, -o must specify a directory other than any of the input directories

  9. Joel says:

    What you want is the -m:merge option. So:
    mstrcacheparser -m:merge -u:cube_directory_dev;cube_directory_test_temp -o:cube_directory_test_final

Leave a Reply

Your email address will not be published. Required fields are marked *