Moving Cubes
This is a hack, so proceed at your own risk! I haven’t tested this or used it in a production environment, so things may break/crash/catch on fire.
Say that you have built a really nice dashboard, deployed it to production, and everyone loves it. This dashboard is based on a cube that takes an hour to load. Now you’ve made modifications to the dashboard and cube in Dev and you’ve just migrated it to Prod. However, the cube instance is invalid because the data doesn’t match effectively taking your beautiful dashboard offline for an hour while the cube refreshes. If only there was a way to move the cube’s data from Dev to Prod …
Intelligent Cube File Storage
When an Intelligent Cube is loaded, it is simultaneously stored on disk and in memory. This is so that the IServer can have the option of unloading old cubes if it runs out of memory or reloading cubes quickly if the server crashes or restarts.
The location that files are stored is configured in your project settings. From there, they are in a folder that contains the server definition name (not necessarily the server name you connect to, but whatever was defined at the time of configuration!) and some random letters and numbers. Inside this folder are two critical components of the cube, the .cube file and the _info.cube file. The first is the actual data and the second is metadata for the cube. There are other .idx files, but it appears that you don’t need them (see the opening two sentences about warnings). These files share the same ID in the name and that ID is generated at the time the cube is loaded. If you delete and reload the cube, it will change, but if you simply unload and reload the cube, it won’t change.
Exploiting the File Storage
First, the requirements:
- You have to have the same project in both locations, ie, a project that you can migrate between.
- The cube has to already exist in both places and be loaded in both places. There doesn’t seem to be any way to get the IServer to load a cube cold from storage, only hot (I’ll explain that in a second).
Thanks for an idea, I’ll try to do so for testing.
I had tried this before without success and just moved on. I just did a quick test and encountered the following error:
2013-05-04 12:51:14.244-05:00 [HOST:*****][PID:1264][THR:2340][Object Server][Error][0x800411D3] The project or configuration has not been loaded in the Content Server; it cannot be accessed
2013-05-04 12:51:14.244-05:00 [HOST:*****][PID:1264][THR:2340][COM][Error][0x800645E4] Cube instance 82C969524B4C36D3A799DE825DF092DA cannot be loaded into memory due to error -2147216941.
I’m very encouraged that someone was able to get this to work, however. We have an actual need to do something like this on a nightly basis.
We didn’t encounter any errors, but found that it must first be loaded in the target server so that it was in the cube monitor. Also, we had to move the .cube and _info.cube files together (it didn’t seem to matter what we did with the .idx’s). We got it to work in both Windows and Linux environments.
I tried it again and received the same error. I remembered what it was that caused me to give up on this idea the first time I tried it; When I looked at the _info.cube file of the source file in a text editor the path and filename were stored in the file. Is this the case with the cubes you are moving as well? Maybe I’m missing something. It certainly wouldn’t be the first time 🙂 The steps I followed are:
1. ensuring that both the source and target cubes are loaded.
2. copying the .cube and _info.cube files to a location on the target.
3. renaming the copied files to match the filenames of the target cubes.
4. copying the renamed files into the target directory
5. unloading the target cubes.
6. loading the target cubes (this is where I get the error.)
This comment has been removed by the author.
Thanks Bryan. I’ve had a need for this multiple times over the last few months. I knew that it MUST be possible, just didn’t have the time to dig deeper. Not surprised that you’d be digging around in this fun stuff. I’ll give it a go and see how it works on 931.
Hi Bryan. I too tried the above trick but got the same error as the one received by Scott. 🙁
Hi All,
I am novice in MSTR and need to desgin cube for my client.
My database size is neary about 50- 70 GB and cube generated out from this DB would would be more than 256 MB size or more than that. Right now Cube has filter on it and in filter i have specifed user id to fetch data only for that user id.
Now we are planning to extend this solution to other user and hence need to fetch data corrosponding to all user ids in cube.
1st Approach: I can fetch all user ids data in cube and corrosponding to each user id login, show filtered cube data to user.
Cons of 1st approach: It makes cube of very large size more than 2 – 4 GB size and hence can break system.Secondly i am not sure weather this approach will work and if yes how to do it practically.
2nd Approach: I can create seprate individual cube for each user.
Cons of 2nd approach: Tere are mor ethan 100 user , creating separate cube fro each user would not be considered a practical solution .
My question is how to design such cube ?
Thanks in Advance,
Vikas
Hi Bryan,
Wondering if you remember, while testing this in Linux, have you tried to use symbolic links instead of copying the files around?
Thanks!
I didn’t try that (and never did use this in production), but why would you want to do that?
Our business is working in a way where they require the same user to have multiple profiles with conflicting rules in terms of data security (long story…). One of the options we are looking at includes having multiple similar projects, with the users having different security filters applied on each of them.
Using links to a shared location instead of having multiple files around to use on the different projects might be a cleaner implementation but our current environment is all Windows based and I couldn’t get this to work with shortcuts (probably because the .lnk extension that is added to the file).
This is not going to be a decisive factor that will make us move everything to Linux but it was just something else to consider.
Thanks for sharing your test results.
As some of the commenters have noticed – this won’t work if you have different Project GUIDs in Dev and Test/Prod. The reason is that the cube instance includes some data structures that reference the project and when the GUID is different it will cause issues trying to load the cube.
But since 9.4.1 Hotfix 5 or so MicroStrategy has included a tool to help make this simpler. Here’s what it looks like on Linux:
[mstradm@vm bin]$ ./mstrcacheparser
This utility parses cube, cache, and inbox info files, and also merges cube index files.
Usage is as follows:
mstrcacheparser -c:cache_directory -u:cube_directory -r:rwd_directory -i:inbox_directory -o:out_file -v:inbox_version -m:cache_options
cache_directory is the full path to the report cache files
cube_directory is the full path to the Intelligent Cube files
rwd_directory is the full path to the document cache files
inbox_directory is the full path to the inbox files
If there is no option “-o”, parsed result will go to output.txt
-v:preorion indicates those inbox files were created by pre-9.0.0 Intelligence Server, otherwise they were created by Intelligence Server version 9.0.0 or newer. This option is required only when parsing pre-9.0.0 inbox files.
-m:lkup parses lkup index files only
-m:pool parses pool index files only
-m:info parses individual info files only
-m:merge merges cube index files from multiple directories into one. -u must specify input directories separated by a semicolon, -o must specify a directory other than any of the input directories
What you want is the -m:merge option. So:
mstrcacheparser -m:merge -u:cube_directory_dev;cube_directory_test_temp -o:cube_directory_test_final