When connecting to Microsoft Analytics Platform System (APS) from SQL Server Data Tools for Visual Studio 2013

I like to keep it simple so I won’t dedicate much space and time to describing how baffling and annoying this was to solve.

OK, that was a lie; I will have a quick rant.
It seems that the “reward” for keeping it Microsoft (i.e.: building your data warehouse on Microsoft technologies instead of linux/open source alternatives) is hours of installation/configuration time (and this is just to connect from the client, I am not ranting about setting up a server here), and an updated (read: inferior) user interface (there are a lot of annoying differences between keyboard shortcuts for SQL Server Studio and SQL Server Data Tools for Visual Studio – I’m sure this could have been avoided).

Microsoft: you can do better (and you know it).

Anyway, here’s the error message:

The ‘ToolboxPackage’ package did not load correctly.

The problem may have been caused by a configuration change or by the installation of another extension. You can get more information by examining the file ‘C:\Users\$user\AppData\Roaming\Microsoft\VisualStudio\12.0\ActivityLog.xml’.

 

Checking said activity log reveals this:

<entry>
<record>956</record>
<time>2016/01/14 17:17:34.102</time>
<type>Error</type>
<source>VisualStudio</source>
<description>No InprocServer32 registered for package [Microsoft.VisualStudio.Toolbox.ToolboxPackage]</description>
<guid>{6F453E1A-31DF-4CD6-AF8B-72AE242272B9}</guid>
<errorinfo></errorinfo>
<path>%windir%\system32\mscoree.dll</path>
</entry>

Cue lots of googling…

I tried this, this, and this before I found this – which did the job.

The fix was to issue this at the command prompt:

devenv.exe /resetuserdata

 

 

It seems that my user profile was corrupted, resetting my user profile made the error message go away, and left me free to do what I actually wanted to do with APS.

 

 

 

 

Advertisements

When writeBatch to HBaseTable generates OOM (Out Of Memory exception)

Whenever I start to work with a new technology stack I spend the first few years knee-deep in doubt.

Not “why isn’t this thing working?” doubt. That kind of doubt comes after a few years of experience, when I’m sure that what I am trying to do should work.
The first few years are full of “I wonder what this thing is doing?” doubt, or “I wonder if I’m using this thing in the right way?” doubt.

 

Today, a spark job (that batched data into a Hbase table) threw an “Out of memory” exception after inserting about 30% of the desired payload.
The knee-jerk response was “throw some more hardware at it”, but after a bit of tinkering a few discoveries were made: the most important of these was that the HTable class does some interesting things internally with threads – it appears to create a thread for each HTable.put() — and that said threads are not GC’d until you call the HTable.close() method.

 

Today’s solution was to get a BufferedMutator for the Hbase table and to explicitly call .flush() and .close() after every write.

	public void writeBatch(List<Put> puts) throws IOException {
		final BufferedMutator bufferedMutator = tableConnection.getBufferedMutator(tableName);
		bufferedMutator.mutate(puts);
		bufferedMutator.flush();
		bufferedMutator.close();
	}

This probably is not the correct way to do this, but it delivered the expected result (comment if you can see something glaringly wrong).

 

When selenium webdriver js / protractor won’t click the div that you want it to click

Somebody. Please! Sit down and dedicate as much of your life as you can to improving the documentation for protractor (and while you’re at it you can do selenium webdriver js as well).

In the test that I want to write I have a div, and when I click it a menu (another div that was previously hidden from view) is displayed, at which point I move the mouse to an option in the menu and I click said option.

Typical everyday usage, right? Should be relatively easy to achieve with protractor/WebDriver, right?
Wrong, because it isn’t relatively easy to achieve. It is, in fact, life threateningly difficult to achieve this with protractor/WebDriver.

Here is the (hideous, and clearly not optimal) way in which I’ve managed to get this done.

        it('should show a user config window if user name is pressed in the menu', function() {
            var menuButton = element(by.css("fooBar-menu-bar div.options"));

            var userWindow;

            browser.driver.wait(function(){
                return menuButton.click().then(function(){
                    return browser.waitForAngular().then(function(){
                        return browser.wait(function(){
                            return element(by.id("MenuOptionsContainer")).isPresent().then(function(flag){
                                if (flag === true){
                                    var item = element(by.id("MenuOptionUser"));
                                    browser.actions().mouseMove(item).perform();
                                    item.click();
                                    return browser.wait(function() {
                                        return browser.isElementPresent(by.tagName("fooBar-window")).then(function(flag){
                                            if(flag === true){
                                                userWindow = element(by.tagName("fooBar-window"));
                                                expect(userWindow.isPresent()).toBe(true);
                                                return flag;
                                            } else {
                                                return false;
                                            }
                                        });
                                    });
                                }
                                else{
                                    return false;
                                }
                            });
                        });
                    });
                });
            });
        });

If you’ve read this far then you’re probably trying to get this to work, too. Or, even better you may have found a way to make it work that is far less awful than the above.
Either way I hope you’ll leave a comment.

Linux (Bash) – Copying very big things from one server to another across a flaky connection

Sometimes a tester needs to go to great lengths to reproduce a failure or compare behaviour … or just to have the slightest chance of understanding what the hell is going on .
Earlier today I had to copy a very big file (that contained the only known data set capable of reproducing the failure I’m investigating) from one server to another. I work behind a proxy server, and there’s also a second, independent proxy server and a firewall inbetween the two servers in question – as a result my initial attempts with SCP and RSYNC all ended in failure (due to connections being lost/sessions being dropped mid-transfer).

This is the command that got the job done:

nohup scp originalFile.zip user@host:path/destinationFile.zip &

More on the use of nohup and &:
http://www.cyberciti.biz/tips/nohup-execute-commands-after-you-exit-from-a-shell-prompt.html

Python 2.6 – Subclassing unittest.TestCase

“I need to write a lot of system and component integration tests. Many of them will involve hitting the database, getting a result set and transforming it into XML/JSON/something else. I´ll need to create some flat files which contain expected results for these kind of tests. It would help me immensly if I had some utility classes to help me out with the reading/parsing/transforming of said utility functions.”
The above is what I should have thought to myself a few weeks back. Naturally, I didn’t. So I began creating test cases with their own utility functions included. Shortly afterwards this way of doing things revealed itself to be silly, and bad. So I went through several rounds of refactoring, and ended up with some utility classes.
Now I want to subclass unittest.TestCase and include all my utility classes in every test case I write.

First attempt:

Code example will go here if I’m ever able to locate it (I started this post a year ago, got distracted and then forgot about it. The code examples I had prepared at the time are now long gone…)

Resulting error:

method __init__ takes 2 parameters only 1 provided

eh? …
Cue head scratching session.
Followed by a bit of googling. Didn’t quite find what I was looking for, though.
http://www.velocityreviews.com/forums/t737111-subclassing-unittest-testcase.html
http://stackoverflow.com/questions/8964461/subclassing-python-unittest-testcase-calling-the-same-main

http://stackoverflow.com/questions/1323455/python-unit-test-with-base-and-sub-class

This last solution seemed great on paper, but multiple inheritance and calling functions in (one of) the super class(es) proved to be bafflingly complicated. I’m using jython/python 2.6, apparently multiple inheritance has been revamped in python 3.0 (but I’m stuck with 2.6 so I’ll never know).

Cue second head scratching session.

I decide to enlist the help of the resident genius on the project and he came up with the following snippet (took him a fair few tries though, which is why I’ve deemed it worthy of being posted here):

from my.testUtilities import DBMock
from my.testUtilities import JSONHelper
import unittest

class MyTest(unittest.TestCase):
    def __init__(self, methodName='runTest'):
        unittest.TestCase.__init__(self, methodName)
        self.DBMock = DBMOCK
        self.JSONHELPER = JSONHELPER

Job done.

Selenium Web Driver Firefox KeyError: self.session_id = response[‘sessionId’]

Had a nice little head-to-head with this issue yesterday.

As with all things Selenium Web Driver: there are many suggested solutions on the pages of the many forums and wikis that discuss the project and it’s hard to determine which, if any, are suitable for the problem at hand.

This solved it for me: 
http://stackoverflow.com/a/17545189/375996