Advanced Minds

As I am sure you will all recall, Austen spent a lot of time recently looking at performance analysis while trying to migrate data from Version 7 to Version 8: <http://groups.google.com/group/hackystat-dev/browse_thread/thread/b6b... While I was hacking the SensorShell on my way back from Europe, I decided to do some performance analysis of my own.  Recall that Austen's data indicates that a single SensorShell spends the vast majority of its time blocked and waiting for HTTP calls to complete. It thus appears that we could speed up data transmission significantly if we could somehow be doing work in one thread while another thread is performing the HTTP call. To investigate this hypothesis, I implemented a class called MultiSensorShell, which is included in today's developer release of SensorShell. The idea of MultiSensorShell is to allow a client to create N SensorShell instances, and then round robin among them when adding data.  So, there is a parameter called numShells in the constructor which enables you to specify how many SensorShells you want created, and another parameter called batchSize which indicates how many SensorData instances you want added to a given SensorShell in a row before you switch to the next SensorShell. So, if you specify numShells = 10 and batchSize = 100, the first 100 times you call MultiSensorShell.add(SensorData), you will send those 100 SensorData instances to SensorShell 1.  Then, the next 100 SensorData instances will be sent to SensorShell 2, and so forth. After sending data to SensorShell 10, it wraps around and starts sending data to SensorShell 1 again. Now, remember that our goal is for our client to not block while SensorData is being sent.  The way this works is though another parameter to the MultiSensorShell constructor, called autoSendTimeInterval.  This specifies the autoSendTimeInterval for each shell.  For example, if you specify this as 0.1, then each shell will create a thread that automatically invokes its send() method every 10 seconds. That's basically it.  What will happen is that each of these SensorShells will create a separate thread that wakes up periodically and sends their accumulated data.  If the client is adding data to any of the other SensorShells when one SensorShell thread wakes up, then the client will not be blocked.  The only time the client will block is when it tries to add data to a SensorShell when its send() thread is in the midst of sending data. (All of the public methods of SensorShell are synchronized, so you can't add() while send() is executing for any given instance of SensorShell.) The MultiSensorShell has a main method that enables you to play with it and looks like this:  public static void main(String[] args) throws Exception {     long totalData = 50000;     int numShells = 10;     double autoSendTimeInterval = 0.05;     int batchSize = 200;     String host = http://localhost:9876/sensor_base_ ;     String user = This e-mail address is being protected from spam bots, you need JavaScript enabled to view it ;     SensorProperties properties = new SensorProperties(host, user, user);     MultiSensorShell multiShell =       new MultiSensorShell(properties, autoSendTimeInterval, numShells, batchSize);     Date startTime = new Date();     for (int i = 0; i < totalData; i++) {       long time1 = new Date().getTime();       SensorData data = makeSensorData(user); multiShell.add(data);       long time2 = new Date().getTime(); System.out.println( Elapsed millis for add: + (time2 - time1)); }     multiShell.send();     long totalTime = (new Date().getTime() - startTime.getTime());     double timePerData = (double)totalTime / (double)totalData;     System.out.println( Total time: + totalTime + time/data: + timePerData);   } As you can see, in this example, I create 10 SensorShells, send a total of 50,000 SensorData instances, with a batchSize of 200.  The main method prints out how long it takes for each add() to complete in order to help you see when the client is paused and when it isn't. First, note that if we set the batchSize to 250 and the numShells to 1, we essentially get Austen's migration situation.  When we do that, we also duplicate his results