Category: Programming

Windows Home Server: Resource Deconfliction

~ rbonini

As more and more Windows Home Server Add-Ons are introduced, WHS becomes more and more like an appliance than a piece of software (and hardware).

More and more Add-ons mean that we ask more and more of our systems. These demands mean that finite resources have to allocated and shared with the WHS software itself.

DEMigrator.exe comes to mind ( the magic behind folder duplication). Since DeMigrator does not actually have a front end ( short of turning off folder duplication), it is impossible to pause or stop it when its running in favour of something more urgent. Granted we could change our backup window, but this is not always convenient or possible.

What WHS needs is some way of managing resources on a much more granular level than process priorities. By that I mean that WHS makes  a logical guess as to what process(es) need to run now  and what processes are less immediate.

So if I use SageTV to record show x at time y and a defrag ( or other processor intensive program) is scheduled to run at the same time, we need resource deconfliction to kick in and sort it out. We can do this in one of two ways: either throttle back the proccessor intensive process or re schedule it ( if the drive isn’t very fragmented a missed defrag pass wont make much of a difference).

Naturally, we can’t expect this souped up task scheduler to be able to handle every occurrence of every program. this means that WHS would simply notify the offending process(es) of the situation and it would be up to the program to implement a responsible and reasonable strategy to handle that.

If you’ve got a high end system running WHS, this discussion isn’t very dramatic. But between backups, defrags, virus scans, DeMigrator, SageTV  and others ad nauseam  ( even automatic Windows Update needs to be able to safely restart) jockeying for resources, something needs to manage this safely and well.

Essentially, this is bringing WHS closer to the headless system originally envisioned. It would save me a lot of Remoteing in every day.

Before we finish, let take a look at the specs for the WHS systems commercially available from HP, etc to get an idea of exactly what resources are available.

The Microsoft minimum spec is 1Ghz and 512Mb RAM and 1x 70Gb drive.

The recommended spec is 64-bit Compatible Intel Pentium 4, AMD x64 or newer with 512Mb Ram and 2x hard drives with a 300GB primary disk.

  CPU RAM Hard Drive
HP Media Smart AMD 1.8 GHZ 64-bit Sempron 3400+ processor 512Mb 2x 500Gb
Norco DS-520 Intel Celeron M 1GHz 1Gb 1x 250Gb
Piranha Home Server Intel Celeron 430 (1.8GHz, 512KB, Conroe) 1GB 2x250Gb
T2-WHS-A3 Harmony Home Server Intel Celeron 220 1.2GHz 512Mb (1Gb Optional) 1 x 500Gb (1Tb/2Tb Optional)
T7-HSA Harmony Home Server Via C7M “Esther” 1.5Ghz 512Mb (1Gb Optional) 1 x 500Gb (1Tb Optional)
My own homebuilt system (Dell  Poweredge SC440) Intel Celeron D 2.8Ghz 2GB 1x160Gb
1x400Gb
2x750Gb

I think this is a pretty representative sample of the entire range. You can get the reviews on these servers and others from We Got Served Hardware page.

NB. The extra possibilities of multi-core  64 bit machines allowing true concurrent execution are mind boggling.

Visual Studio Install Error 1935 (HRESULT: 0×80070BC9) Fix

~ rbonini ~ 4 Comments

As  I wrote here last week, getting Visual Studio 2008 installed was a bit of a problem for me on my main Desktop PC.

And I couldn’t find a fix anywhere. So since the installation was successful on my laptop ( they are both nearly identical systems) I set about trying to find some difference between them.

I came up with the fact that I’d had Visual Studio 2008 Visual Web Designer Express installed and had uninstalled it before my Visual Studio 2008 Pro install.

So in the finest tradition of Voodoo Troubleshooting I did the following:

  1. Mounted  the Visual Studio Express Editions DVD image available from Microsoft here
  2. Installed Visual C# Express ( it looks as if any edition will do)
  3. Restarted
  4. Uninstalled only Visual Studio C# Express (the runtime prerequisites will also uninstall)
  5. Restarted
  6. Installed Visual Studio 2008 Professional

I’m not quite sure why this works. I put forward the idea that it fixes the registry or the .Net Install ( see my earlier post for details).

Happy Coding 🙂 !

Installing Visual Studio 2008

~ rbonini ~ 1 Comment

UPDATE: I found a fix. See here.

Right. Let get this straight. I’m running Vista Business with Visual Studio 2005 Standard installed (and All the extras – SQL Server etc).

The short version is that Visual Studio 2008  Professional refuses  to install itself. It installed .Net 3.5, Document Explorer 2008 and the Web Authoring Component and then quit at some point while installing Visual Studio itself.

Heres the error log:

Microsoft Visual Studio 2008 Professional Edition – ENU: [2] ERROR:Error 1935.An error occurred during the installation of assembly ‘Microsoft.VC90.DebugCRT,version=”9.0.21022.8″,publicKeyToken= “1fc8b3b9a1e18e3b”,processorArchitecture=”x86″,type=”win32″‘. Please refer to Help and Support for more information. HRESULT: 0x80070BC9.

I’ve no idea what is going on. If you Google search  the Error Code you get this error for VS2005 (or SP1), .Net 2 or SQL Server.  Searching by HRESULT points to this MSDN Forum where the discussion is about a VS2008 Compile problem.

Now here’s the thing. It installs perfectly on my laptop (also running Vista Business with Visual Studio 2005 installed with all the bells and whistles). So the download ( and no, its a perfectly legal copy) is definitely not corrupted.

HELP!!!!!

On Robotic Fish

~ rbonini

I was reading this weeks New Scientist ( the print edition, mind you) and this story  about what the US Navy’s Office of Naval Research is doing caught my eye:

AGILE robotic fish that look like the real thing are being developed to act as government spies.

The article goes onto say that the fish will have cameras and communicate with each other using sonar.

To anyone that has read Michael Crichton’s Prey, this sounds suspiciously like a multi-agent system, albeit one that uses physical agents rather than computer simulated ones.

Wikipeedia has this to say:

The exact nature of the agents is a matter of some controversy. They are sometimes claimed to be autonomous. For example a household floor cleaning robot can be autonomous in that it is dependent on a human operator only to start it up. On the other hand, in practice, all agents are under active human supervision. Furthermore, the more important the activities of the agent are to humans, the more supervision that they receive. In fact, autonomy is seldom desired. Instead interdependent systems are needed.

[…]

MAS systems are also referred to as “self-organized systems” as they tend to find the best solution for their problems “without intervention”.

[…]

The main feature which is achieved when developing MAS systems, if they work, is flexibility, since a MAS system can be added to, modified and reconstructed, without the need for detailed rewriting of the application. These systems also tend to be rapidly self-recovering and failure proof, usually due to the heavy redundancy of components and the self managed features, referred to, above.

Although we’re not likely to see these become evolving, man-eating piranhas, it is something to keep an eye on (if you’ve read the book, you’ll know where I’m coming from).

And it demonstrates a physical application of  this technology that, while the agents are not strictly independent, they are not exactly predictable either.

At least, that is the way I understand it.

Its Been a While – WHS Add in

~ rbonini

I’ve been off ill. Pretty badly in fact, which is why  haven’t posted. or done much programming.

I’m adding the finishing touches to the service – mainly making sure it won’t crash itself or the Server and that it actually does what I intend it to do.

So I’m gradually turning my attention to the scheduled task that will upload files. This is a nightmare mainly because of the large number of possible scenarios. These mainly have to do with updated files. It depends what has changed – the image itself (i.e. after a trip through Photoshop or Elements) or the metadata (keywords, tags, etc).

I’m also getting to grips with the settings and console tabs in terms of what will go where. I tend to try to nail down the user interface first in my software projects so that I know what user interface logic needs to be implemented. No use writing lines of code that, although doing something quite logical and correct, turn out not to be needed.

I’ve yet to forget about this little project- just wish Omar Shashine would release his code for his Send to Smugmug App. Don’t forget to check his site for the latest version.

WHS and SmugMug- Architecting the Add-In

~ rbonini

The structure of an Add-In is important to what we are trying to accomplish here.

I could put all my code as part of a Console Tab and have people manually getting it working.

That would no doubt work, but its kind of lame and falls short of expectations.

So we want an application that reacts in real-time to folder and file changes, but is also able to be scheduled ( say for 3am after a defrag or back-up session).

This requires a number of different application blocks, each an application in its own right.

To see how an Add-In works in code, the WHS Photo Uploader for Flickr is hosted on Codeplex. Unfortunately its a totally different application for a totally different service. But I did get a good look at the Console and Settings Tabs to see what the code would/should look like.

If you want to download and run it, I’d download the WIX Installer XML for Visual Studio 2005 to allow the  Add-in’s installer to be recognised by VS and compiled.

So the lesson is that Console Code is run from the Console (doh!) and any file monitoring code will only work while the console is running.

So we’ll require a service to be running to monitor our gallery folders and record the events to our Master XML file. Hence this application will consists of XML code and our FileWatcher Object. It will need to use the Windows API to allow it to be installed and run  as a service.

Then, we want to schedule our uploads so that they are not taking over your broadband connection. We could use System.Threading to calculate the number of seconds remaining and have the timer call our upload method once the time is reached.

However, we want to keep things as simple as possible. This means using windows’ Scheduled Tasks to call our uploader (an exe file). Scheduling Tasks is for some reason not in the .Net Framework 2.0 so we’ll use a library to wrap all the C++ Interop code. There is a CodeProject article here on it with the library. This also means we can run tasks with a users username and password (i.e “Administrator” and whatever the password is) so tasks will run, in theory, even when no one is logged in.

So we have three applications here, all rather trivial:

  • The Console Tab and Settings Tab with code for usernames/passwords as well as rescheduling uploads and manually starting the uploader. I have the idea of getting album stats and displaying the charts by using the Google Charts API, but that’s for version 1.1.
  • The Service app that will take file and folder change events and writes them to our Master Settings file. This runs independent of any user being logged in.
  • The Uploader that is scheduled to run at such and such a time to execute the events in our Settings file on the server using he SmugMug API. Also runs independent of any users being logged in.

Slowly but surely the Add-in progresses.

WHS and SmugMug – A Word About XML

~ rbonini

XML is a markup language in the same family as HTML. This means you have nested elements in a  structured document.

Its structure makes it ideal to read and write data quite easily and in  a logical format. Since the elements all have names, the documents are usually human readable. This is of course a disadvantage for a proprietary file format.

Reading and writing XML is quite easy with Visual Studio and the .Net Framework. You can use an object called XMLSerializer that will do it automatically, but it depends on you structuring you data for it. It is also great for recurring data. In the example below we are reading in multiple FileEvent object.

The way I prefer doing it is using XMlReader and XMLWriter. Like so:

reader.Read();

 for(int loops =0; loops<count;loops++){
                    FileEvent temp = new FileEvent();
                    reader.ReadStartElement();
                    temp.FilePath=reader.ReadElementString();
                    temp.EventType = reader.ReadElementString();
                    reader.ReadEndElement();
                    mastersettings.Events.Enqueue(temp);

}

reader.Close();

Line for line, the read and write methods of your application is exactly the same as your final XML document, unless you use a loop.

Once you know how to use XmlReader and Writer, its easy to use. Essentially you have to ensure that your elements open and close at the right points it the code to ensure that you get a properly formatted Xml Document.

Second, make sure you are reading the right data type. I use strings as they are quite easy to use, even for dates and such. Most of my errors come from, that.

Third, make sure that you read and write methods read and write the exact same data at the exact same points ( you don’t want to be reading a name when you expect a date, etc).

Fourth, you need to use the WriteStartDocument method to start writing an Xml file and WriteEndDocument for ending one. thing of them as a huge node that encapsulates rest of the document. Just remember, one document per file.

And that’s all there is to it.

More later on how I’m using XML as a data source.

Related Post: WHS and SmugMug – Keeping Track of Files

WHS and SmugMug – Keeping Track of Files

~ rbonini ~ 1 Comment

The question that has to be asked at the beginning of every software project, hobbyist or commercial is: “What are we trying to achieve here?”.

For this project, the aim is to move all the contents of one folder to a gallery on SmugMug – and keep them up-to-date. This means that modifications ( tags, edits, etc), additions and deletions are replicated on the gallery in question.

This requires two things:

  • The program to be aware of what is happening to those folders
  • And for the program to be aware of what has already been uploaded.

Number one requires a FileWatcher Object from System.IO.

The Filewatcher throws an event for creations, deletions  and modifications. This allows us to determine the nature of, say, a create event and decide if its a file or directory that has been created. Based on that we can take other action.

We do this by calling System.IO.Directory.Exists(string path) that returns a boolean. You could use the File.Exists(string path) method as well.

Second, a running tally has to be kept of what is going on. We do that by maintaining a master file in our root directory (say \\Photos\SmugMug)  and child files in each directory. The actual directory may or may not be exposed as a setting via the Console.

The master file is essentially has a queue of file events to execute on the server the next time its syncs. This stops large file  ( i.e. large numbers of files) operations overloading the server (resulting in multiple create events) as Demigrator.exe is working on those files as well. So better  to queue the uploads to take place when the disks are idle. I’m using XML and handwriting the read and write procedures – you’ll understand why at the end of the post.

The second requires a file in each directory with the SmugMug Gallery ID and  a list of each file and its SmugMug ID in that Directory. This allows us quick access to files instead of iterating over the lot and comparing each file to the one on SmugMug.

As far as scheduling the uploader is concerned, I haven’t gone into that in too much detail. I could schedule an exe to run using the task scheduler. Its not a bad way of doing things. Norton Internet Security does just that for its scheduling. That, of course has a disadvantage of spitting up the add-in into a logical front-end and back-end.

And by the way, I’m writing this in Visual Studio 2005 using C#. I’m going to do a second version in C# Express 2008 (to use LINQ-to-XML, mainly) for a comparison.

WHS and Smugmug 1

~ rbonini ~ 2 Comments

The first thing I looked for after signing up for SmugMug was a WHS Add-In since there is one for Flikr. Lo and behold, there isn’t one.

So last night I started writing an application architecture post when it occurred to me that the best way of explaining it was to write a program from scratch.

Enter the SmugMug Add-In idea. So I’ll be doing that in a short series of posts.

First things first, I haven’t decided whether to use Visual Studio 2005 or Express 2008.

Second, the WHS SDK is already in your servers’ Program Files\Windows Home Server folder, so don’t bother looking online for it. You’ll need these two files added as references in Visual Studio ( or Express):

  • HomeServerExt.dll
  • Microsoft.HomeServer.SDK.Interop.v1.dll

Third, the Smugmug API can be downloaded from SmugMug.com. And it looks to be rather straight forward to write code against it (Note – I always say that looking at an API for the first time, then regret it later). And you’ll need an API key from SmugMug. And you’ll need to add the API in as a reference in Visual Studio or Express.

For a quick look at it, Evan Leventhal’s .NET wrapper SmugMug.NET, is a good start.

Fourth there the way the program will work is pretty straight forward :

  • The folder name can be the name of the gallery OR an XML file in the folder will identify which gallery it corresponds to (that is the SmugMug Gallery ID) ala SyncToy.
  • Since I keep far more photos than I post in \\Server\Photos,  it would be a good idea to create a separate folder containing all the gallery folders for syncing.
  • As far as checking the folders themselves, the .Net 2.0 has a FileWatcher object ( or something to that effect).
  • A settings tab will have a list of folders in the gallery folder that will be synced by default unless you uncheck it.
  • As far as a console tab goes, I have no idea – stats, gallery browser, etc.
  • And scheduling of some sort, could do somthing ala SyncToy (and no, I’m not french).

I’m writing this primarily for myself  as I’m fed up with having to Remote in and use the Send To SmugMug shell app but it will be packaged neatly and on Codeplex once its finished, I imagine.