Subversion Source Control

This post has been bouncing around my head for sometime since finding the time to write has been a little hard.

The fact is that the whole Subversion vs Git vs TFS vs [insert preferred system here] debate can get to the point where its quasi-religious (as are numerous other tech debates – windows vs mac being one of the notable ones).

Now I, bad developer that I am, have only recently started using Subversion. The need to sync source code between my desktop and my laptop outgrew the copy/paste via Windows Home Server share approach.

The fact that Subversion is free is also a big deal maker ( compared to, say Team Foundation Server).

I have been using it on another free product, namely Netbeans 6.01 for my Java development. Its nothing major, only university projects. 

The fact is that Netbeans comes with Subversion and CVN support baked into the IDE. Additionally, Netbeans uses its own Local History feature to keep track of your files wether you’re using formal source control or not. Every time you build your project it makes a commit to your Local History. This means that one has both Local history for the small changes and Subversion for the big changes.

The fact is that having it baked into the IDE makes all the difference – it allows one to interact with the code in revisions directly. Let me explain. One can make line-by-line rollbacks from your previous versions wether you are using Local history or Subversion or both. Powerfully, the rollbacks count as changes to your code and are committed right back to the database.

(I plan to do a post on the value of a good IDE soon as well)

image

In the (Subversion) case above, the local copy is out of date (on the right) and the remote changes (on the left) are shown in blue. The changes were made on my laptop, committed and I’m comparing them against the desktop copy of the files.

By clicking on the blue arrow one can insert the changes into the local copy with out inserting all the changes. The same applies for replacements (in cases where the lines have been modified rather than outright replaced).

The green area in the local copy no longer exists and by clicking on the red cross one can delete the highlighted lines.

Local History works in exactly the same way.

This is a small example and one needs to actually use it to understand the power of the concept.

Now, the fact is that the vast majority of my coding gets done in Visual Studio. And Visual Studio , the TFS edition aside, does not support source control. Period.

However, VisualSVN actually has an Add-In for Visual Studio that brings this functionality into the IDE. The problem is  that the Add-In isn’t free. Its $49 per licence. Personal licenses are restricted to one per order ( but not corporate licenses) and open source projects qualify for free licenses. And, no, I haven’t yet decided whether to get a license.

I agree with Jeff Atwood that software such as Firefox should take the most popular Add-Ins and fold them in the main code base. Which gives rise to the question that if Source Control is so important and popular (is it for non-TFS Visual Studio users????), why doesn’t Microsoft add it in?

I mean its only Subversion. Surely Microsoft must have some pretty good selling points related to why TFS is, like, totally better than Subversion ( ๐Ÿ˜‰ ). That takes care of the “They already have source control” pundits.

Source Control is simply good programming practice and if Microsoft is serious about attracting people to its platform (and the freely available express editions shows that they are somewhat serious), they should provide it.

Contrast Visual Studio to Netbeans and that’s enough said on the subject.

Now the easiest way to install Subversion is to head over to Jeff’s blog post about it. If you don’t subscribe to Jeff’s blog, i suggest that you do – its invaluable.

I went the VisualSVN server route which set everything up with the addition of  the ability to browse the server using a web browser (it does this by using Apache). To avoid Apache conflicting with Windows Home Server, its running on a Virtual Server virtual machine ( that image runs a few other odds and ends). Setting up Virtual Server on WHS is itself the subject of another blog post. It took me literally 2 minutes.

Before letting you go, Scott Hanselman has a great podcast on Subversion vs Git for Source Control and it comes highly recommended.

Windows Home Server: Resource Deconfliction

As more and more Windows Home Server Add-Ons are introduced, WHS becomes more and more like an appliance than a piece of software (and hardware).

More and more Add-ons mean that we ask more and more of our systems. These demands mean that finite resources have to allocated and shared with the WHS software itself.

DEMigrator.exe comes to mind ( the magic behind folder duplication). Since DeMigrator does not actually have a front end ( short of turning off folder duplication), it is impossible to pause or stop it when its running in favour of something more urgent. Granted we could change our backup window, but this is not always convenient or possible.

What WHS needs is some way of managing resources on a much more granular level than process priorities. By that I mean that WHS makes  a logical guess as to what process(es) need to run now  and what processes are less immediate.

So if I use SageTV to record show x at time y and a defrag ( or other processor intensive program) is scheduled to run at the same time, we need resource deconfliction to kick in and sort it out. We can do this in one of two ways: either throttle back the proccessor intensive process or re schedule it ( if the drive isn’t very fragmented a missed defrag pass wont make much of a difference).

Naturally, we can’t expect this souped up task scheduler to be able to handle every occurrence of every program. this means that WHS would simply notify the offending process(es) of the situation and it would be up to the program to implement a responsible and reasonable strategy to handle that.

If you’ve got a high end system running WHS, this discussion isn’t very dramatic. But between backups, defrags, virus scans, DeMigrator, SageTV  and others ad nauseam  ( even automatic Windows Update needs to be able to safely restart) jockeying for resources, something needs to manage this safely and well.

Essentially, this is bringing WHS closer to the headless system originally envisioned. It would save me a lot of Remoteing in every day.

Before we finish, let take a look at the specs for the WHS systems commercially available from HP, etc to get an idea of exactly what resources are available.

The Microsoft minimum spec is 1Ghz and 512Mb RAM and 1x 70Gb drive.

The recommended spec is 64-bit Compatible Intel Pentium 4, AMD x64 or newer with 512Mb Ram and 2x hard drives with a 300GB primary disk.

  CPU RAM Hard Drive
HP Media Smart AMD 1.8 GHZ 64-bit Sempron 3400+ processor 512Mb 2x 500Gb
Norco DS-520 Intel Celeron M 1GHz 1Gb 1x 250Gb
Piranha Home Server Intel Celeron 430 (1.8GHz, 512KB, Conroe) 1GB 2x250Gb
T2-WHS-A3 Harmony Home Server Intel Celeron 220 1.2GHz 512Mb (1Gb Optional) 1 x 500Gb (1Tb/2Tb Optional)
T7-HSA Harmony Home Server Via C7M โ€œEstherโ€ 1.5Ghz 512Mb (1Gb Optional) 1 x 500Gb (1Tb Optional)
My own homebuilt system (Dell  Poweredge SC440) Intel Celeron D 2.8Ghz 2GB 1x160Gb
1x400Gb
2x750Gb

I think this is a pretty representative sample of the entire range. You can get the reviews on these servers and others from We Got Served Hardware page.

NB. The extra possibilities of multi-core  64 bit machines allowing true concurrent execution are mind boggling.

Hardware Upgrades- Vista and WHS

I just upgraded my main PC from XP to Vista  Business. And stuck a new graphics card in.

Before we discuss the Windows Experience Index, the only down side of the upgrade is that there are no Vista drivers for the front ports ( USB, SD, etc)  or the built in sound card from Compaq. None. Their driver downloads page for the model ( SR1629 UK) essentially says “best of luck” for those upgrading to Vista.

In fact, I suspect that the model number of my PC is different to that on the page as the picture does not show the front card reader and the drivers don’t show up for it. Any Suggestions???

This is extremely inconvenient, as you might imagine.I’ve lost 3 USB ports and  a 9-in one card reader. The sound issue was fixed by buying a cheap SoundBlaster Creative card.

I bought a  Radeon HD 3650 PCI- Express card.  512MB of GDDR3 RAM (that has a 1.73GHZ clock). A 790Mhz engine clock. Crossfire X support as well as full HD. The graphics that come out of this thing are amazing.

I bought it mainly to play Flight Simulator X on it and it rocks ( though I’m still adjusting the settings to get the best combination of  graphics and speed/playability).

Here’s the System information page with the Windows Experience Index:

image

And here’s the breakdown:

image

I must say I was expecting a dramatic improvement, but not by THAT much. Can’t remember what the original score was before I put the card in, but it was pretty dramatic.

There is a pretty in-depth discussion of the Index on the Vista Team blog here.

Moving on.

My Windows Home Server has been screaming for the last week that the USB drive had failed ( its connected and turned on and has been tried on multiple USB ports on multiple PCs). I’m not sure precisely what happened but I’m a hard drive short and have 200Gbs of space left.

image

So I’ve ordered two Samsung SpinPoint F1 750GB SATA-II drives. One to replace the failed USB hard drive and the other to replace the aging 80Gb IDE drive. Since there is the hard drive replication feature, I’ve gone down the path of more drives rather than higher drive capacity.  This should take my total drive capacity to about 2 TB, which is plenty. I go through hard drive space rather fast, mainly cause I use SageTv to record Tv and virtual server to run the occasional VM ( I have a 44 GB VM, to give you an idea of size).

I got all this great hardware from a UK company called Overclockers UK. Great customer service. They have no problem dealing with returned items (I had to return a AGP card because my system was PCI-E). They are quite reasonably priced and have some really amazing specials quite often.

My hard drives were shipped 40 minutes after my confirmation arrived in my inbox. And that really is amazing service. If i need hard ware, they’re my first port of call. Well done guys!!

Its Been a While – WHS Add in

I’ve been off ill. Pretty badly in fact, which is why  haven’t posted. or done much programming.

I’m adding the finishing touches to the service – mainly making sure it won’t crash itself or the Server and that it actually does what I intend it to do.

So I’m gradually turning my attention to the scheduled task that will upload files. This is a nightmare mainly because of the large number of possible scenarios. These mainly have to do with updated files. It depends what has changed – the image itself (i.e. after a trip through Photoshop or Elements) or the metadata (keywords, tags, etc).

I’m also getting to grips with the settings and console tabs in terms of what will go where. I tend to try to nail down the user interface first in my software projects so that I know what user interface logic needs to be implemented. No use writing lines of code that, although doing something quite logical and correct, turn out not to be needed.

I’ve yet to forget about this little project- just wish Omar Shashine would release his code for his Send to Smugmug App. Don’t forget to check his site for the latest version.

WHS and SmugMug- Architecting the Add-In

The structure of an Add-In is important to what we are trying to accomplish here.

I could put all my code as part of a Console Tab and have people manually getting it working.

That would no doubt work, but its kind of lame and falls short of expectations.

So we want an application that reacts in real-time to folder and file changes, but is also able to be scheduled ( say for 3am after a defrag or back-up session).

This requires a number of different application blocks, each an application in its own right.

To see how an Add-In works in code, the WHS Photo Uploader for Flickr is hosted on Codeplex. Unfortunately its a totally different application for a totally different service. But I did get a good look at the Console and Settings Tabs to see what the code would/should look like.

If you want to download and run it, I’d download the WIX Installer XML for Visual Studio 2005 to allow the  Add-in’s installer to be recognised by VS and compiled.

So the lesson is that Console Code is run from the Console (doh!) and any file monitoring code will only work while the console is running.

So we’ll require a service to be running to monitor our gallery folders and record the events to our Master XML file. Hence this application will consists of XML code and our FileWatcher Object. It will need to use the Windows API to allow it to be installed and run  as a service.

Then, we want to schedule our uploads so that they are not taking over your broadband connection. We could use System.Threading to calculate the number of seconds remaining and have the timer call our upload method once the time is reached.

However, we want to keep things as simple as possible. This means using windows’ Scheduled Tasks to call our uploader (an exe file). Scheduling Tasks is for some reason not in the .Net Framework 2.0 so we’ll use a library to wrap all the C++ Interop code. There is a CodeProject article here on it with the library. This also means we can run tasks with a users username and password (i.e “Administrator” and whatever the password is) so tasks will run, in theory, even when no one is logged in.

So we have three applications here, all rather trivial:

  • The Console Tab and Settings Tab with code for usernames/passwords as well as rescheduling uploads and manually starting the uploader. I have the idea of getting album stats and displaying the charts by using the Google Charts API, but that’s for version 1.1.
  • The Service app that will take file and folder change events and writes them to our Master Settings file. This runs independent of any user being logged in.
  • The Uploader that is scheduled to run at such and such a time to execute the events in our Settings file on the server using he SmugMug API. Also runs independent of any users being logged in.

Slowly but surely the Add-in progresses.

WHS and SmugMug – A Word About XML

XML is a markup language in the same family as HTML. This means you have nested elements in a  structured document.

Its structure makes it ideal to read and write data quite easily and in  a logical format. Since the elements all have names, the documents are usually human readable. This is of course a disadvantage for a proprietary file format.

Reading and writing XML is quite easy with Visual Studio and the .Net Framework. You can use an object called XMLSerializer that will do it automatically, but it depends on you structuring you data for it. It is also great for recurring data. In the example below we are reading in multiple FileEvent object.

The way I prefer doing it is using XMlReader and XMLWriter. Like so:

reader.Read();

 for(int loops =0; loops<count;loops++){
                    FileEvent temp = new FileEvent();
                    reader.ReadStartElement();
                    temp.FilePath=reader.ReadElementString();
                    temp.EventType = reader.ReadElementString();
                    reader.ReadEndElement();
                    mastersettings.Events.Enqueue(temp);

}

reader.Close(); 

Line for line, the read and write methods of your application is exactly the same as your final XML document, unless you use a loop.

Once you know how to use XmlReader and Writer, its easy to use. Essentially you have to ensure that your elements open and close at the right points it the code to ensure that you get a properly formatted Xml Document.

Second, make sure you are reading the right data type. I use strings as they are quite easy to use, even for dates and such. Most of my errors come from, that.

Third, make sure that you read and write methods read and write the exact same data at the exact same points ( you don’t want to be reading a name when you expect a date, etc).

Fourth, you need to use the WriteStartDocument method to start writing an Xml file and WriteEndDocument for ending one. thing of them as a huge node that encapsulates rest of the document. Just remember, one document per file.

And that’s all there is to it.

More later on how I’m using XML as a data source.

Related Post: WHS and SmugMug – Keeping Track of Files

WHS and SmugMug – Keeping Track of Files

The question that has to be asked at the beginning of every software project, hobbyist or commercial is: “What are we trying to achieve here?”.

For this project, the aim is to move all the contents of one folder to a gallery on SmugMug – and keep them up-to-date. This means that modifications ( tags, edits, etc), additions and deletions are replicated on the gallery in question.

This requires two things:

  • The program to be aware of what is happening to those folders
  • And for the program to be aware of what has already been uploaded.

Number one requires a FileWatcher Object from System.IO.

The Filewatcher throws an event for creations, deletions  and modifications. This allows us to determine the nature of, say, a create event and decide if its a file or directory that has been created. Based on that we can take other action.

We do this by calling System.IO.Directory.Exists(string path) that returns a boolean. You could use the File.Exists(string path) method as well.

Second, a running tally has to be kept of what is going on. We do that by maintaining a master file in our root directory (say \\Photos\SmugMug)  and child files in each directory. The actual directory may or may not be exposed as a setting via the Console.

The master file is essentially has a queue of file events to execute on the server the next time its syncs. This stops large file  ( i.e. large numbers of files) operations overloading the server (resulting in multiple create events) as Demigrator.exe is working on those files as well. So better  to queue the uploads to take place when the disks are idle. I’m using XML and handwriting the read and write procedures – you’ll understand why at the end of the post.

The second requires a file in each directory with the SmugMug Gallery ID and  a list of each file and its SmugMug ID in that Directory. This allows us quick access to files instead of iterating over the lot and comparing each file to the one on SmugMug.

As far as scheduling the uploader is concerned, I haven’t gone into that in too much detail. I could schedule an exe to run using the task scheduler. Its not a bad way of doing things. Norton Internet Security does just that for its scheduling. That, of course has a disadvantage of spitting up the add-in into a logical front-end and back-end.

And by the way, I’m writing this in Visual Studio 2005 using C#. I’m going to do a second version in C# Express 2008 (to use LINQ-to-XML, mainly) for a comparison.

WHS and Smugmug 1

The first thing I looked for after signing up for SmugMug was a WHS Add-In since there is one for Flikr. Lo and behold, there isn’t one.

So last night I started writing an application architecture post when it occurred to me that the best way of explaining it was to write a program from scratch.

Enter the SmugMug Add-In idea. So I’ll be doing that in a short series of posts.

First things first, I haven’t decided whether to use Visual Studio 2005 or Express 2008.

Second, the WHS SDK is already in your servers’ Program Files\Windows Home Server folder, so don’t bother looking online for it. You’ll need these two files added as references in Visual Studio ( or Express):

  • HomeServerExt.dll
  • Microsoft.HomeServer.SDK.Interop.v1.dll

Third, the Smugmug API can be downloaded from SmugMug.com. And it looks to be rather straight forward to write code against it (Note – I always say that looking at an API for the first time, then regret it later). And you’ll need an API key from SmugMug. And you’ll need to add the API in as a reference in Visual Studio or Express.

For a quick look at it, Evan Leventhal’s .NET wrapper SmugMug.NET, is a good start.

Fourth there the way the program will work is pretty straight forward :

  • The folder name can be the name of the gallery OR an XML file in the folder will identify which gallery it corresponds to (that is the SmugMug Gallery ID) ala SyncToy.
  • Since I keep far more photos than I post in \\Server\Photos,  it would be a good idea to create a separate folder containing all the gallery folders for syncing.
  • As far as checking the folders themselves, the .Net 2.0 has a FileWatcher object ( or something to that effect).
  • A settings tab will have a list of folders in the gallery folder that will be synced by default unless you uncheck it.
  • As far as a console tab goes, I have no idea – stats, gallery browser, etc.
  • And scheduling of some sort, could do somthing ala SyncToy (and no, I’m not french).

I’m writing this primarily for myself  as I’m fed up with having to Remote in and use the Send To SmugMug shell app but it will be packaged neatly and on Codeplex once its finished, I imagine.

Media Centre or not?

The issue of streaming media (TV, Music, Photos) across a network really zeros in on the reason for having a home network.

For those of you who read this blog, you’ll know that I’ve been trying out SageTv for Windows Home Server. And it works great, if you set aside the EPG issues. Recorded TV is of high quality, crisp and clear; and watching from a SageTv Client is only slightly spotty (mainly due to the server being busy and the wireless connection).

My problem is two fold. One WHS is actually a server and is busy all the time and only has a Celeron D.

Two, SageTV shows only play in SageTv itself or in Real Player. Requiring re-encoding to watch via WMP11 or for use in Movie Maker. Which is a pain since a 27 minute show comes out at 650MB at Normal quality and takes about 4 and a half hours for Windows Media Encoder to process.

The EPG issue that I’m encountering means that the Program Guide only has info for about 4 channels out of 24. It is irritating in the extreme since identifying shows requires that you watch them. UK EPG support is quite spotty the forums tell me.

The solution to all of this is, I think, a Media Centre PC. A dedicated hardware device. The whole point of dedicated hardware is to share the load. My broadband connection goes though a router instead of a PC. A GPU handles 3D graphics instead of a normal, general purpose processor. In this case, WHS can stick to the business of serving files and taking backups and let the Media Centre take the load.

I’ve only heard positive reviews for Media Centre Edition (XP) and it seems pretty good. On top of that, Microsoft has a good strategy as far as extenders are concerned such as the Xbox 360 and so on. Also it should slip seamlessly into an all-Microsoft network.

I’d be using WHS for Storage, and WMP11 to watch the content on a PC. It also makes the purchase of a 360 more likely (I’ve been sitting on the fence lately and can’t decide either way).

To actually build the hardware, I’d use the same method that I did for my Windows Home Server. Order the shell (Chassis, power supply, motherboard, processor(s), hard drive) from Dell and then order the rest of the kit from Amazon.

The thing is, how much juice do you need in a Media Centre? SageTV uses about 15% CPU at record time and not much RAM.

Dual Quad Core 2 might be pushing it a little far. A single Core Duo 2 might cut it, but then again, you don’t want “might” to be anywhere near your Media Center. Specially since after-market upgrading can be extremely expensive for hardware like this.

RAM is dependant on whether Vista Media Centre is 64-bit. I’d say that 2 or 4 GB is safe enough.

The next thing is required for this is, well, a network. Streaming media over wireless is Ok if its small enough and you are close enough to the Access Point to get the and excellent signal ( In my house that practically means sitting right next to it). For real media streaming, you need Gigabit Ethernet. More on that to come.

Microsoft Search Server

I had next to no idea about it until Mary J Foley’s article about it.

I’m still a bit stupefied as to what exactly it does. Search. Yeah, I got that bit. But what, aside from stuff the server is installed on?

Index content on file servers, web sites, Windows SharePoint Services, Microsoft Office SharePoint Server, Exchange Server public folders, and Lotus Notes repositories. Find additional Indexing Connectors in the Search Connector Gallery.

I checked the Connector Gallery. It’s simply an explanation of the types of connectors.

As far as I can tell, Connectors allow target systems to be indexed by the Search Server.

And how does Windows Desktop Search come in to this whole thing???

And its free. The very best part of any software system you can find ๐Ÿ™‚ .

Microsoft’s site also mentions that you can program against it. And that is really cool. A WHS Add-In anyone???

Though, in hindsight, its a wonder Microsoft never incorporated the Express version with WHS somehow. it would have ben an extra killer feature that would have tied right in with the Remote Access bit.

I’m adding it to my WHS v2 Feature Wishlist.