Building a Feed Reader on Windows Azure – Screencast Part 1

As I announced last month, I’ll be screen casting my efforts on the Windows Azure Feed reader.

Well, a couple of weeks ago, I just went ahead and did the first episode. The delay between recording it and posting is due to a weeks holiday and a further week and a half spent offline due to my ISP.

I’ve open sourced the code to http://windowsazurefeeds.codeplex.com.

In this episode I cover:

  • the data access layer
  • associated helper code
  • I add a skeleton Feed class
  • I add all the Windows Azure Roles that we’ll be needing

I’ve tried not only to explain what I’m doing, but also why I’ve done things this way – with a view to how it will affect things down the road.

So here is the screencast:

Next time we’ll begin integrating things with our worker and web roles. We’ll be working with blobs and queues. And we’ll start chewing through some live data.

I’ll try keeping to a weekly schedule, but my schedule of late is anything but regular and predictable.

Enjoy.

Oil Spills Live Feeds App 1.4 – Alpha

Just pushed out a new version of the application. I added the live feeds from the Ocean Intervention II and Viking Poseidon.

So there are a total of 12 feeds available forming 6 panels:

SpillFeeds6

As usual you can add and remove panels as you wish:

image

 

This is an alpha release, so it still needs work.

 

You can download it from here: http://oilslickfeeds.codeplex.com/releases/view/48477

I’ll let you know when a more polished build is done.

Screencast Plans

So, I’m prepping to make a start on the screen cast series on Building a Feedreader on Windows Azure. I thought I’d make a list of things to do both  for the screencast itself and for the code.

So this post will be updated as I think of things.

After the test I ran by doing a screencast on the Oil Spills app, I figured that I’m dispensing with the webcam overlay – its too distracting.

Also as the test showed, I’m gonna have to deal with frequent interruptions – for drinking water, for screwing up the explanation of something etc.  So I’m gonna have to record each part in a number of separate chuncks and join them together with Windows Live Movie Maker.

Also, I decided that its best not to start from scratch. I’m gonna have some of the code already written, namely the data access layer for Windows Azure Tables. Two reasons for this. One I have to make adjustments of the previous version of DAL. Two it’s practically boilerplate code and I don’t want to bore you.

After the final commit for each episode, I’ll post the link to the changeset, so you can download the code.

Finally, as we get later into the series (even the earth took 7 days to create), I may do an intro to recap the previous week using the webcam.

Now I do want to add one new feature for sure – and thats shared items, as Google Reader does. This will be entirely straightforward. items will be added to a RSS feed of shared items from that user. On request, i just read the blob and return the contents as an RSS file.

Now one thing is for certain – I’m definitely doing to do some things wrong in my code. So please be nice and pointed out. :)  Or, even better, submit a diff file correcting it 🙂

Thoughts on Windows Azure Storage

I’m sitting here planning what to do with the re-write of the Feedreader I wrote for university. Naturally, the storage solution is always going to be a contentious issue. The NoSql versus SQl debate plays a lot into it.

But, for the most part, i used Windows Azure tables and blobs for its simplicity over SQL Azure. In saying that, SQL is not my favourite thing in the world, so make of that what you will. But also, for a demonstrator application, the use of something completely new played in my hands very well.

So the re-write is also meant to be a demonstrator application. so the Windows Azure storage is staying.

But, not so fast. because Windows Azure Storage  needs something. The way I used Tables and Blobs essentially was as a poor mans object database. Thus meant that there was a lot of leg work involved in this, not to mention tedious plumbing code. The fact is, I think that the is the most logical use case for Windows Azure Storage – where in metadata is stored in the tables and the object themselves are in the blobs.

What I would like to be added, then, is the ability to formalize this relationship in code. Some way of saying “hey, this value in this TableRow actually points to a blob in this container”. So I can call “getBlob()”, or something on a Row an get the blob back. Now, to be clear, I don’t mean this to be a foreign key relationship. I don’t want to cascade updates and deletes. And i certainly don’t want my hands tied by making the Blob Attribute column (or whatever) mandatory.

Now, this could be added right now. And in fact am considering doing that. But support for this in the Storage Client library would be nice. But weather backend support is needed, or in fact a good idea, is another question entirely. The implications of making such a fundamental change on the back end. For example, say you’ve exposed a table via OData. How do you expose the blob as well? And given the nature of the use case, the fact that it is needed on a table by table basis makes it much easier to limit any such functionality to the Tools library only.

I can hear you asking yourself why I’m asking for support in the Storage Client library if I can gin up and duct tape together some Extension Methods? I’m a big proponent of the idea that anything that we use in intact with software, be that actual applications or libraries we use in our code, has a Natural user interface. Think of it, the API methods that are exposed for us as developers to use are in fact a User Interface. so I’m asking for a better user interface that supports this functionality without me having to do the legwork for it. In so delivering support, it is perfectly possible, indeed likely, that the library code that Microsoft ships will be more efficient than whatever code I can write.

My final report on the project did call out VS Windows Azure Tools, mainly for not making my life easier. So I’m looking forward to the new version (1.3) and seeing how it goes, particularly with regard to storage.

Now performance-wise, the version I wrote last wasn’t exactly fast at retrieving data. I suspect that this was due to a) my own code in-efficiencies and b) the fact that my data wasn’t optimally (obviously) normalized. Its also probable that better use of MVC viewdata (actually, I think that should be “better use of MVC, period”) and caching will improve things a lot as well.

Oil Spill Live Feeds App 1.3 & Screencast

This is my attempt to kill 2 birds with one stone.

So, item number one is a new version of the Oil Spills application. New features added are:

  • ability to add and remove panels
  • ability to refresh feeds that have gone down
  • all panels now have the same size – the application will adjust this based on the number of panels and the size of the form
  • added some application icons

You can get the new release here: http://oilslickfeeds.codeplex.com/releases/view/47469

This may seem like a little, but its really a total re-write.

Being able to remove panels that you do not want to watch makes the application more memory efficient. Though there is more work that can be done in this area, there is only so much you can do when streaming 8 live feeds.

Item number 2 is the fact that I did a short, 6 minute screen cast discussing the innards of the application. Now, it may seem silly to do a screencast for something so trivial, but I wanted some practice. It turns out that its not as easy as it sounds. So this video is my 4th attempt (which I’m still not 100% happy with, mainly because of the overlay).

I did this with Expression Encoder 3 Screen Capture. I usually use Community Clips for this sort of stuff. But, its not a half bad screen capture program. Expression Encoder Screen capture will let you add an overlay from an external camera, in this case, my laptops integrated webcam. I’m not too sure about this for future screencasts – so do let me know what you think.

So here it is:

My RSSCloud Server: Thinking of doing some screencasts.

This year was my last at Uni ( actually, i still have an exam to write, so the past tense isn’t accurate). As is typical with Honours year undergraduates, a final year project was set.

If you are regular reader of this blog, you’ll probably know that what i picked was a RSSCloud server running on Windows Azure. However, as they say on the home shopping networks, THERES MORE! My project needed a little more body to it. So I added an online Feedreader, in other words a poor (dirt-simple) imitation of Google Reader.

Now, this app uses a number of technologies for which it would be a pretty cool demo project. Windows Azure itself (obviously), Windows Azure Tables and Blobs, LINQ, WCF, MVC2 and so on. This includes it being a demonstrator of the RSSCLoud specification itself.

Although its an academic submission, my lecturers are fine with me opensourceing it.

Given the rise of .Net 4, and the experience points gained writing the first version, I feel that everyone would be better served with a rewrite. Not to mention the fact that It’ll give me a chance to use the new Windows Azure Tools for Visual Studio.

As I re-write it, I think a screencast series is in order. All the code will be checked in to codeplex. This’ll give everyone a chance to double check my logic (particularly interested in what Dave Winer thinks of my implementation of RSSCloud).

So, firstly, What do you think?

And secondly, anyone know a good hosting provider? I don’t know about Youtube. But Vimeo looks pretty good. If their limit is 500Gbs/per week upload space, it’ll give me chance to do one video each week, more or less.

I have all the software required to pull this off. So thats not a problem. I actually did a screencast of a live coding session in class for one of my lectures (writing an interpreter turns out to be pretty fun, actually).

i think this would be a pretty good contribution to the community as a whole.

Theres an App for that: BP Oil spill live feeds

SpillFeeds

SpillFeeds2 SpillFeeds3 SpillFeeds4

Like that. Took me 30 minutes.

Head over to http://oilslickfeeds.codeplex.com/ to get it.

Note: These are live streams from the ROV’s monitoring the damaged riser. Please be aware, as these are live streams they may freeze or be unavailable from time to time.

(thanks to the Channel 4 TV guys for putting the original web page together)

Update 08/06/2010  There was a problem with the install files. I’ve uploaded a new version. Please do let me know if there are further problems.

Update 24/06/2010  Version 1.3 has been released. please see the details here: https://rbonini.wordpress.com/2010/06/24/oil-spill-app-1-3-screencast/

Running Windows Communication Foundation on Windows Azure

In a perfect world, these two Microsoft products would Just Work. Unfortunately, it does not quite work that way in practice.

In general, WCF can be run in a number of ways.

  • The First is to use IIS to host your service for you. IIS will take care of port mapping between internal and external ports.
  • The Second is to self host. This means that your process will create a ServiceHost, passing in the appropriate class. Your service is responsible for error handling and termination of the WCF service. The service class itself can contain the Service behaviour attributes or you can stick them in the app.config file.  This also means that you have to ensure that the WCF service is listening  on the correct port when you create it.

In Windows Azure terms, the first method would be a Web role, and the second method would be a worker role.

It should be noted that all of the above methods and code patterns are perfectly legitimate ways to create and host a WCF service. The question here is, which of these ways will run on Windows Azure.

Programming for Windows Azure is generally an easy thing so long as your application start up code is correct. If there are any exceptions in your start up code, the role will cycle between Initializing, Busy and Stopping. Its enough to drive you mad. Therefore, if you choose to self host, you need to be sure that your worker role code runs perfectly.

For all your Windows Azure applications its also a good idea to create a log table using Windows Azure tables to assist you in debugging your service.

Now, before you begin any WCF Azure work, you need to install a patch. WCF can exhibit some strange behaviour when deployed behind a load balancer.Since all Windows Azure service instances are behind a LB by default, we need to install this patch. This patch is also installed in the cloud.

You can find the patch, as well a thorough run down of all the WCF issues in Azure here. While I’m not attempting to tackle all of the problems, and indeed the solutions to them, i do aim to get a basic service working with as little fuss as possible.

IIS Hosting

It must be said that i did try all sorts of settings in the config file. It turns out that the one with almost no actual settings is the one that works. basically this forces us to rely on IIS to do the configurations for us. On one hand this is a good thing, saving us the hassle. On the other, there is some loss of control that the programmer in me does not like.

Firstly, lets look at the config file. This section is vanilla. And basically what you’ll get when you add the WCF Service Web Role via the Add New Role dialog.

<system.serviceModel>
<services>
<service name="WCFServiceWebRole1.Service1" behaviorConfiguration="WCFServiceWebRole1.Service1Behavior">
<!-- Service Endpoints -->
<endpoint address="" binding="basicHttpBinding" contract="WCFServiceWebRole1.Service1">

</endpoint>
<endpoint address="mex" binding="mexHttpBinding" contract="IMetadataExchange"/>
</service>
</services>
<behaviors>
<serviceBehaviors>
<behavior name="WCFServiceWebRole1.Service1Behavior">
<!-- To avoid disclosing metadata information, set the value below to false and remove the metadata endpoint above before deployment -->
<serviceMetadata httpGetEnabled="true"/>
<!-- To receive exception details in faults for debugging purposes, set the value below to true.  Set to false before deployment to avoid disclosing exception information -->
<serviceDebug includeExceptionDetailInFaults="false"/>
</behavior>
</serviceBehaviors>
</behaviors>
</system.serviceModel>

If you deploy this file to the cloud, the role will initialise and start up correctly. However, at this point, the service is not aware of anything outwith the load balancer. In fact, if you deploy the WCF web role template to the cloud it will start up and run perfectly, albeit only behind the LB.

Now for the actual service code.  This code is actually from a REST service I wrote. It basically provides a interface for the RSSCloud specification (sorry Dave). So we can talk to the RSS Cloud server using HTTP Post, REST and any WSDL client.

[ServiceContract]
[AspNetCompatibilityRequirements(RequirementsMode = AspNetCompatibilityRequirementsMode.Allowed)]
[ServiceBehavior(InstanceContextMode = InstanceContextMode.PerCall)]
public class Service1
{

[WebInvoke(Method = "POST", UriTemplate = "pleasenotify/notifyProcedure/{notifyprocedure}/port/{port}/protocol/{protocol}/domain/{domain}/url/{url}"), OperationContract]
String RequestNotification(string notifyprocedure, int port, string path, string protocol, string domain, string url)
{
// this change to the storage model is intended to make scaleing easier.
// the  subscribers will be stored perblog in individual blobs
// the table will store the blogids, the blogurls and the last modified date of the blob file.
DateTime Timestamp = DateTime.Now;
XmlDocument thedoc = new XmlDocument();

string name = BlobNameServices.generateBlobName(url);

if (name == null)
{
name = BlobNameServices.CreateBlobEntry(url, source);
}
// Set the metadata into the blob

string hash = Utils.HashToBase64(domain + path + port + protocol + url);

insertTable(name, hash, notifyprocedure, port, path, protocol, domain, url);

requestqueue.AddMessage(new CloudQueueMessage(name));

return "reply";
}

[WebInvoke(Method = "POST", UriTemplate = "recivenotify/url/{url}"), OperationContract]
String RecieveNotification(string url)
{
recieverqueue.AddMessage(new CloudQueueMessage(url));

return "Recieved";
}

[WebInvoke(Method = "POST", UriTemplate = "ping/url/{value}"), OperationContract]
String Ping(string value)
{
char[] chars = value.ToCharArray();

//we have a potential inconsistancy here - the url is passed as an arguement,
//whereas for notifications the blog id is passed as an argument

pingqueue.AddMessage(new CloudQueueMessage(value));

String result = "You said: " + value;

return result;
}

}

I arrived at this code by taking the vanilla WCF webrole and combining it with the WCF REST template and replacing the template code with my own code

Amazingly enough, it will work. Up to a point though. When you hit the svc file, it points you to a WSDL BEHIND the load balancer, using the local instances’  address. Obviously, we can’t get to it. So, we have to modify our WSDL very slightly to take advantage of the patch mentioned above.

<behaviors>
<serviceBehaviors>
<behavior name="WCFServiceWebRole1.Service1Behavior">
<!-- To avoid disclosing metadata information, set the value below to false and remove the metadata endpoint above before deployment -->
<serviceMetadata httpGetEnabled="true"/>
<!-- To receive exception details in faults for debugging purposes, set the value below to true.  Set to false before deployment to avoid disclosing exception information -->
<serviceDebug includeExceptionDetailInFaults="false"/>
<useRequestHeadersForMetadataAddress>
<defaultPorts>
<add scheme="http" port="8080" />
</defaultPorts>
</useRequestHeadersForMetadataAddress>
</behavior>
</serviceBehaviors>
</behaviors>
<pre>

If you hit the service now, the WSDL file is now pointing to the correct address and you can get to it. So any WSDL client can now talk to your service.

You want to make sure that the port selected in in the webrole settings matches the port you have in the Request headers section. Whether that is port 80 or something else. If you have an HTTPS end point, you need to add it as well to get it to work.

Self Hosting

Now, I’ve tried self hosting. This is the idea.

string port="";
try
{
port = RoleEnvironment.CurrentRoleInstance.InstanceEndpoints["WCFEndpoint"].IPEndpoint.Port.ToString();
}
catch
{
port = "8000";
}
try
{
using (ServiceHost svcHost = new ServiceHost(typeof(RssNotifyService), new Uri(http://northwind.cloudapp.net: + port)))
{

svcHost.Open();
while (true)
{
//do somthing
}
svcHost.Close();
}
}
catch(Exception ex)
{
CloudStorageAccount acc = CloudStorageAccount.FromConfigurationSetting("DataConnectionString");
LogDataSource log = new LogDataSource(acc);
log.Insert("WCF",ex.Message,"Error",DateTime.Now);

}

For this to work you need to have a HTTP in port set in your web role settings. In this case it was port 8000.

While the above will work in the development fabric, it won’t  in the cloud. But it demonstrates the idea.

At the end of the day, there is not very much to set up if you are writing a new WCF service with minimal custom configuration.

So, install the patch and watch that config file.

PS. Many thanks to Steve Marx ( Windows Azure Program Manager) for his help with this.

Codeplex Project Updates

Meant to do this quick rundown a few weeks back.

  • WCF Client Server Chat– No new development. However, I do think that I’ll have a ready to run version to post. No timing on this. But since this is apparently the most popular project here, I’m feeling the pressure.
  • RSSCloudNotify. Yes, there is a bug in the code. Sometimes it makes it through to the server and sometimes not. This project is in active development, it being a part of my honours year project for uni. The current build was a quick test project to get an idea of the scale of the task. So it WILL be re-written properly leveraging the full ASP.Net stack.
  • WHS2Smugmug. This is also under active development, albeit with a hiatus while I complete my last few months at uni. I have definitely not forgotten about this. Currently, its awaiting testing and revising before releasing a beta. (Don’t forget to follow the project on twitter @whs2smugmug)
  • FFNotify – No new development for awhile. But I do plan to fix the bugs in the current revision. Freindfeed is pretty snappy importing blogs now, so this is not exactly high priority (if you feel different, give me a shout).

Naturally, all the above fixes etc (save for RSSCloudNotify) are subject to how things go at uni. Indeed, some of the stuff I will be writing for my honours year project directly translate back to these projects.

These are all projects that I care deeply about. And are projects that I stated to solve my own problems. Unfortunately there are only so many hours in the day 🙂

Thanks for your interest and support.

Roberto

Announcing RSSCloud Notify for Windows Live Writer

CloudRSS

I’m developing a project this year around the RSS Cloud protocol.

As a starting point, I’ve written a a short and sweet WLW plugin.

I’ve got it installed locally (eating my own dogfood), mainly cause useful in testing my system once its written.

This may be a bit of a contradiction in terms since wordpress.com already supports RSSCloud (you’ll see a RSSCloud tag in my feed if you view source. It gives the details of the wordpress.com aggregator).

By default it will use Dave Winer’s test server as an endpoint so make sure you change your settings before use.

Its available at codeplex here: http://rsscloudnotify.codeplex.com/

The source is in the repository if anyone wants to look at it.  Its not quite ready for prime time yet. 🙂

Update 17/11/09 Updated to version 1.01