Until we get our cross-posting in order, I’ll post links here to each of my posts over there, starting with these:

• How we built a service bus for all of our application instances to talk to one another (a series of 5 posts and counting)
• How we got ready for the huge traffic spike on US election night in November 2010
• How much faster Visual Studio is when running on solid state drives

But please subscribe over there to get all the goodies from the whole team.

To all you US citizens over 18:  VOTE!  And then watch the election results as they come in.  Works well on your mobile phone or iPad, too.

The next big question is: How do we divide our CMS into large-grained sets of discrete functionality – business services?

A good CMS cleanly separates content from presentation.  Even simple blog-oriented CMSs have this separation.  This blog is currently hosted in WordPress.  By simply downloading a different WordPress theme, I can change not only colors and fonts, but the entire structure of the site.  I can add widgets to the sidebar, change the header image, show either full posts or excerpts on the main page, add buttons to share on Twitter and Facebook, change URLs or move pages around into an entirely different hierarchical structure – all without touching the content of a single post.

With a large news-oriented CMS, this separation is even more important.  We deliver news not just to browsers, but to all kinds of applications and devices – from iPads to Bing Maps and Google Earth, from Microsoft Surface tables to a 70s-style arcade game machine in Rockefeller Center.  But our reporters and editors shouldn’t have to worry about – or even be aware of – all of that.  They just need to worry about crafting great stories.

Thus, the most obvious business services within our CMS are what we call Editorial and Delivery.

As a news organization, we not only deliver the news – we consume it in enormous quantities.  Feeds like the AP, UPI, Agence France-Presse, and hundreds of other text and photo sources all flow in 24 hours a day.  Then there are data sources like weather, stock market tickers, lottery results, etc.; it all has to be gathered, organized, searched, categorized, edited, and made available for publishing.  Elections produce enormous amounts of data that we need to sort, search, and aggregate.  All of this functionality makes up the Ingest business service.

One type of ingested content absent from the list above is video.  NBC News (a parent of ours) is one of the largest producers of video news content in the world.  We support the websites for msnbc.com, cnbc.com, todayshow.com, plus shows like Rachel Maddow, Countdown, Dateline, Nightly News, Meet the Press, and more.  Video is different from text and photo content in many ways:  sheer file size, bandwidth consumption, encoding standards (and the computing powered needed for encoding), editing tools, advertising model, etc.

As a result of this complexity, video is a technical and editorial specialty unto itself.  I’m no expert in video, so I’ll leave the detailed discussion to others.  But as a major functional area that has to integrate with everything else in our CMS, Video is a business service.

Our other parent (Microsoft) gives us access to a wealth of technology.  Not only the platform building blocks of Windows Server, ASP.NET, WCF, etc.; but also cool stuff like the ability to scan huge amounts of content and determine the most important people, places and concepts mentioned.  And to automatically determine which stories, photos, and videos are related to one another.  Check out the Related and Topics areas near the bottom of a story page to see what I mean.  Like video, we want this capability integrated with many other parts of the system, yet it is more or less a world unto itself.  This is the Topics business service.

With any non-trivial system, we also need to concern ourselves with day to day operations and maintenance.  How do we easily deploy to and configure all those servers in multiple data centers?  How do we monitor them and get notified when something isn’t working?  How do we get detailed troubleshooting data for a server or process gone awry?  While some of these capabilities need to be part of the implementation of every service, the data aggregation, monitoring and notifications are business functionality that forms the Operations service.

In future posts, I’ll dig into these services and their component applications, focusing first on the Delivery service.

As we began work on a new CMS for msnbc.com, one of the major pain points with the existing system that we wanted to address was its database oriented architecture – or DOA.   No, not the popular video game (searching for DOA is borderline NSFW).  Rather, “dead on arrival” – the morbid term for an ambulance patient who doesn’t survive the trip to the hospital.

A multitude of dependencies on a single central database is, indeed, DOA.  First, it doesn’t scale well up to billions of page views per month, or make it easy to achieve redundancy across data centers.  Doing so requires all sorts of magic with clusters, mirroring, replication, etc. – in other words, expensive hardware and even more expensive people.  (The reason it has worked up to now is our top-notch operations team.)

Second, all of those dependencies mean that changes to the database schema are effectively impossible. 

A CMS like ours isn’t a single application.  It’s an ecosystem of applications, built by different teams over a long period of time and performing a wide variety of functions: filtering and aggregating hundreds of inbound news wires in different formats, encoding all the video from one of the largest news organizations in the world, supporting journalists on tight deadlines around the clock, and – oh, yeah – serving up all that content to millions of people a day. 

Most of those applications create or manipulate content that is stored in the central database.  Thus, any change to the database is likely to have side-effects.  A change to support one application, simple on its own, turns out to require a major refactoring effort in an unrelated application.  Fixing bugs becomes a game of whack-a-mole; new features become more expensive and take longer and longer to release.  Worse, a great new feature may not get built at all.

This isn’t really a problem with databases, of course.  It’s a problem with dependency management.  But it is exacerbated by the fact that accessing a database is inherently a blocking operation – you have to wait for the result.  Any single resource that must be available all the time in order for any other part of the system to function correctly is going to cause these types of problems.

For all of these reasons, we opted instead for a service oriented architecture (SOA).  That doesn’t mean simply exposing everything as a web service.  It means partitioning the system into “business services” – large grained, autonomous sets of functionality.  Being autonomous includes having one’s own private database (assuming there is some data for that service to persist).  Private means private – the database is not shared with other services.

SOA is not exactly a radical choice, but it does require a radical change in thinking – for developers, for admins, even for end users.  It requires letting go of 100% consistency 100% of the time in favor of eventual consistency, letting go of “one database to rule them all,” and accepting that maintaining multiple, service-specific versions of the data won’t be the end of the world.

It takes a lot of time and constant effort to completely change your world view.  It’s not unlike a procedural developer being introduced to object orientation.  It takes a lot of practice, and dozens of “Aha!” moments to cross the conceptual chasm.  As an organization, we’re still in mid-air.  I expect we’ll eventually land safely on the other side, and not wind up DOA.

In a future post, I’ll write about how we drew the boundaries between our business services.

The visible part of the change is an entirely new look for stories.  You can take a tour.  There are some great new features, including the ability to easily embed video, like so:

Visit msnbc.com for breaking news, world news, and news about the economy

Alas, I had nothing to do with all those great features…  My contribution was in the invisible parts – the underlying platform on top of which some smart, creative folks have built the new pages.  There’s quite a bit more to it than meets the eye. 

I’ll be posting on a bunch of topics related to what we did, how we did it, and why we did it that way.  The highlight reel will include: SOA, EDA, messaging, service bus, WCF, MSMQ, ASP.NET MVC, unit testing, TFS, performance testing and profiling, Avicode, automated deployments, and a lot more.  To the extent I’m allowed to, I’ll share some code, too.  Stay tuned…