The next big question is: How do we divide our CMS into large-grained sets of discrete functionality – business services?

A good CMS cleanly separates content from presentation.  Even simple blog-oriented CMSs have this separation.  This blog is currently hosted in WordPress.  By simply downloading a different WordPress theme, I can change not only colors and fonts, but the entire structure of the site.  I can add widgets to the sidebar, change the header image, show either full posts or excerpts on the main page, add buttons to share on Twitter and Facebook, change URLs or move pages around into an entirely different hierarchical structure – all without touching the content of a single post.

With a large news-oriented CMS, this separation is even more important.  We deliver news not just to browsers, but to all kinds of applications and devices – from iPads to Bing Maps and Google Earth, from Microsoft Surface tables to a 70s-style arcade game machine in Rockefeller Center.  But our reporters and editors shouldn’t have to worry about – or even be aware of – all of that.  They just need to worry about crafting great stories.

Thus, the most obvious business services within our CMS are what we call Editorial and Delivery.

As a news organization, we not only deliver the news – we consume it in enormous quantities.  Feeds like the AP, UPI, Agence France-Presse, and hundreds of other text and photo sources all flow in 24 hours a day.  Then there are data sources like weather, stock market tickers, lottery results, etc.; it all has to be gathered, organized, searched, categorized, edited, and made available for publishing.  Elections produce enormous amounts of data that we need to sort, search, and aggregate.  All of this functionality makes up the Ingest business service.

One type of ingested content absent from the list above is video.  NBC News (a parent of ours) is one of the largest producers of video news content in the world.  We support the websites for msnbc.com, cnbc.com, todayshow.com, plus shows like Rachel Maddow, Countdown, Dateline, Nightly News, Meet the Press, and more.  Video is different from text and photo content in many ways:  sheer file size, bandwidth consumption, encoding standards (and the computing powered needed for encoding), editing tools, advertising model, etc.

As a result of this complexity, video is a technical and editorial specialty unto itself.  I’m no expert in video, so I’ll leave the detailed discussion to others.  But as a major functional area that has to integrate with everything else in our CMS, Video is a business service.

Our other parent (Microsoft) gives us access to a wealth of technology.  Not only the platform building blocks of Windows Server, ASP.NET, WCF, etc.; but also cool stuff like the ability to scan huge amounts of content and determine the most important people, places and concepts mentioned.  And to automatically determine which stories, photos, and videos are related to one another.  Check out the Related and Topics areas near the bottom of a story page to see what I mean.  Like video, we want this capability integrated with many other parts of the system, yet it is more or less a world unto itself.  This is the Topics business service.

With any non-trivial system, we also need to concern ourselves with day to day operations and maintenance.  How do we easily deploy to and configure all those servers in multiple data centers?  How do we monitor them and get notified when something isn’t working?  How do we get detailed troubleshooting data for a server or process gone awry?  While some of these capabilities need to be part of the implementation of every service, the data aggregation, monitoring and notifications are business functionality that forms the Operations service.

In future posts, I’ll dig into these services and their component applications, focusing first on the Delivery service.

As we began work on a new CMS for msnbc.com, one of the major pain points with the existing system that we wanted to address was its database oriented architecture – or DOA.   No, not the popular video game (searching for DOA is borderline NSFW).  Rather, “dead on arrival” – the morbid term for an ambulance patient who doesn’t survive the trip to the hospital.

A multitude of dependencies on a single central database is, indeed, DOA.  First, it doesn’t scale well up to billions of page views per month, or make it easy to achieve redundancy across data centers.  Doing so requires all sorts of magic with clusters, mirroring, replication, etc. – in other words, expensive hardware and even more expensive people.  (The reason it has worked up to now is our top-notch operations team.)

Second, all of those dependencies mean that changes to the database schema are effectively impossible. 

A CMS like ours isn’t a single application.  It’s an ecosystem of applications, built by different teams over a long period of time and performing a wide variety of functions: filtering and aggregating hundreds of inbound news wires in different formats, encoding all the video from one of the largest news organizations in the world, supporting journalists on tight deadlines around the clock, and – oh, yeah – serving up all that content to millions of people a day. 

Most of those applications create or manipulate content that is stored in the central database.  Thus, any change to the database is likely to have side-effects.  A change to support one application, simple on its own, turns out to require a major refactoring effort in an unrelated application.  Fixing bugs becomes a game of whack-a-mole; new features become more expensive and take longer and longer to release.  Worse, a great new feature may not get built at all.

This isn’t really a problem with databases, of course.  It’s a problem with dependency management.  But it is exacerbated by the fact that accessing a database is inherently a blocking operation – you have to wait for the result.  Any single resource that must be available all the time in order for any other part of the system to function correctly is going to cause these types of problems.

For all of these reasons, we opted instead for a service oriented architecture (SOA).  That doesn’t mean simply exposing everything as a web service.  It means partitioning the system into “business services” – large grained, autonomous sets of functionality.  Being autonomous includes having one’s own private database (assuming there is some data for that service to persist).  Private means private – the database is not shared with other services.

SOA is not exactly a radical choice, but it does require a radical change in thinking – for developers, for admins, even for end users.  It requires letting go of 100% consistency 100% of the time in favor of eventual consistency, letting go of “one database to rule them all,” and accepting that maintaining multiple, service-specific versions of the data won’t be the end of the world.

It takes a lot of time and constant effort to completely change your world view.  It’s not unlike a procedural developer being introduced to object orientation.  It takes a lot of practice, and dozens of “Aha!” moments to cross the conceptual chasm.  As an organization, we’re still in mid-air.  I expect we’ll eventually land safely on the other side, and not wind up DOA.

In a future post, I’ll write about how we drew the boundaries between our business services.

In an event-driven architecture (EDA), business services communicate with one another through asynchronous messaging, in order to reduce the coupling between them.  By communicating asynchronously, a business service can continue to operate (for some period of time) without any contact with other business services.  That’s where their “autonomous” quality comes from.

For example, a retailer might have an Inventory service, an Order service, an Inventory service, a Billing service, and a Shipping service.  The Order service can take orders even if the other services are unavailable.  When those services become available, queued messages can be exchanged — inventory gets allocated, orders get shipped and customers get billed.

Suppose an order is placed over the web.  The Order service might publish an OrderReceived message.  The Inventory service would then publish either an InventoryAssignedToOrder or an ItemOutOfStock message, depending on what was actually available.  The Order service would subscribe to those types of messages.  If it received the first, it would then publish an OrderApproved message.  If it received the second, it might take other action, such as sending an email to the customer.

You may have noticed that we just created a messaging protocol.  That is, the Order service must correlate multiple messages related to the same business process, and keep track of state related to those messages.  Within the Order service, the logic would be something like this:

• When an order is placed (in an inbound web request), save it, publish an OrderReceived message, and send the customer a response message that the order was successfully received.
• When an InventoryAssignedToOrder message is received, look up the order and record the assigned inventory quantity next to the quantity ordered.
• When inventory has been assigned for all quantities of all products in the order, publish an OrderApproved message.
• When an ItemOutOfStock message is received, send an email to the customer.
• When an hour passes after sending an OrderReceived message, if not all quantities in the order have been assigned inventory and no ItemOutOfStock message has been received, publish an OrderDelayed message.

This is a saga — a long-running process within a single service that requires the service to correlate messages and track state across those messages.  Note that this is just one side of the equation.  The Inventory service might have its own saga. 

Notice how each of the saga’s business rules start with “When”.  These rules are what make the architecture event-driven.  Notice also how simple the rules are.  Especially interesting is that the technical implementation maps almost directly to the business rules.  We would create an OrderProcessingSaga class, which has methods for Handle<OrderReceived>(), Handle<InventoryAssignedToOrder>(), etc.  The simple logic of each method is described in one of the bullets above.  (The final bullet is implemented in Handle<Timeout>().)

How would we accomplish the same process in a synchronous RPC (remote procedure call) architecture?  In RPC (like typical HTTP-based web services), a request is issued and the requestor waits for the response.

• When an order is placed (in an inbound web request), save it.  The inbound connection remains open.
• The Order service opens a new connection to the Inventory service.  The connection remains open while the Inventory service determines whether inventory is available for all the quantities in the order.  The response to the Order service includes data on any product SKUs that are out of stock.  The response is sent and the connection from Order service to Inventory service is closed.
• The Order service looks for any out of stock SKUs in the response.  Based on that, it creates the appropriate message (“order confirmed” or “items out of stock”), sends the customer a response message.

At first glance this might seem no different than the message-based process above.  But there are a number of problems with the RPC approach that reach across most of the -ilities:

Reliability.  Even with all the redundant hardware in the world, at some point the Inventory service will be down.  During that time, no orders can be accepted.  If each of the two services has 99% uptime, the total uptime for the system from the customer’s perspective is 99% x 99% = 98.01%.  The system as a whole is less reliable the least reliable component.  To get the system to 99% uptime, both services need uptime of at least 99.5%.  That means a greater investment in hardware, maintenance, etc.

In the message-based approach, the Order service can happily process orders without the Inventory service for up to an hour (or whatever rule is applied in the saga).  The services are not temporally coupled — they don’t always have to be running at the same time.  From the customer’s perspective, the uptime is 99%. 

The Inventory service can be taken down for maintenance.  Perhaps it could be deployed to two cheap servers instead of to an expensive active-passive cluster.  All without affecting the uptime of the Order service.

Performance.  In the RPC approach, the customer submits an order and then waits: while the Order service records the order, while the Order service contacts the Inventory service, while the Inventory service responds, and while the Order service figures out what message to return.  What if the Inventory service takes 5 seconds to determine whether inventory is available?  The customer sees a spinning ball.  In the message-based approach, the customer waits only for the Order service to record the order.  A response is them immediately sent to the customer. 

Now add the latency of communication between the services.  If they are in different data centers, that could add half a second to the customer’s wait time in the RPC approach.  It adds zero time in the message-based approach.

Scalability.  In the RPC approach, connections between the customer and the Order service remain open for longer (as described above).  Let’s say the Order service receives 10 orders/second.  If the connection remains open for 2 seconds, the Order service has to support 20 simultaneous connections.  If the message-based approach reduced the connection time to 1 second, the Order service only has to support 10 simultaneous connections — for the same volume of orders.  This disparity can become much worse under real-world traffic peaks.  Customer requests may wind up queuing (decreasing performance further) or even timing out.

The same problem occurs between the Order service and the Inventory service, as connections are consumed there in the same proportion.

Transparency.  Take another look at the logic for the RPC and message-based approaches.  In which one is the business logic more readily apparent?  Which one would do you think a sales rep would more easily understand?  I believe that the message-based approach wins handily on both counts.  As mentioned above, the business logic maps almost directly to the actual implementation classes and methods.  While a good RPC design can also have this quality, transparency is very natural and very easy in the message-based approach.

Maintainability.  Let’s add shipping to our business process.  In the RPC approach, during the order request, just after we assign the inventory to the order, we would open a connection to the Shipping service, ask it to ship the order, wait for its response, and then send the response to the customer.  With this third service, we’ve exacerbated all of the reliability, performance and scalability issues described above. 

All three systems have to be running.  At 99% uptime for each service, our overall our system is now down to 97.03% uptime.  We don’t want to ship the order unless we actually have all products in stock, so we have to wait for the request to the Inventory service to return before we talk to the Shipping service.  Our connections from the customer are now up to 3 seconds each, and our Order service has to support 30 simultaneous connections.

What happens when we’ve assigned inventory and then the call to the shipping service fails?  We have to free the inventory somehow, so we place the whole order request in a transaction.  That transaction flows across each service call, so that if shipping fails the inventory assignment is rolled back.  The transaction management adds half a second to the customer’s wait time, and we’re up to needing 35 simultaneous connections on our Order service.

In the message-based approach, we create a new saga within the Shipping service.  It has a rule saying: When an OrderApproved message is received, try to ship the order.  If successful, publish an OrderShipped message; otherwise publish an OrderFailed message.

We add rules to the OrderProcessingSaga:  When an OrderShipped message is received, update the order status; and when an OrderFailed message is received, update the order status and send an email to the customer. 

And we add a rule to the Inventory service’s saga:  When an OrderFailed message is received, release any inventory assigned to the order.

We haven’t had to make a single change to the processing logic for the customer’s initial request to place the order.  We haven’t had to create distributed transactions, because the business rules in the various sagas handle failures correctly.  We haven’t reduced our system uptime.  We haven’t lengthened the time a customer waits for a response, and therefore we don’t have to support more simultaneous connections from customers.

Conclusions

The business process described here is pretty simple.  Real world processes get much more complicated:  Add a credit check before assigning inventory.  Bill the customer after the order ships, after getting sales tax computation from an external service.  Substitute an equivalent product if the one ordered has been discontinued.  Yikes! 

If you read “service-oriented” as “synchronous RPC-style web services,” you’ll quickly be in big trouble with a system of any significant size.  However, if you read “service-oriented” as “event-driven, autonomous business services communicating with asynchronous message across service boundaries,” you’ll have a much better chance of actually enjoying the maintenance of your system.  I’ll take the latter, thanks.

As someone who consumes a lot of guidance from Microsoft’s p&p team, and looks to gurus like Nicholas Allen, Michele Leroux Bustamante and Juval Lowy for all sorts of WCF insights, my working definition of a service has been a very technical one.  A service is a cohesive set of functionality described by a contract.  In WCF terms, it’s a .NET interface decorated with a [ServiceContract] attribute, along with one or more implementations of that interface.

One of Udi‘s major themes in the training was how writing code is usually the easy part.  It’s design that can be hard, and good design is always closely related to the business problem you are solving.  The “Aha!” for me was that a service in SOA has very little to do with technology — a service is a business concept.  Business services are what SOA is all about.  Apparently my mistake is not uncommon.

Bill Poole has written a series of excellent posts on SOA.  I highly recommend them as a primer on business services, including how they relate to web services.  (In fact, you should just go back to his first post from early 2008 and read them all.  It’s well worth a couple hours of your time.)  From Bill:

So – the primary concern in SOA is the identification and definition of the architecture’s constituent services. When applying SOA in a business context we are concerned with identifying business services. A business service serves a specific business purpose, and is defined strictly in business terms.

We define the responsibilities of a business service in terms of the cohesive business area/capability the service supports, and we define the interactions with other services in terms of the business events and operations exposed by the service, as well as the service level agreements provided by the service and the policies imposed by the service on its consumers.

A business service is conceptual and maps to a business capability, not to a development artifact.  It’s the “Shipping service”, not IShippingService.  Asynchronous messaging is the only way that the Shipping service interacts with other business services — as a subscriber (PaymentReceived message) or as a publisher (OrderShipped message).  Its internal implementation may include web services that talk to one another synchronously or asynchronously, or that wrap legacy, COTS, or external systems (like an SAP shipping module or a UPS-provided web service).  But those web services are not accessible by anything outside of the Shipping service.

The promised benefits of SOA seems much more realizable when you think in terms of business services.

Take loose coupling as one example.  We all know that it’s a good thing, and we are told that web services help us achieve it.  But how loose is the coupling between a web service and its clients?  As long as I don’t change the web service contract, I can change the implementation of the service without affecting any of its clients.  There is no direct binary dependency on the service or its interface.

Alas, in the real world, if I change the business rules inside my web service in any significant way, it’s likely that I’ll also have to change the way that clients use it in subtle or not-so-subtle ways.  The subtle ones are where the real headaches are, especially when I try to get reuse by having multiple clients accessing my web service from slightly different business contexts.

If the client is using synchronous RPC-style calls to the service, it is temporally coupled to the service — it’s has to wait for the service to finish its work before it can continue.  Even if the service uses asynchronous, one-way operations, all too frequently it is receiving what Bill calls command messages, where the client is instructing the service to take some action.  That’s a form of coupling, as well — the client can’t fulfill its responsibilities without the service.

Business services interact with other business services solely through asynchronous messaging — primarily event messages.  That’s about as loose as it gets.

None of this is to say that web services are bad in any way.  They’re a powerful tool in our developer toolbox, and I love what WCF does for me.  But web services are an implementation detail within business services.

This post is just to say a big thanks to Udi and to Headspring for sponsoring the course.  It was a great week, and I was lucky to meet a bunch of great people.  In addition to Udi: Aaron Jensen, Jacob Lewallen, Jimmy Bogard, Blake Carroway, Eric Hexter, and some other very nice guys.  It was a bit humbling to be in a small room with all of that intellectual horsepower.

I’ll echo Aaron and say that if you ever have the chance to take this course — or just to talk to Udi and absorb some of what’s in his big brain — do it!  And now that he’s an expert on chicken fried steak, don’t forget to ask him about that, too.