Every IT generation has its seminal tome that transcends time and connects the dots in a way that no book had before it. For the object oriented generation in the 1980s, it was the Gang of Four (GoF) book. For the application architecture generation in the 1990s, it was Fowler’s book on patterns (PoEAA). “RESTful Web Services” will be, in my opinion, that book for the 2000s Web services generation.

There is something absolutely special about this book that readers of GoF or PoEAA will immediately recognize and appreciate. The book covers a breadth of technologies and ideas yet it helps the reader see how they all connect. It uses short code samples (in Ruby, the choice of this generation) to illustrate rather than obfuscate the ideas. Most importantly, it makes the complex comprehensible and delivers epiphany-like experiences throughout the book.

There are too many highlights in this book to enumerate in this review. However, some of the coverage that I appreciated most included:

• The chapters on resource-oriented design, since there was practically no written information available on this topic prior to this book
• The chapter on resource-oriented best practices
• An overview of the service building blocks, including the different representational formats and WADL, which I wasn’t aware of
• The chapter comparing and contrasting RESTful services with the “Big” (e.g. SOAP) service overhead that is common in most enterprise environments

I would have liked to see this book touch on simple POX versus true REST and handle the resource-oriented security concerns in a bit more detail but you can only ask so much of any one book. I’m fairly confident that “RESTful Web Services”, like the seminal tomes that have gone before it, will become assumed reading 

One of the most common misunderstandings or missteps in following a particular software process is to follow that process in a blind, one-size-fits all fashion. This is especially true if you are mandated organizationally to use a particular software process. Just as the founding fathers could not have foreseen the challenges of the modern world when authoring the Constitution, there is no way that the creators of a generic software process could understand the nuances of every particular project where their process would be used.

 

Processes need to be malleable so that they don’t incite the “baby with the bathwater” response in which the entire process is tossed aside because it proves too inflexible. Good processes can and should be customized. There are several methods that can be used to achieve this end. One of my favorite articles on this topic is an oldie from Per Kroll (“Dr. Process”), entitled How Can We Customize the RUP? In this article, Per lays out 3 basic methods that can be used to customize the RUP:

 

1. Make comprehensive changes to the RUP with the Process Workbench (Heavy)
2. Produce a custom RUP configuration using RUP builder (Lighter)
3. Create your own development case and place it in the project-specific Web site. (BINGO)

First reading this article caused me to re-examine the development case, an often overlooked project artifact. I fell in love with the idea of the development case but the RUP and online examples I could find at the time didn’t live up to my expectations. Between online examples, my ideas, and a great section on the development case in the book Software Development for Small Teams: A RUP-Centric Approach, I pulled together a template that I’ve carried with me, more or less unchanged, until this very day. I’ve included a copy of this template for you to download. The version that is included in Pennsylvania’s BSCoE Software Engineering Process (SEP) is very similar in many ways. You can find that version in the center ring of the online SEP.

 

 

The beauty of the development case is that: (1) it’s very lightweight, just a Word file; (2) it works equally well with a commercial process tool or with no process tool; (3) facilitating a meeting to walk through a development case forces people to communicate about what will and won’t work for them and to make decisions that often have very beneficial impacts for their projects. The best part of the development case is that it works very much like object-oriented inheritance. Customizations from higher levels in an organization can trickle down to lower layers of an organization with customizations and/or additions being provided at every layer where this is required to add value.

If you work in the IT world today, you have a greater chance of not finding a Starbucks on a randomly selected block in Manhattan than you do of not hearing the term “SOA” during your daily workday. It’s unfortunately not nearly as often that I hear or read something about SOA that sounds reasonable, practicable, and overcomes my otherwise skeptic view of all the hype that so often masquerades as SOA. The recent ITConverations Podcast with Todd Biske and Ed Vasquez from MomentumSI is just the sort of real-world advice that avoids the hype and gives a heavy dose of ground truth reality.

Todd and Ed offer some pragmatic advice with respect to SOA and highlight challenges that really resonate with me based upon my experiences. Amongst these are the lack of a single design time and run time governance toolset and the importance of the mindshift from developing enterprise applications to developing reusable enterprise functionality that can be leveraged by different applications. Most importantly, however, is their insight in the importance of thinking of services as products. I have found this to be one of the single largest hurdles to achieving reuse – be it component-based or service-based reuse. Proficiency in application development in no way guarantees proficiency in product-based service development. In fact, most application development organizations that I have dealt with have little or no experience with product-based development. I couldn’t possibly cover all of the disciplines needed to master product-based development in this short post but I’ll refer you to a couple of sources that I feel give more than an adequate introduction.

• Beyond Software Architecture: Creating and Sustaining Winning Solutions – Despite the fact that I only gave it 3 stars in my Amazon review, this is the definitive read on product-based development.
• Framework Design Guidelines: Conventions, Idioms, and Patterns for Reusable .NET Libraries – Although ostensively a book on framework design, this book gives some great guidance on developing an API to meet the needs of its users, not the needs of its designers.
• Enabling the Service Oriented Enterprise – Not much product development here. This article, however, fills in the missing pieces of product development that are exclusive to services.

I plan on recommending this Podcast to a number of people that I work with or have worked with in the past. I can’t expect everyone that I deal with to spend time digesting countless books and articles on the topic of SOA. As a matter of fact, given some of the materials out there, this might just prove more confusing than useful. An hour of time to listen to a Podcast (especially this one) isn’t too much to ask though. You’ll learn an awful lot in this hour.

Technologists who spend their time working on line of business projects are typically exposed to subtle and not-so-subtle messages about business ethics. Most recently, the implementation of the Sarbanes-Oxley Act has touched most of our lives in some way, shape, or form. At a bare minimum, it has heightened our awareness of how terribly wrong things can go when the trust afforded certain practices (accounting and reporting, in the case of SOX) proves to be misplaced.

As strong as the reaction to Enron, Tyco, and WorldCom debacles was; it surprises me again and again to see just how willing the corporate community is to shrug off major IT project failures. This is especially true when you compare the strict regulations, codes of conduct, and ethical guidelines in place for the public accounting community and the almost complete lack of anything resembling these structures in the professional software engineering community. Enron and Tyco may be examples of otherwise ample process controls gone awry. In the software engineering community, we don’t even have a process.

The recent IT conversations Podcast on the utter failure of the project to stand up the FBI Virtual Case File system reinforced this point for me. For those of you who want to get the down and dirty on the FBI Virtual Case File story, the IEEE has a pretty good article containing what appears to be a fairly objective history of the project. “So what does this have to do with ethics?”, you ask.

I think most laypeople, let alone professional software engineers, perceive about halfway through the IEEE article that this project was a train wreck in the making. So why was this not apparent to the 200 plus contractors and countless FBI staff involved in the project? Have we, as a profession, been conditioned into silence or is virtually every software project so fraught with journeys into uncharted waters that we have become ethically ambivalent to the dangers?

The real “story within the story” here is the tale of Matthew Patton, a security contractor who, despondent about lack of management concern, posted his concerns about the project to a Web-based bulletin board. The Web post, which has been archived online, lead to an FBI investigation, denial of his clearance, and a situation in which Mr.Patton had little choice but to resign. Perhaps it’s just me but I find it a bit ironic that Mr.Patton was forced out of the FBI project in the same year that an FBI employee made it onto Time magazine’s “Person of the Year” cover along with whistleblowers from Enron and WorldCom.

There have been movements in our profession (if I can refer to it as such) to institute a code of ethics and similar measures. Both IEEE and the Association of Computing Machinery (ACM) support a unified software engineering code of ethics. With membership in these organizations representing just a fraction of the population engaged in software engineering, this code has nowhere near the professional impact of similar measures released by organizations such as the American Institute of Certified Public Accountants (AICPA). Where does this leave us as software engineers?

• We need more people like Matthew Patton amongst our ranks because there are certainly more train wrecks like the FBI Virtual Case File system just waiting to happen.

• We have to start taking software engineering as a profession seriously. The fact that late and over-budget projects have come to represent the norm needs to change.

• We need to think long and hard about how to police ourselves as a profession. Accounting and auditing tried, failed, and then Congress stepped in and told them how to run the show as part of Sarbanes-Oxley. If we don’t figure out how to do this ourselves, someone will tell us how we have to run our show.

We as software engineers are bound by the same laws that we created. There are no silver bullets for the issue of ethics in software engineering. There is only the fact that a lot of software projects are delivered late, over-budget, or both. It is our ethical obligation to speak up or support those who do speak up and call out the doomed death-march projects before they’ve done too much harm – to us, to our organizations, and to our profession.

I’ve added a second state government pattlet to the portfolio. This one is for the case entity. Although a bit more simplistic than our previous pattlet, Case still has some very interesting nuances, such as the use of aspect-oriented techniques to account for associating the writing of case history records with various case-related transactions.

In addition, I have made some minor revisions to the Case Transfer pattlet, upped the version number, and reposted that as well. I’ve also started added these pattlets to two open source pattern repositories.

On PatternShare.org, these two pattlets were the first examples to be added under the business architecture category. I’ve found PatternShare to be very usable and friendly. In essence, it’s a Wiki for patterns. Unfortunately, after the initial loading of the well-known patterns (e.g. GoF, Fowler), there has been little activity in this community, which is a real shame. I was especially disappointed at the non existence of business patterns, since this is an area where I see so much potential.

On et.gov, the process of adding and mapping a pattern was very interesting. You are actually required to submit create and submit an XML mapping describing your asset(s) and then post this XML file out to your site. The XML file I submitted can be found here. The underlying assets seem to be mapped against the FEA. In many ways, this process is similar to the one I described in my reusable asset post. The only difference is that the discovery mechanism in et.gov is very rudimentary, not near that of some of the commercial tools out there.

As before, the pattlet is licensed under the Creative Commons “Attribution Share-Alike” license. As a next step, I’m looking to move outside of case management for a while and potentially deal with participants. I’m especially interested in working on a Participant Relationship pattlet, since I’ve been having a lot of discussions in that area as of late. I’d also like to fully open source all of this work, including the source code, and post it out to a Subversion repository for people to download. There just don’t seem to be enough hours in the day to get this all done. Any help or assistance would be greatly appreciated.

Case.pdf (165.74 KB)

In my efforts to understand what drives software reuse or the lack thereof, I’ve been looking for formal reuse standards, processes, and practices. I’ve been examining technology agnostic materials as well as researching the approaches taken by each of the respective major camps – software factories for Microsoft and Model Driven Development (MDD) from the Java community. This post, in particular, is more concerned with technology agnostic materials.

 

Being a long time believer in Scott Ambler’s Enterprise Unified Process (EUP), the strategic reuse discipline contained in the EUP was the first place I turned for process guidance. The reuse discipline, like the rest of the EUP disciplines, is structured in the same fashion as any other RUP discipline, which makes learning pretty intuitive if you’re familiar with the RUP. The workflow of the strategic reuse discipline is illustrated in the image below. The workflow is pretty straightforward and the harvest, buy, build, evolve approach to the preparation of assets agrees with what I’ve observed in practice. If you’re interested in this process, you can check out a brief synopsis of the strategic reuse discipline or purchases Scott’s book on the EUP.

 

EUPStrategicReuseWorkflow.jpg (37.51 KB)

The Reusable Asset Specification (RAS) is an OMG standard that I have been recommending to colleagues for some while now. In terms of readability, the document is rather dry and academic in nature. In terms of real world applicability, I am aware of several systems that claim to offer RAS compliance. This includes the tool that we use, Logidex, which offers a RAS compliant plugin for Rational XDE. Where the document shines is not in readability or direct applicability but in the way it holistically addresses the capture of metadata about reusable assets. I’ve been recommending that folks read this document with a focus on the UML and XSD diagrams. There is a lot that can be learned about structuring asset metadata that will prove valuable to anyone attempting to classify their software development assets.

 

As a footnote, the IBM article on reusable assets, recipes and patterns provides a good introduction to the MDD approach, as espouse by one of the major Java vendors. Jack Greenfield’s article on software factories is not all that far removed from the techniques espoused by IBM but, as always, the tooling is quite different.

This weekend, I set out on the daunting task of trying to add an Apache Web server to my existing Windows 2003 production server installation. “Why would you go about doing such a crazy thing”, you might ask. The answer is that, in short, it’s the only way to host an HTTP-accessible instance of Subversion on a Windows box. I’m looking to consolidate all of my hosted software: .NET, Java, and infrastructure, onto a single platform. Since I use Subversion to enable my location independence with respect to computers I use, this application needed to be ported as well.

I expected to learn a lot of new stuff about Subversion and Apache through this process. I didn’t expect (perhaps ignorantly so) that I’d encounter “learning opportunities” on Windows as well. As it turns out, Microsoft has gone ahead and made the assumption that if you’re running IIS on your Windows Server, nothing else should be running on port 80. That is, out of the box, IIS exhibits very greedy behavior, binding to all available IP addresses on the server, not just the ones that are explicitly assigned to Web sites.

It turns out that this is a case of socket pooling gone awry. To solve this problem, you must literally slap IIS on the hand and tell it to stop listening to all of those ports. After trying a couple of legacy solutions, such as setting the DisableSocketPooling property, I landed upon a solution that works in the IIS 6.0 post-Winsock era. This Microsoft Support Article provides all of the details you would need to manage the IP inclusion list in IIS 6.0 and solve this problem should you run into it. Also, should you need the Windows 2003 support tools referenced in this article but not have immediate access to your Windows 2003 CD, you can find the tools here as well.

I’m happy to report that I now have IIS and Apache running side-by-side on the same Windows machine; each on port 80 using different IP addresses. Getting Subversion up and running was another issue entirely…

I ran across some very unique work from the state of Missouri in the area of project oversight the other day. Missouri’s well documented approach to project oversight is not only a great state government practice; it is by far the best documented practice in the public or private sector that I could find in this area. The project oversight methodology was nominated for a NASCIO recognition award in 2004 under the State IT Management category.

The five guidance documents on Missouri’s project oversight website represent a comprehensive, end-to-end project oversight methodology. In addition to detailed process flows, artifact definitions and templates, the methodology also includes simple but very useful explanations, such as the table illustrating the differences between project management and project oversight, which I’ve reproduced below.

ProjectOversight.png (7 KB)

In my mind, items such as Missouri’s project oversight approach are the types of knowledge capital that would have been great seeds on which to found the Government Open Code Collaborative (GOCC), which I’ve blogged about in the past. Since the GOCC appears to be all but dead, I don’t know what other alternatives are available for inter-State exchange of government practices, outside of NASCIO. I’ve decided to mirror Missouri’s documents on my site and have included links below. Hopefully, Missouri can find a way to proactively open source these documents, even if no common exchange exists for such materials.

Project Oversight, Part 1 (598 KB)
Project Oversight, Part 2 - Chapter 1 (2086.5 KB)
Project Oversight, Part 2 - Chapter 2 (2019.5 KB)
Project Oversight, Part 2 - Chapter 3a (1820.5 KB)
Project Oversight, Part 2 - Chapter 4 (1042 KB)

Over the last several months, I’ve really been trying to get my arms around SOA and develop a meaningful opinion and knowledge base on this so often used, even more often abused, and ever-more-frequently maligned three letter acronym (TLA). Along the way, I’ve discovered a couple of great resources that have helped shape my thinking and hone my implementation skills on the topic:

• Prentice Hall’s Service-Oriented Computing Series - Right now this series consists of just two books by Thomas Erl, “Service-Oriented Architecture: Concepts, Technology, and Design” and “Service-Oriented Architecture: A Field Guide to Integrating XML and Web Services”. These two books, however, are tantamount to the old and new testament; literally comprising the SOA bible. The series is also slated to be expanded into a more complete collection, along the lines of Addison-Wesley’s Martin Fowler and Kent Beck signature series.

Thomas’s books have helped me to understand how my traditional proxy and wrapper-based viewpoints fit into service design and how I might be able to improve the robustness of SOA interfaces built using these patterns. These books also reinforced my positive experiences with contract-driven design and have rekindled my interest in XML schema definitions, which I haven’t used extensively for years.

• Enterprise Service Orientation Maturity Model (ESOMM) – “Maturity model… uh oh, here comes the heavy handed approach to SOA”, you might be thinking. In my opinion, however, this is the most dense (that is, succinct and knowledge rich) piece of material about SOA that has been published to date and a must read for anyone looking to role out SOA to their enterprise. The ESOMM defines 4 layers, 3 perspectives, and 27 capabilities required to support a SOA (see diagram below). The maturity levels are based upon SEI’s Capability Maturity Model (CMM), but the similarities pretty much end there. As with the core CMM, think of this as a roadmap towards evaluating and improving your organization’s SOA capabilities – not as a report card.

   
esommgif.gif (32.31 KB)   

• Service-Oriented Analysis and Design (SOAD) – This is a nifty article which seeks to bridge the gaps between the object oriented and business process oriented design and modeling and the requirements of modeling for an SOA. The article does a great job of walking through traditional approaches that most people are familiar with and then adding SOAD-specific elements to the design. The article concludes with a short case study that includes traditional models such as a business process workflow, class diagram, and state diagram and then augmenting this with a service breakdown model and a rather interesting business interaction model (see diagram below) that integrates SOA specific concerns into a more traditional UML sequence diagram.

SOAD_LG.gif (29.3 KB)

• Java and .NET Specific Implementation Materials – Jeffrey Hasan’s book Expert Service-Oriented Architecture in C# is by far the best text in the .NET realm. Jeffrey starts with a very solid approach of WSDL and XSD contract driven design and then gradually introduces the new WS-* standards, integrating them one-by-one. The original book covers WSE 2.0 with his newest text covering WSE 3.0. SOA Integration Using Java EE 5 Web Services is the best text that I’ve found from a Java vantage point.  I like the fact that this book starts out with REST (Representational State Transfer) type services to show how things look before all of the standards-compliant overhead is added. Good coverage is provided for JAX-WS 2.0, JAXB, and JSR-compliant packaging. The book is not yet available on the open market but the work in progress is available as a “Rough Cut” book through O’Reilly’s Safari.

      

• BPEL-Specific Materials – BPEL specific implementation details haven’t seemed to make it into any published books yet. The vendors offer a good deal of online materials in this area. To learn it this way though, you’re going to have to commit yourself to a particular implementation. Good materials can be found in the following places:

o     JBoss - JBoss jBPM

  o     Sun - NetBeans Enterprise Pack

o     Oracle - BPEL Process Manager

o       .NET - Windows Workflow Foundation

I’ve recently been re-reading Scott Ambler’s excellent work on the Enterprise Unified Process and focusing my attention on the strategic reuse discipline, in particular. Dealing with this on a day-in, day-out basis, I’m trying to apply this particularly to the business domain that I work in, state government. I like the way that Scott went about illustrating the enterprise management disciplines with the traditional RUP workflow maps. Above and beyond this, he borrows from an earlier article of his, A Realistic Look at Object-Oriented Reuse, to create a couple of diagrams that really hit home. I’ve taken the opportunity to adopt these diagrams to enterprise work being done in state government. The adopted diagram can be found below.

Reuse.png (79.92 KB)

The comments to the right of the chart demonstrate that there are three coarsely delineated segments of reuse:

1. Areas where domain-specific reuse is occurring now. Unfortunately, these areas fall towards the lower end of the benefit spectrum. In this segment, wholesale cross-state system transfers are particularly widespread. In many cases, the cost of the system being transferred is negligible or nonexistent due to federal funding rules. However, it is rare when these systems can be transferred without acquiring the knowledge of the people that created them. In many cases, this is an acceptable trade off because the domain in questions is inhospitable to other solutions and orthogonal to the domain for which they were originally created. Make no mistake, however, this is more a reuse of human resources and knowledge capital than it is of system elements.
2. Areas where domain-specific reuse is not warranted. In the center of the above chart lie frameworks and artifacts. With industry standard processes such as the RUP, MSF, and a variety of Agile processes, there is little need to create domain-specific processes or artifacts. The same holds true for frameworks, where Sun, Microsoft, and a variety of vendors and open source options are likely to be much more stable and better supported than any inhouse created options. This should not preclude you, however, from customizing these processes and frameworks and, as necessary, extending them to meet your exact needs.
3. Areas where domain-specific reuse should be occurring. These are the higher-order levels of reuse. This includes domain-specific analysis patterns for state government as well as a variety of architected solutions. These solutions may include COTS products such as the one from Curam Software, solutions targeted as cross-cutting domains such as licensing, and leveraging emerging data exchange standards to encourage the sharing of knowledge and data between business partners and communities of practice.

It’s been a while since I’ve posted my last entry. To get myself back in the groove of things, I thought that it might be nice to post something lighthearted and entertaining that your average tech weenie would enjoy. Now I don’t know if you follow the gurus of the technology world but my research has turned up a set of long lost brothers amongst the talking (or blogging) heads. Check out the two pictures below. The one on the left is Bruce Schneier, cryptography and computer security wunderkind and designer of several cryptographic algorithms, including Blowfish and Twofish. The one on the right is Martin Fowler, refactoring, pattern, and agile god.

I couldn’t believe it myself when I first saw this but a side-by-side of these two gentlemen makes things perfectly clear. There’s got to be some common lineage here. It’s crazy to think about what our world might have been blessed with were these two great minds not separated at birth. Agile cryptography? Algorithmic refactoring?

I enjoyed Harry Piereson’s well thought-out response to David Chappel’s entry on SOA and the Reality of Reuse. I couldn't have said it better myself, though that won’t stop me from trying. The way I see it, David brings to light the fact that the emperor has no clothes and then Harry tells you why the emperor is naked. The focus on business context in Harry’s entry really caused me to think about why business logic reuse fails.

The use of the word “context” stirred up in my mind the classic black box, gray box, white box argument. We can expect Java buttons or .NET Windows objects to behave in a low context, black box manner. From contextual business objects, we can at best expect gray box behavior; although white box is much more realistic. I guess in the box world, the opacity of the box is tantamount to the amount of context that can get through.

This said, in business object environments that demand high opacity/context, can and should we strive for reuse? I think the answer is still a resounding “yes”. We just need to do it in a realistic manner. So what does this mean? If our experience has shown that neither objects nor Web services are the appropriate level of abstraction for business logic reuse, where do we turn?

In my mind, analysis patterns provide the answer to this question. Analysis patterns are the long lost stepbrother of the popular design patterns. Martin Fowler does a great job of categorizing common analysis patterns in his book of the same name. However, it is Eric Evan’s Domain Driven Design book that provides the real insight into the process that will lead to analysis pattern identification.

The true beauty of analysis patterns is that they not only work within their target domain; they have the uncanny ability to shed new light on similar situations across domains. I can well imagine that the case transfer pattlet we created applies not only to the state government business domain for which it was intended, but also fairly well to the transfer of cases in the legal system domain.

In such high context, business rule driven transactions such as case transfers, I would never aspire to build a reusable object or service. Letting the analysis pattern speak for itself, I feel as if I’ve gotten all of the reuse capability I need to at that level. Attempting to create a standard programmatic solution that incorporates all the complexities of the pattlet’s context is likely to cause more problems than it solves. As caretakers of the business logic we need to understand the negative impacts of overengineering a solution and learn to use the right colored box for the job.

Following up on a long-standing desire to get domain knowledge out of our heads and onto paper, a colleague and I engaged in writing our first state government pattlet. We spent about two weeks of our spare time putting together an abstract approach to case transfer based upon our varied experiences. We finally have a draft version which we feel comfortable sharing online.

As far as I know, this is a first-of-its-kind endeavor for state government. We drew heavily on Fowler’s Analysis Patterns: Reusable Object Models as background material for documenting the patterns. The underlying analysis, design, and approach are all original, though. Please understand that the pattlet is not perfect. We’ve marked it as a 0.1 version to reflect its state and we intend to update it over the next couple of weeks.

We would really appreciate your feedback on the pattlet. Specifically, we’re looking to understand if there are important case transfer nuances which we missed, flaws in the analysis and design, or significant domain details which might shed new light on how to approach this problem. I know that most people feel very comfortable with email but I ask you to please consider using the comments section on this page. That way, everyone can benefit from your insights.

Note that the pattlet is licensed under the Creative Commons “Attribution Share-Alike” license. This license falls along the lines of common open source software licenses, allowing you to modify and re-disseminate this as you see fit, so long as you attribute the original work to us and any new work goes out under the same license as the original. We plan on distributing future pattlets under the same license. It seems to be a happy compromise between rights and reuse.

Right now, we are only giving access to the final product in PDF format but we have kicked around the idea of hosting sample code, pattern documents, and design models on a Subversion share. Let us know if this would be interesting or if we’d just be wasting our time in this endeavor. Finally, you can find the pattlet by clicking on the link below.

Case Transfer.pdf (164.24 KB)

In addition to the technical assets (Java Framework) that I’ve mentioned in previous blog postings, BSCoE also makes a set of software process assets available. These software process assets are arranged into disciplines and collected under the umbrella of BSCoE’s Software Engineering Process (SEP). The BSCoE SEP is available online to the general public in either a browsable or downloadable format.

The SEP is based roughly upon the Rational Unified Process (RUP) and Microsoft Solutions Framework (MSF). Those looking at the sample assets will notice the similarities with the standard RUP templates. The process component of the SEP is specifically vague, leaving decisions such as formality versus agility, process activities, and roles to the projects employing the SEP. In particular, projects have several options for SEP customization including document-driven (RUP-style development case), local modifications to the process, or modifications with intent to contribute back changes to BSCoE for inclusion in the master SEP distribution. The SEP is conceptually similar in some ways to Ivar Jacobson’s new Essential Unified Process (EssUP). However, whereas the EssUP variability comes through the selection of practices, SEP’s variability comes through the selection of artifacts. 

For those of you thinking of creating your own software processes, I can only recommend the experience. Using the RUP and the Rational Method Composer or the MSF and the new Team System templates, although educational, is not an easy process. This is especially true if you intend to employ lightweight, agile processes on your projects. In this case, modifying the base processes from the RUP or MSF is tantamount to carving a chess piece from a sculpture sized block of marble.

Fortunately, as always, there are many that have gone before us in this endeavor. The sources highlighted below were of great help to us in the creation of our process and are highly recommended whether you require guidance or are just looking to better understand software engineering processes in general:

• Scott Ambler’s Writings – This guy is truly the best place to start when you’re looking for anything process related. All roads will eventually lead through his work. Particularly interesting are Scott’s Agile Unified Process and his Enterprise Unified Process. The former is a lightweight version of the RUP focusing on test driven development, agile modeling, agile database techniques, and refactoring. The latter is an extension to the RUP covering enterprise disciplines not mentioned in conjunction with many software processes such as portfolio management, enterprise architecture, strategic reuse, and software process improvement. In addition, it adds two new phases to the RUP, production and retirement.
• Philippe Kruchten’s Books – The two Addison-Wesley Professional books Rational Unified Process Made Easy and Rational Unified Process: An Introduction are seminal works on the RUP. Often overlooked is Software Engineering Processes: With the UPEDU, a book that runs through an educational version of the Unified Process with some pretty decent explanations and online examples.
• Craig Larman’s Books – Craig’s books, although not focused on process engineering or the intricacies of processes, convey an awful lot of information in real world contexts. Agile and Iterative Development: A Manager’s Guides is one of the best ways for non-techie types to get their arms around lower ceremony development processes. Applying UML and Patterns focuses on modeling and design patterns but does so in the context of a process using the Unified Process infused with Agile methods, making it a source of great contextual information.

In addition, we have been picking up reference artifacts along the way to illustrate best practices and real world examples. One of the artifacts that I fell in love with was the Yummy SAD, available online as HTML and downloadable in document format via FTP. This is one of the most generic, understandable, and widely-applicable examples of a software architecture document (SAD) that I have come across. One of the big selling points of the Yummy SAD is that aside from just being an artifact, it also comes with an approach to architectural decomposition. In particular, it espouses an architecturally significant use case / quality attributes based approach to documenting and realizing your architecture. Explaining software architecture in this fashion has helped me clarify the relevance and importance of the SAD on more than one occasion.

Please feel free to share any best practice artifact references or software process tips that you might help accumulated over the years in the comments section below.

I tried answering Tad Anderson’s comment within the bounds of my August 8^th posting but eventually decided that this topic really warranted a posting of its own. My thoughts around using Extreme Programming and other Agile approaches to software development are pretty well formed. As an ex-soldier, military analogies seem to work particularly well for me:

• On the high end, the truly capable XP team is small, lightweight, and meets the requirements Tad set forth in his post. In many ways, effective Agile teams parallel our military’s elite special forces (e.g. SEALs, Green Berets, Delta Force, etc).
• In the middle, there are well-trained, capable teams that practice UP, ICONIX, and other iterative processes with varying degrees of agility. These teams may infuse certain XP practices into their process as they are needed. The individuals on these teams may have the mettle to be elite special forces. However, due to project requirements, team size, or other factors they are not “active operators”, in military parlance. These folks operate in a fashion analogous to our highly-skilled tactical units, such as the 82^nd Airborne, 10^th Mountain, and the like.
• On the lower end (of the agility scale, that is) are teams that mirror our more traditional Army units: infantry, mechanized infantry, armor, artillery, etc. Their capabilities are more geared toward larger, more structured engagements. Backing these capabilities are detailed approaches and tactics, significantly greater support and infrastructure requirements, and longer lead times to get a team on the ground to effectively engage the problem.

So where does this leave us? To me, at least, it’s pretty clear that, on software projects, as in combat, there is no one-size-fits-all approach. I would no more likely try to tackle building an air traffic control system with XP than I would send in a special forces team to face off against a Soviet armored column. Analogously, I would neither call in the third armored division to handle a jungle-based guerilla insurgency nor try to use CMM level 5 processes to build a simple Web-based e-commerce application.

Deciding what type of software development process to use is ultimately a managerial responsibility. From this decision, to paraphrase a saying, will come 90% of your happiness or misery. Making the right decision requires that the manager has an intimate understanding of his team’s capabilities, understands the scope of the project, and has correctly assessed the level of involvement that can be expected from project stakeholders. This knowledge is, by no means, easy to come by; which helps explain why there is so often a dissonance between the approach applied and the one required. Just like with a SEAL team, there are limitations on the number of individuals that have the raw capabilities to be effective members of an Agile A-Team. If you done an honest assessment and are certain that the Agile approach is right for the project and you have an A-Team, then go for it.

As a footnote, this military analogy brings to light interesting questions of contingent or complementary force parallels in software engineering. That is, Operation Iraqi Freedom was fought with a mix of closely coordinated heavy armor, airborne, and special forces units. I can imagine that within an enterprise, or even within a project, there might be room for similar collaboration between high ceremony teams, medium-weight iterative teams, and light-weight agile teams. I’ve started to see the first signs of this on projects and would be interested to hear your take on this.

I received an email from one of my clients a couple of days back referring to a quote of mine on Cenqua’s website under their Clover.NET product. This brought me back to an article I wrote for Dr.Dobb’s Journal when build tools, unit testing products, and continuous integration were making their way from the Java world over to .NET. I decided to meander over to Dr.Dobb’s site (I still call it a site although a barrage of emails from CMP Media constantly reminds me that it’s a new and improved portal) and see if the article is still available.

Googling around, I was quickly able to locate the articles, which were published in two parts: Part I and Part II. Reading through the articles, I reflected on how far we’ve come in the .NET community since then. Continuous Integration (CI) and the tools supporting it have been embraced by the community and have found fairly widespread acceptance on commercial and open source products alike. In fact, the pendulum has swung so far in this direction that Microsoft decided that it was time that they undertook the effort to re-create these tools and repackage them within proprietary Microsoft products. The ultimate incarnation of this effort is Team System, which represents Continuous Integration and everything else tangentially related to the software development process rolled into one… and then given a shot of steroids - the proverbial 500 pound gorilla.

If you were thinking that the previous statement was a ramp-up to a bout of low-brow Microsoft bashing, you’d be wrong. Sure, Team System is huge and some of its beta versions were more like pre-Alpha. The last couple of demos I’ve seen, running off of laptops, I’ll add, looked not only fairly stable but pretty darn interesting. Since it was engineered from the ground up as an integrated suite of tools, the integration points appear much more polished than they do with the Rational Enterprise Suite. I can’t speak for how well the product works or for the validity of the third party extension approach. I can, however, attest to hearing quite a bit of interest from clients in the new product. If Microsoft would clearly articulate its strategy for integrating modeling tools with Team System, I would go as far as to say they’d had themselves a single and were picking up speed and heading for second.

If you pull back the covers on Team System, you’ll recognize all of the components that make up your standard Continuous Integration environment; the same ones I wrote about in the article. They’re all there and then some: a build engine, unit testing tool, code coverage reporting, static code analysis, and a completely revamped VSS version control system. This overt infringement on the open source’s domain might rub you the wrong way if you’re a proponent of the open source tools. I still love (and regularly employ or recommend) these tools but I’m open to the way that Microsoft went about making these ideas their own.

Employing James Newkirk, the creator of NUnit, to design the Team System unit testing tool leverages the lessons learned in building and maintaining NUnit and recognizes a valuable member of the open source community for his contributions. The story of a phenomenal documentation tool, NDoc, ends on a less positive note. Whether or not Sandcastle (Micrsoft’s codename for their new documentation tool) would have otherwise spelled the end for NDoc will never be known. The utter lack of financial and maintenance support for NDoc left its creator, Kevin Downs, frustrated enough to relinquish NDoc project ownership. Thankfully, Microsoft has picked up the ball here and I’m sure that once the bugs are worked out, we’ll be able to easily churn out documentation for our .NET 2.0-specific code.

“So where does this leave us?” you might ask. It leaves us with options. Choose to use the open source tools or their Team System counterparts. If you’re feeling particularly lucky, you can write some custom build script tasks and mix and mingle the two. Either way, acceptance of continuous integration will continue to grow in the .NET community. Furthermore, CI will continue to be augmented by tools of ever-increasing sophistication, be it Team System from Microsoft or a Trac or NProject-like variant from the open source community. This is good news for those employing the .NET platform or those considering a move in that direction.

Post Footnotes: In case you’ve been living under a rock for the last 5 years, Fowler’s explanation of continuous integration is considered the seminal work and will get you up to speed quickly. The image above comes from my article and, while not nearly as eloquent or enjoyable as Martin’s explanation; I’ve found it a good and quick way to explain CI to absolute beginners and pointy-haired boss types alike. I’ve included a link to my original code below for the sake of completeness. My apologies in advance since it’s a bit dated.

CI SRC.zip (1.27 MB)