In the final installment of this March 1^st trilogy, I’ll hit something really topical, the precipitous fall of the Dow Jones Industrial Average (DJIA) two days ago on February 26^th. For those of you just diligently minding your stakes in the market, you probably got wind that all the major indexes tanked pretty badly on the 26^th. The Dow, however, tanked in a rather unnatural fashion that was quite different from the others.

You can see the “Live” and recalculated Dow Jones numbers on the graph above. Although the actual  Dow (in orange) did drop pretty steeply, there’s no way it could have done the near vertical drop that the graph indicates without some outside intervention. It turns out that this outside intervention was provided by a “computer glitch”, as reported in the general press. More specifically, one of the queuing systems that feeds data to the system that actually calculates the Dow Jones index got backed up under record trading volumes. When market watchers got suspicious that the index movement did not align with the sliding values of its core components, they decided to investigate. Discovering the backlog, they manually switched over to alternate systems. When these systems worked down the backload, a new DJIA was calculated and voila, we had a 200 point drop in a couple of minutes.

As painful as this must have been for professional traders and as painful as it’s likely to be (in terms of lawsuits) for the Dow Jones company, this is really interesting from an IT point of view. If you’re an IT person looking for insight into how these markets work, I thoroughly recommend the book Practical .NET for Financial Markets. It’s one of the most advanced .NET texts I’ve ever read and, as a positive side effect, introduced me to a completely unfamiliar business domain, financial markets. Reading this book will not only make you a more educated developer (whether or not you use .NET), it will help you understand the causes and effects of events like those on the DOW two days ago.

If you work in the IT world today, you have a greater chance of not finding a Starbucks on a randomly selected block in Manhattan than you do of not hearing the term “SOA” during your daily workday. It’s unfortunately not nearly as often that I hear or read something about SOA that sounds reasonable, practicable, and overcomes my otherwise skeptic view of all the hype that so often masquerades as SOA. The recent ITConverations Podcast with Todd Biske and Ed Vasquez from MomentumSI is just the sort of real-world advice that avoids the hype and gives a heavy dose of ground truth reality.

Todd and Ed offer some pragmatic advice with respect to SOA and highlight challenges that really resonate with me based upon my experiences. Amongst these are the lack of a single design time and run time governance toolset and the importance of the mindshift from developing enterprise applications to developing reusable enterprise functionality that can be leveraged by different applications. Most importantly, however, is their insight in the importance of thinking of services as products. I have found this to be one of the single largest hurdles to achieving reuse – be it component-based or service-based reuse. Proficiency in application development in no way guarantees proficiency in product-based service development. In fact, most application development organizations that I have dealt with have little or no experience with product-based development. I couldn’t possibly cover all of the disciplines needed to master product-based development in this short post but I’ll refer you to a couple of sources that I feel give more than an adequate introduction.

• Beyond Software Architecture: Creating and Sustaining Winning Solutions – Despite the fact that I only gave it 3 stars in my Amazon review, this is the definitive read on product-based development.
• Framework Design Guidelines: Conventions, Idioms, and Patterns for Reusable .NET Libraries – Although ostensively a book on framework design, this book gives some great guidance on developing an API to meet the needs of its users, not the needs of its designers.
• Enabling the Service Oriented Enterprise – Not much product development here. This article, however, fills in the missing pieces of product development that are exclusive to services.

I plan on recommending this Podcast to a number of people that I work with or have worked with in the past. I can’t expect everyone that I deal with to spend time digesting countless books and articles on the topic of SOA. As a matter of fact, given some of the materials out there, this might just prove more confusing than useful. An hour of time to listen to a Podcast (especially this one) isn’t too much to ask though. You’ll learn an awful lot in this hour.

In the first of my beginning-of-the-month speed blogs, I’d like to start with the topic that I’ve been putting off longest, building applications on top of Amazon’s Web services. For those of you that think of Amazon.com as just a book store, or an e-commerce platform, or [add your assumption here], think again. In the last year or so, Amazon has released a number of services that have not only established them as a first-class platform, but have pushed the envelope considerably on the idea of virtualization.

This trend started with Amazon’s Simple Storage Service (S3) and was expanded on with their Simple Queue Service (S2). Late last year, Amazon blew the roof off with the limited release of their Elastic Compute Cloud (EC2) service. EC2 is a configurable computing environment where you can load up virtual instances (roughly equivalent to your average Linux server) – on demand. You can load or unload these instances to meet cyclical demand and you only get charged for the computing time that you use. Couple this grid computing potential with S3 for storage and S2 for queuing and let your imagination be the limit.

Doug Kaye’s architecture for GigaVox Audio Lite, an audio file transcoder, might give you some ideas about what can be done with these services. The related ITConversations podcast contains plenty of details, with both Kaye and Jeff Barr, from Amazon, on the wire with Phil Windley. This podcast does a good job of conveying the vision and the realities of using these services. Some of the discussions, like managing and developing on top of a globally distributed queuing system where a call like size = MyQueue.size doesn’t really make any sense, will cause you to stop and think.

The final new Amazon service that I didn’t mention is Mechanical Turk, which is, simply put, a computer interface onto human services. This one has been around for a while and never got much attention from me until I heard the Podcast. Interestingly, the site castingwords.com, which produces Podcast transcriptions, farms out their work via Mechanical Turk. The combination - digital input in (MP3); digital input out (electronic transcript); virtual computing; virtual workforce – that’s pretty radical. There are some other great uses for this service mentioned on the Podcast as well.

The Podcast is absolutely worth a listen. And if this sort of thing piques your interest, you’re not alone. The queue to get into the EC2 beta program is a few months long. If you’re even remotely interested, get in line now.

One of my original intents of registering the beckshome.com domain name was to publish photos of my new baby son or daughter. That was two years and two daughters ago and, until this weekend, photos were nowhere to be found on my blog. I host my blog on the Windows platform and had no desire/time to do any of the following: (a) buy a separate package for image management; (b) cobble together an ASP.NET solution to manage my photos; (c) switch blogging software to a tool like Community Server that has integrated photo management. Furthermore, I already manage my photos on Flickr and I’m more than happy with the service, user experience, and the cost-benefit. What I really needed was a way to integrate my existing Flickr photos into my current .NET-based blog (DasBlog). The pursuit of this goal is what this blog entry is all about.

 

Being a regular blog reader, what I’ve seen a lot of out there are the Flickr badges. These badges, available in either HTML or Flash versions (like the one below), are pretty slick and can be found pretty much everywhere on the Web. The problem with these badges is that they only offer the opportunity for shallow integration. Click on the badge and bye-bye blog, you’re zipped off to Flickr’s site to look at the photos. Since I aspired to achieve a bit deeper integration, I needed a different approach.

 

 

Next thing that I looked into was programmatic access to the Flickr API or a pre-existing solution that I could use wholesale or reproduce with little effort on my part. The Great Flickr Tools Collection has a vast assortment of very interesting tools – none of which quite seemed to meet my needs. I checked out the Flickr.NET API Library, which was written about in a Coding4Fun post and can be found for download here. It’s very well done and, although it probably won’t be the last time I mention this API in my blog, it will be the last time I mention it in this posting.

 

What I eventually stumbled on was a simple and elegant solution that got me exactly what I wanted by embedding the Flickr slideshow viewer into a custom page on my existing blog. Paul Stamatiou has an excellent post on his blog on how to do just that. By using an iframe and setting some API attributes you can get this up and running very quickly; qualifying this as a super easy hack that just works. Note that only photos marked as public will be displayed.

 

With the Flickr slideshow viewer up and running, I only needed to add the ability to select between multiple photo groups and I was done. With the actual Flickr viewer taking care of all of the real AJAX work, all that was needed was a bit of light JavaScript to tie this all together. Below you will find the code that does all of the lifting.

 

<script type="text/javascript">

      function changeSlideshow(url, title)

      {

            document.getElementById("SlideShow").src=url;

            document.getElementById("Title").innerHTML=title;

            return false;

      }

</script>

 

Clicking on any of the photo group links / thumbnails makes a call to the above function passing the URL for the slideshow in the manner stipulated in Paul’s article for populating the slideshow viewer. The title is also passed so that the title of the page can be updated. You can see this at work on my new photopage. By viewing the page source, you can see the exact mechanism I used to make this work. If you’re have any questions, feel free to drop me a line.

I realize that the title of this blog post is something that one would probably not associate with a blog that purports to cover technology. Bear with me please and surf over to YouTube (or view below) to look at the video posted by the Pennsylvania Fish and Boat Commission. I think you will agree that the video is very well done. Much like the partnership between Google and the Pennsylvania Tourism Office that I blogged about a couple of months back, this posting to YouTube represents some signs of forward thinking, Web-based focus, and creativity on the part of state government.

 

 

Granted, with tourism and fishing, we are looking at revenue generating sectors of state government. You don’t yet see this sort of thing from Welfare service divisions. How long will it be though until other sectors of state government start looking at public Internet services as an alternate channel for communications with their constituents? The economics of these services surely works in their favor (that is, they’re free). It then becomes a question of what the best channel is to reach one’s constituents.

Face-to-face interaction, snail mail, and government sponsored portals will be around for a while to come, I surmise. But as usage of cell phones, email, inexpensive audio devices and collaborative community-based Internet services like YouTube continue to permeate through the layers of citizenry, state governments will not be able to ignore these viable communication mediums. The one question that remains is whether state governments are prepared to engage in the two way dialog and deal with the power shift that these new communication vehicles will invariably bring.

The innovation engine at Yahoo is heating up, looking to get Yahoo back in the race with the “Big Boys”, rivals Google and Microsoft. In an environment categorized by copycat service offerings and one-upmanship, Yahoo’s offerings are refreshingly unique. I cover three of the most recent services that I’ve been playing around with and that I think will prove entertaining to my readers as well – Pipes, OmniFind Yahoo! Edition, and TagMaps.

• Pipes – No less a luminary than Tim O’Reilly called Pipes “A milestone in the history of the Internet”.  Pipes is a browser-based visual editor that allows you to take input from one source and pipe it (in UNIX parlance) to another source. Along the way, you can apply a series of filters and transformations to manipulate the data. The data sources start and end as common feeds (RSS, RDF, etc). What you do with the data between its input and output is constrained mostly by your imagination.

Yahoo! was certainly not first on the scene with this idea. Dapper and others have preceded them in this regard. What Pipes brings to the party that no one else does is a really cool visual environment that allows you to trace the path of the data through the transformations and filers, interactively debugging along the way based upon the value of the successive outputs. It also has this cool reuse flavor to it, where you can experiment with, tweak, learn from, and potentially improve or fork off new versions of other peoples pipes or just reuse them in a black box sense.

After you’ve read Tim O’Reilly’s introduction, I encourage you to play around with Pipes. Although seeing is believing, you’ll learn best by actually doing.

• OmniFind Yahoo! Edition – Product of a nifty partnership with IBM, the OmniFind Yahoo! Edition is an enterprisey search solution that is the baby brother to IBM’s commercial OmniFind enterprise product. Built on top of the open source Apache Lucene search engine, OmniFind has the solid lineage necessary to be considered worthy of the task.

The product is a very easy install, whether on Windows or Linux, requiring very few steps to get the product up and running. OmniFind returns search results against locally indexed documents and the Internet, with the results being returned in the familiar Yahoo! look and feel. For those interested, the UI can be styled to match a particular site’s look and feel or you can go the direct route and work with the exposed REST APIs.

With the pricetag (free) and support for a couple hundred file types, there’s little not to like about OmniFind. Search performance has proven to be very fast with a few thousand documents. Indications are that the tool scales pretty well although the indexing process can be quite processor intensive and there are a couple of known issues with cleaning up very large temp files that could eat into your available disk space.

• TagMaps – I stumbled onto this product a couple of weeks ago while looking for some information about creating GeoRSS feeds. TagMaps is another way of visualizing data (tags in this case) on maps. I must confess that seeing tags on a map takes a bit of getting used to. I found that the best way to indoctrinate myself was by using Trip Explorer

Trip Explorer is a mashup of TagMaps and Yahoo! Travel users' public Trip Plans. What’s cool about Trip Explorer is that the clustering of tags reveals hidden tour gems that you might not otherwise find on a traditional map mashup. These gems become more evident (and detailed) as you progressively zoom in.

TagMaps is built upon Yahoo’s Flash maps, which are very interesting in their own right and need to be experienced if you haven’t yet had the chance. Aside from using Yahoo’s canned Explorer TagMaps, of which Trip Explorer is one, you can create your own TagMaps mashups. Simply create a GeoRSS feed or select an existing GeoRSS feed that returns a set of weighted tags for a given lat/lon bounded box. Easier said than done, I know. I’ll be writing more about how to do this in a coming blog entry. Until then, give this a look.

I don’t like to do book reviews back-to-back but Founders at Work has kept me pretty busy reading (and not writing) over the last couple of weeks. The book definitely deserves a five star rating and at $13 for the e-book version, it really is a great deal. My review follows…

This is an absolute must read if you’re job, your passion, or both (if you’re lucky) has anything to do with creating technical innovation. “Founders at Work” is a wonderfully meander through the stories of successful company founders – across several decades. Far from focusing on just those who made it big during the first dot-com boom or those who are profiting from Web 2.0, Jessica also includes some of the true pioneers in the field. She recognizes that, not only do these industry veterans have valuable stories to convey but, since many of them are helping to steer companies and venture capital funds to this day, their advice is quite topical and current.

From the great introduction right through the final interview, this book is packed with great anecdotes, advice, information and inspiration. Makes you wonder as to what the story is behind the story - how did Jessica get unfettered access to such a broad array of the founding fathers?

I’ve included some illustrative quotes from the book below. Give them a read and then go pick up this book. The printed copy is a bargain and the e-book version is a steal. It may turn out to be one of the best investments you ever make.

• “You guys are nuts. Throw out your business plan. Your customers—or potential customers - are telling you what your business should be. The business plan was only used to get you the money. Why don’t you rewrite a business plan that is focused just on providing what your customers want?” - Q.T. Wiles advice to Charles Geschke (Cofounder, Adobe) on the real purpose of a business plan
• “There were some warning signs. Consider McKinsey, which holds itself out as one of the world’s leading repositories of knowledge on how to manage a business. They say they’ll never grow their company by more than 25 percent per year, because otherwise it’s just too hard to transmit the corporate culture. So if you’re growing faster than 25 percent a year, you have to ask yourself, ‘What do I know about management that McKinsey doesn’t know?’” – Philip Greenspun (Cofounder, ArsDigita) on scaling corporate culture
• "That [not improving core product quality] was probably the biggest mistake we made. And that’s the advice I give everybody. All those little coupon schemes, this is what General Motors does. They figure out new rebate schemes because they forgot all about how to design cars people want to buy. But when you still remember how to make software people want, great, just improve it." – Joel Spolsky (Cofounder, Fog Creek Software)
• “I think some people slept; I know I didn’t sleep at all.” - Max Levchin (Cofounder, PayPal)
• “There were times when we were really broke before we had our angel investment, when only one guy who had children was getting paid.” – Caterina Fake (Cofounder, Flickr)

With nearly 21 of the 32 interviewees having the term “Cofounder” in their titles, Joel Spolsky’s advice seems perhaps to reflect best on what was critical to the success of these companies. “But because they never really take the leap and quit their job, they can give up their dream at any time. And 99.9 percent of them will actually give up their dream. If they take the leap, quit their job, go do it full-time—no matter how much it sucks—and convince one other person to do the same thing with them, they’re going to have a much, much higher chance of actually getting somewhere.”

Windows Power Tools is a collection of brief tutorials and overviews of freeware and open source .NET development tools. What kind of rating you might give this book depends largely upon what type of background that you’re coming from. If you’re the kind who has stuck religiously to the Microsoft Press series of books and acknowledge only the old testament, than this book will be either an epiphany (5 stars) or outright blasphemy (1 star). If continuous integration, test-driven development, and object relational mapping (new testament type stuff) are terms that you are fairly conversant with, then this book will probably land somewhere in the 2-4 star range.

Since I put myself in the 2-4 star group, I’ll start by mentioning that there are great online tomes of knowledge that contain most of the tools listed in this book and a bunch others not listed here. One of the most respected and well linked lists belongs to the author of this book’s forward, Scott Hanselman. His Ultimate Developer and Power Users Tool List for Windows has been dutifully updated on an annual basis. Despite the fact that there are free, decent resources out there that fill some of the same purposes as this book, I enjoyed thumbing through the book and picking out tools I hadn’t heard of to fill in some knowledge gaps.

The main reason that I landed on a 3 star rating instead of a 4 star rating is that the brief tutorial format that worked so well for James when describing Visual Studio functionality is his previous book, Visual Studio Hacks, just doesn’t do justice to tools that represent significant pieces of an application or support infrastructure. I would have preferred to see less tools and deeper coverage in certain areas. Understandably, since not everyone would want to see the same tools as me; a broader, shallower approach trades off depth and detail for marketability. I’ve included my complete list of pros and cons below so that you can see how I came to my rating:

Pros

• Great reference book with enough of an introduction to get you started with a broad array of tools
• If you’re an O’Reilly Safari subscriber, this book is included in your subscription
• The authors aspire to keep materials current on the book’s companion Web site. At the time of this review, the site is little more than a list of tools in the book

Cons

• Lots of this material is available for free on the Web, if you have the time and inclination to find it
• Introductions to tools are not sufficiently in depth to communicate any more than the most rudimentary of use cases

Technologists who spend their time working on line of business projects are typically exposed to subtle and not-so-subtle messages about business ethics. Most recently, the implementation of the Sarbanes-Oxley Act has touched most of our lives in some way, shape, or form. At a bare minimum, it has heightened our awareness of how terribly wrong things can go when the trust afforded certain practices (accounting and reporting, in the case of SOX) proves to be misplaced.

As strong as the reaction to Enron, Tyco, and WorldCom debacles was; it surprises me again and again to see just how willing the corporate community is to shrug off major IT project failures. This is especially true when you compare the strict regulations, codes of conduct, and ethical guidelines in place for the public accounting community and the almost complete lack of anything resembling these structures in the professional software engineering community. Enron and Tyco may be examples of otherwise ample process controls gone awry. In the software engineering community, we don’t even have a process.

The recent IT conversations Podcast on the utter failure of the project to stand up the FBI Virtual Case File system reinforced this point for me. For those of you who want to get the down and dirty on the FBI Virtual Case File story, the IEEE has a pretty good article containing what appears to be a fairly objective history of the project. “So what does this have to do with ethics?”, you ask.

I think most laypeople, let alone professional software engineers, perceive about halfway through the IEEE article that this project was a train wreck in the making. So why was this not apparent to the 200 plus contractors and countless FBI staff involved in the project? Have we, as a profession, been conditioned into silence or is virtually every software project so fraught with journeys into uncharted waters that we have become ethically ambivalent to the dangers?

The real “story within the story” here is the tale of Matthew Patton, a security contractor who, despondent about lack of management concern, posted his concerns about the project to a Web-based bulletin board. The Web post, which has been archived online, lead to an FBI investigation, denial of his clearance, and a situation in which Mr.Patton had little choice but to resign. Perhaps it’s just me but I find it a bit ironic that Mr.Patton was forced out of the FBI project in the same year that an FBI employee made it onto Time magazine’s “Person of the Year” cover along with whistleblowers from Enron and WorldCom.

There have been movements in our profession (if I can refer to it as such) to institute a code of ethics and similar measures. Both IEEE and the Association of Computing Machinery (ACM) support a unified software engineering code of ethics. With membership in these organizations representing just a fraction of the population engaged in software engineering, this code has nowhere near the professional impact of similar measures released by organizations such as the American Institute of Certified Public Accountants (AICPA). Where does this leave us as software engineers?

• We need more people like Matthew Patton amongst our ranks because there are certainly more train wrecks like the FBI Virtual Case File system just waiting to happen.

• We have to start taking software engineering as a profession seriously. The fact that late and over-budget projects have come to represent the norm needs to change.

• We need to think long and hard about how to police ourselves as a profession. Accounting and auditing tried, failed, and then Congress stepped in and told them how to run the show as part of Sarbanes-Oxley. If we don’t figure out how to do this ourselves, someone will tell us how we have to run our show.

We as software engineers are bound by the same laws that we created. There are no silver bullets for the issue of ethics in software engineering. There is only the fact that a lot of software projects are delivered late, over-budget, or both. It is our ethical obligation to speak up or support those who do speak up and call out the doomed death-march projects before they’ve done too much harm – to us, to our organizations, and to our profession.