In the final installment of this March 1st trilogy, I’ll hit something really topical, the precipitous fall of the Dow Jones Industrial Average (DJIA) two days ago on February 26th. For those of you just diligently minding your stakes in the market, you probably got wind that all the major indexes tanked pretty badly on the 26th. The Dow, however, tanked in a rather unnatural fashion that was quite different from the others.
You can see the “Live” and recalculated Dow Jones numbers on the graph above. Although the actual Dow (in orange) did drop pretty steeply, there’s no way it could have done the near vertical drop that the graph indicates without some outside intervention. It turns out that this outside intervention was provided by a “computer glitch”, as reported in the general press. More specifically, one of the queuing systems that feeds data to the system that actually calculates the Dow Jones index got backed up under record trading volumes. When market watchers got suspicious that the index movement did not align with the sliding values of its core components, they decided to investigate. Discovering the backlog, they manually switched over to alternate systems. When these systems worked down the backload, a new DJIA was calculated and voila, we had a 200 point drop in a couple of minutes.
As painful as this must have been for professional traders and as painful as it’s likely to be (in terms of lawsuits) for the Dow Jones company, this is really interesting from an IT point of view. If you’re an IT person looking for insight into how these markets work, I thoroughly recommend the book Practical .NET for Financial Markets. It’s one of the most advanced .NET texts I’ve ever read and, as a positive side effect, introduced me to a completely unfamiliar business domain, financial markets. Reading this book will not only make you a more educated developer (whether or not you use .NET), it will help you understand the causes and effects of events like those on the DOW two days ago.
If you work in the IT world today, you have a greater chance of not finding a Starbucks on a randomly selected block in Manhattan than you do of not hearing the term “SOA” during your daily workday. It’s unfortunately not nearly as often that I hear or read something about SOA that sounds reasonable, practicable, and overcomes my otherwise skeptic view of all the hype that so often masquerades as SOA. The recent ITConverations Podcast with Todd Biske and Ed Vasquez from MomentumSI is just the sort of real-world advice that avoids the hype and gives a heavy dose of ground truth reality.
Todd and Ed offer some pragmatic advice with respect to SOA and highlight challenges that really resonate with me based upon my experiences. Amongst these are the lack of a single design time and run time governance toolset and the importance of the mindshift from developing enterprise applications to developing reusable enterprise functionality that can be leveraged by different applications. Most importantly, however, is their insight in the importance of thinking of services as products. I have found this to be one of the single largest hurdles to achieving reuse – be it component-based or service-based reuse. Proficiency in application development in no way guarantees proficiency in product-based service development. In fact, most application development organizations that I have dealt with have little or no experience with product-based development. I couldn’t possibly cover all of the disciplines needed to master product-based development in this short post but I’ll refer you to a couple of sources that I feel give more than an adequate introduction.
Enabling the Service Oriented Enterprise – Not much product development here. This article, however, fills in the missing pieces of product development that are exclusive to services.
I plan on recommending this Podcast to a number of people that I work with or have worked with in the past. I can’t expect everyone that I deal with to spend time digesting countless books and articles on the topic of SOA. As a matter of fact, given some of the materials out there, this might just prove more confusing than useful. An hour of time to listen to a Podcast (especially this one) isn’t too much to ask though. You’ll learn an awful lot in this hour.
In the first of my beginning-of-the-month speed blogs, I’d like to start with the topic that I’ve been putting off longest, building applications on top of Amazon’s Web services. For those of you that think of Amazon.com as just a book store, or an e-commerce platform, or [add your assumption here], think again. In the last year or so, Amazon has released a number of services that have not only established them as a first-class platform, but have pushed the envelope considerably on the idea of virtualization.
This trend started with Amazon’s Simple Storage Service (S3) and was expanded on with their Simple Queue Service (SQS). Late last year, Amazon blew the roof off with the limited release of their Elastic Compute Cloud (EC2) service. EC2 is a configurable computing environment where you can load up virtual instances (roughly equivalent to your average Linux server) – on demand. You can load or unload these instances to meet cyclical demand and you only get charged for the computing time that you use. Couple this grid computing potential with S3 for storage and SQS for queuing and let your imagination be the limit.
Doug Kaye’s architecture for GigaVox Audio Lite, an audio file transcoder, might give you some ideas about what can be done with these services. The related ITConversations podcast contains plenty of details, with both Kaye and Jeff Barr, from Amazon, on the wire with Phil Windley. This podcast does a good job of conveying the vision and the realities of using these services. Some of the discussions, like managing and developing on top of a globally distributed queuing system where a call like size = MyQueue.size doesn’t really make any sense, will cause you to stop and think.
The final new Amazon service that I didn’t mention is Mechanical Turk, which is, simply put, a computer interface onto human services. This one has been around for a while and never got much attention from me until I heard the Podcast. Interestingly, the site castingwords.com, which produces Podcast transcriptions, farms out their work via Mechanical Turk. The combination – digital input in (MP3); digital input out (electronic transcript); virtual computing; virtual workforce – that’s pretty radical. There are some other great uses for this service mentioned on the Podcast as well.
The Podcast is absolutely worth a listen. And if this sort of thing piques your interest, you’re not alone. The queue to get into the EC2 beta program is a few months long. If you’re even remotely interested, get in line now.
I realize that the title of this blog post is something that one would probably not associate with a blog that purports to cover technology. Bear with me please and surf over to YouTube (or view below) to look at the video posted by the Pennsylvania Fish and Boat Commission. I think you will agree that the video is very well done. Much like the partnership between Google and the Pennsylvania Tourism Office that I blogged about a couple of months back, this posting to YouTube represents some signs of forward thinking, Web-based focus, and creativity on the part of state government.
Granted, with tourism and fishing, we are looking at revenue generating sectors of state government. You don’t yet see this sort of thing from Welfare service divisions. How long will it be though until other sectors of state government start looking at public Internet services as an alternate channel for communications with their constituents? The economics of these services surely works in their favor (that is, they’re free). It then becomes a question of what the best channel is to reach one’s constituents.
Face-to-face interaction, snail mail, and government sponsored portals will be around for a while to come, I surmise. But as usage of cell phones, email, inexpensive audio devices and collaborative community-based Internet services like YouTube continue to permeate through the layers of citizenry, state governments will not be able to ignore these viable communication mediums. The one question that remains is whether state governments are prepared to engage in the two way dialog and deal with the power shift that these new communication vehicles will invariably bring.
The innovation engine at Yahoo is heating up, looking to get Yahoo back in the race with the “Big Boys”, rivals Google and Microsoft. In an environment categorized by copycat service offerings and one-upmanship, Yahoo’s offerings are refreshingly unique. I cover three of the most recent services that I’ve been playing around with and that I think will prove entertaining to my readers as well – Pipes, OmniFind Yahoo! Edition, and TagMaps.
Pipes – No less a luminary than Tim O’Reilly called Pipes “A milestone in the history of the Internet”. Pipes is a browser-based visual editor that allows you to take input from one source and pipe it (in UNIX parlance) to another source. Along the way, you can apply a series of filters and transformations to manipulate the data. The data sources start and end as common feeds (RSS, RDF, etc). What you do with the data between its input and output is constrained mostly by your imagination.
Yahoo! was certainly not first on the scene with this idea. Dapper and others have preceded them in this regard. What Pipes brings to the party that no one else does is a really cool visual environment that allows you to trace the path of the data through the transformations and filers, interactively debugging along the way based upon the value of the successive outputs. It also has this cool reuse flavor to it, where you can experiment with, tweak, learn from, and potentially improve or fork off new versions of other peoples pipes or just reuse them in a black box sense.
After you’ve read Tim O’Reilly’s introduction, I encourage you to play around with Pipes. Although seeing is believing, you’ll learn best by actually doing.
OmniFind Yahoo! Edition – Product of a nifty partnership with IBM, the OmniFind Yahoo! Edition is an enterprisey search solution that is the baby brother to IBM’s commercial OmniFind enterprise product. Built on top of the open source Apache Lucene search engine, OmniFind has the solid lineage necessary to be considered worthy of the task.
The product is a very easy install, whether on Windows or Linux, requiring very few steps to get the product up and running. OmniFind returns search results against locally indexed documents and the Internet, with the results being returned in the familiar Yahoo! look and feel. For those interested, the UI can be styled to match a particular site’s look and feel or you can go the direct route and work with the exposed REST APIs.
With the pricetag (free) and support for a couple hundred file types, there’s little not to like about OmniFind. Search performance has proven to be very fast with a few thousand documents. Indications are that the tool scales pretty well although the indexing process can be quite processor intensive and there are a couple of known issues with cleaning up very large temp files that could eat into your available disk space.
TagMaps – I stumbled onto this product a couple of weeks ago while looking for some information about creating GeoRSS feeds. TagMaps is another way of visualizing data (tags in this case) on maps. I must confess that seeing tags on a map takes a bit of getting used to. I found that the best way to indoctrinate myself was by using Trip Explorer
Trip Explorer is a mashup of TagMaps and Yahoo! Travel users’ public Trip Plans. What’s cool about Trip Explorer is that the clustering of tags reveals hidden tour gems that you might not otherwise find on a traditional map mashup. These gems become more evident (and detailed) as you progressively zoom in.
TagMaps is built upon Yahoo’s Flash maps, which are very interesting in their own right and need to be experienced if you haven’t yet had the chance. Aside from using Yahoo’s canned Explorer TagMaps, of which Trip Explorer is one, you can create your own TagMaps mashups. Simply create a GeoRSS feed or select an existing GeoRSS feed that returns a set of weighted tags for a given lat/lon bounded box. Easier said than done, I know. I’ll be writing more about how to do this in a coming blog entry. Until then, give this a look.
Entries (RSS)