TechYob

I gave the Google MapReduce paper a quick read today. It outlines the approach Google uses to abstract over paralleled computation, distribution of data and fault tolerance associated with deploying large data-sets over large clusters of commoditised hardware. Bottom line, any web startup with significant scale aspirations should raise MapReduce or the open source Hadoop as a brown-bag topic for the engineering team.

Thoughts:

1. Adapting a Functional programming model and the discipline of atomic task execution is a smart way to reduce complexity for many internet scale information processing problems.
2. The MapReduce library significantly reduces the knowledge acquisition phase for new Google engineers, and hence helps scale people as much as it helps to scale code.
3. MapReduce is already being used in some machine-learning applications. It is likely that the corollary is also true. Numenta’s hierarchical temporal memory (HTM) model is very interesting in its addition of cascasding expectation optimization.
4. Prioritization of tasks corresponding to different business applications would be an emerging issue. If the end-game is building an internet operating system to be made available to all developers a la Amazon’s Elastic Computing Cloud (and it should be!) this resource management and task isolation is key.
5. Could the master-worker communication be replaced with P2P between workers?
   

Friedman is talking about the Flat World.  Bezos is building it.

Techcrunch lets the cat out the bag today about Amazon’s yet-to-be-released SDS Service - rumoured to stand for “Simple Data Service”.  This coincides with an article I read this morning over breakfast in the Australian Financial Review about Amazon’s web-services-as-a-service venture.  Rob Hof gives an insight into the tech offerings that may well position Amazon as one of the leading players in utility computing.  

Amazon’s Elastic Computer Cloud and Simple Storage Services just made it easier for kids to start their own Web 2.0 company with some loose pocket change.  Back in the dot-com boom when working for Accenture I remember the massive tier-3 architectures we’d build and deploy for ecommerce companies.   Now I can enter the game for 10 cents an hour for a server computer, and 15c per gigabyte per month for storage. 

This got me thinking back to a comment made to me by one of the respected wags here in Sydney over a couple of Pale Ales

“I can’t afford a million dollar house in Paddington.  But I can rent one!”

Bezos may have some doubters on Wall Street, but Im a believer.