Thursday, December 31, 2009

My tech industry predictions for 2010

As both an author and as a technical consultant, I am fairly opinionated in terms of what I expect for next year (tomorrow!). Here are my predictions:
  • There will be pressure to reduce IT expenditures. Increasing trend to favor outsourcing deployment platform (e.g., Google AppEngine and Heroku), infrastructure (e.g., Amazon AWS, RackSpace, both relational and NoSLQ data stores), and software as a service (e.g., CRM, etc.). Cloud computing gets more real.
  • Research will be concentrated on shorter term profits rather than long strategy. China might be an exception to this: a friend reports that he has seen willingness in China to fund very long term Artificial Intelligence research.
  • More people will spend more time using web based information and recreational resources using portable devices. There will be a shortage of wireless bandwidth in some areas.
  • Use of the Java platform will stay strong, but with more emphasis on alternative languages like JRuby, Scala, and Clojure.
  • Skills and education in data analysis, all types of modeling, text mining, and analytics will continue to be in increasing demand.
  • There will be well publicized sporadic failures by cloud service providers but customers will be tolerant because of cost savings.
  • The importance of the hugely increasing availability of all types of data will encourage development of new business areas like knowledge processing as a service.
  • Although the Semantic Web has seen slow adoption in the last decade, Linked Data, a new and more popular subset of the Semantic Web, will see wide adoption in the next few years.
  • The wide support for Linked Data will help make computer to computer interactions easier and more profitable.
  • Geo location will become increasing important. We are likely to see many interesting smaller scale systems using technology like PostGIS and Spatial Lucene that will be able to compete in local markets with Google, Microsoft, and Yahoo.
  • I think that the next decade will be a time of micro web businesses: a huge number of small scale web portals that offer very specific services to a relatively small number of customers who are willing to pay a use fee. Most of these will be deployed on very low cost cloud hosting. Many of these will be tightly integrated with the Facebook platform, the Wave platform, etc.
  • In the next 10 years: the USA and Europe will become much less important in technical inovation because their large debt liabilities will create business and research unfriendly environments.
  • In the next 20 years: educated, creative, and/or generally productive people will identify more with similar people around the world than with people in their own countries who may get bogged down in what looks to be a crushing world wide economic collapse (but not quite as bad in developing countries). There will be more of a feeling of world wide citizenship, the world will evolve first into a safer place (a trend over the the last 20 years has seen more safety, less civilian deaths - something that most people don't realize) and then eventually into greater prosperity.
Next year looks like an exciting time in our industry!

Best Wishes for a Happy New Year

Thursday, December 24, 2009

$1.5 trillion a year for "defense" spending, little money left for local governments; living locally; Happy Holidays

If you factor in the cost of the debt to pay for our military spending then I think that a reasonable estimate of our yearly defense spending is about $1.5 trillion.

The amount we spend on defense swamps other US government spending, including social programs. This is just my opinion, but I believe that we could keep our country relatively safe (compared to other countries) and spend far less money. The problem is basically banana republic style corruption: too much money is made by special interests for there to be any meaningful reform of military spending. The same comment is also true of other corporate interests like Wall Street, industrialized food production, pharmaceutical companies, insurance, etc. This is all enabled by total corporate control of the news media and shameful corruption in the lobbying industry and our federal government.

Fortunately, for most of us, life is still very good despite corruption of the world's "elite." Again, this is just my personal opinion: it is best to act in ways that help yourself, your family, and local community. Avoid doing business with large banks, preferring credit unions. Prefer buying food from local producers. Form or join local organizations for food distribution, computer clubs (promote open source solutions to help people use old PCs), and help with local school fund raising activities. Prefer to buy goods that are manufactured as close to where you live as possible.

Joy in life comes from our family and social interactions and in acting in service to others. A materialistic lifestyle is something that you have been conditioned to accept by lifetime exposure to media - try to resist the temptation to acquire material goods that you really don't need.

And, most of all:

Have a Merry Christmas and a Happy Hanukkah!

Friday, December 18, 2009

Building the EtherPad system and perusing the source code

The EtherPad collaboration-enabled online system is now open source. Cool. Google bought the company, the developers are joining the Wave team, and their product is now released under the Apache 2 license.

Be sure to follow the instructions (failing to set the environment variable for the path to a MySQL client JAR file produces a strange "cp -f" error that has hung up a few people trying to build the system, as reported on Hacker News). It only took about 15 minutes to download the source and build the system - simple enough.

After running the system and trying it, I used IntelliJ 9 to set up a project (choose project from existing source, main directory etherpad/trunk) and I am spending some time perusing the Scala and JavaScript code. Really nice looking code base, and reading through the Scala code will be an education. My JavaScript skills are a little weak, but I might still take a careful look at the JavaScript code to understand how they used Comet.

Monday, December 14, 2009

Amazon Elastic Load Balancing (ELB) is pretty cool

Using this service costs $0.025/hour so it may make sense to just run HAProxy yourself on a EC2 instance, but then you have to worry about fault tolerance/recovery if that instance fails. The ELB cost is small in comparison to running a cluster of EC2 instances and "outsourcing" as much of your system as possible to AWS (e.b., SimpleDB, Elastic Load Balancing, Relational Database Server, EBS, etc.) can certainly reduce both the complexity of your application architecture and also your implementation costs.

Here are my notes for a simple ELB setup for an AMI that contains a Rails web application:
export EC2_PRIVATE_KEY=pk-....pem  # different on your system
export EC2_CERT=/Users/markw/.ec2/cert-...pem # different on your system

ec2run -k gsg-keypair ami-e767ds71
ec2run -k gsg-keypair ami-e767ds71

Note: specifying "gsg-keypair" matches later doing ssh -i ~/.ssh/id_rsa-gsg-keypair ...

elb-create-lb MarkTest123 --headers --listener "lb-port=80,instance-port=4242,protocol=HTTP" --availability-zones us-east-1b

elb-configure-healthcheck MarkTest123 --headers --target "HTTP:4242/ping" --interval 30 --timeout 3 --unhealthy-threshold 2 --healthy-threshold 2

elb-register-instances-with-lb MarkTest123 --headers --instances i-X,i-Y

* substitute real EC2 IDs for i-X and i-Y, and this works great!

ssh -i ~/.ssh/id_rsa-gsg-keypair
create a file public/ping - put a line of text in it
start Rails on port 4242

* note: repeat for other EC2 instance. In production, obviously you would want to start your Rails apps using /etc/init.d/...

Then, hitting this URL works:


elb-delete-lb MarkTest123
and delete EC2 instances
It took me a little while to get ELB working, so I hope that my notes save you some time.

The cost of commoditization of computing: infrastructure and software

Discipline, a new view of system architecture, and rigorous automation procedures are required to take advantage of Amazon EC2, Google AppEngine, and other infrastructure as a service providers. Last week a customer commented on the rapid pace of Amazon's innovation. Yesterday they announced a new way to generate revenue from unused server instances by letting users bid on a spot market for unused EC2 instances.

Discipline When you own your own server farm, even if it is just a few back room servers, you can spread out applications over your servers in a haphazard way and usually get away with some sloppiness in your process architecture. When you are dealing with someone else's infrastructure a more disciplined approach is just about mandatory.

New view of system architecture Both Google and Amazon have published papers on dealing with very large scale geographically disperse systems comprised of many components, some of which are guaranteed to fail. These companies have also shared how for some applications they trade off immediate data consistency (between distributed data writers and readers) for scalability.

Rigorous automation procedures Large distributed fault tolerant systems involve two types of software development: core application development and writing automation code to manage managing servers (starting/stopping, DNS management) and partitioning an application over available servers.

Software The open source and free software movements have hugely contributed to the commoditization of computing. Even not counting operating system and networking infrastructure, almost all of the code in the systems that we create is written by other people and it is usually open source software. I would argue that in addition to classical skills like algorithms and raw coding skill, a modern developer must be well versed in available frameworks and libraries, their strengths and weaknesses, and knowledge of modifying and using them.

Commoditization comes with cost This is something that I am in the process of learning about. For most of my career I have had either dedicated hardware for development and deployment (starting in the 1980s, developing world wide distributed systems, I had expensive dedicated servers), or in the last 10 years, my "world" consisted of spending a lot of effort building systems on top of open source frameworks (J2EE, Rails, etc.) and deploying to a leased server, or a VPS for projects with few users. In this old scenario, deployment and admin would be a very small part of the effort of creating a new system. Now, designing, implementing, and deploying to Amazon AWS, Google AppEngine, etc. has changed that because of the effort it takes to properly "live" in a commodity computing world.

Sunday, December 13, 2009

IntelliJ version 9.0

This week JetBrains gave me an upgrade license for version 9 of IntelliJ. I don't do too much Java development anymore - mostly maintenance on some of my old projects and new AppEngine and Google Wave development.

Overall, version 9 is a nice upgrade: nicer source code repository integration, built in task management, the IDE seems faster, etc.

I use the JetBrains RubyMine product almost every day. Long term, my use of IntelliJ 9 will depend on how good the AppEngine support is. As a test, I generated a new AppEngine project, added some home page material, edited the appengine-web.xml file to specify a registered app name, set the version, then deployed to Google's servers with no problems. I did have a problem importing an AppEngine project from Eclipse but eventally realized that I needed to go to Module Settings -> Artifacts, and drag "Available elements" from the right window pane to the <output root> tree display in the left pane.

I tried writing a simple GWT 2.0 (new version) web app. When I created a new project, no template GWT client and server code was generated - a small disappointment. I deleted that project and used a command line tool to generate a project:
/Users/markw/Documents/WORK/gwt-2.0.0/webAppCreator -out mytest com.mytest
I then created a new IntelliJ project "from existing source", and had no problems using the generated template project, running the local development server, and using the web app in Firefox (after installing the GWT FireFox plugin).

I am really busy with client work right now but I hope to get some time next year to do an AppEngine based project of my own, and I am almost sure that I will use IntelliJ 9 for that because general Java development is a bit better, I think, than using Eclipse, and it looks like the AppEngine support is OK.

Monday, December 07, 2009

Balkanisation of Ruby?

When I first started using Ruby, Matz's C-Ruby was mostly the only game in town.

I am also an enthusiastic user Ruby 1.9.x, JRuby, and MacRuby. Seeing the Ruby spec being developed in Japan under government funding, and that it is for Ruby 1.8.7, I get a feeling of déjà vu as a long time Lisp user.

The balkanisation of Lisp has been more than a small nuisance for me in the last 25 years. I would prefer that all Ruby implementations eventually implement a common specification, but I would rather it be for something that looks like 1.9.x.

Sunday, December 06, 2009

Privacy and Security in the Internet Age

Just some advice that I give friends and family:
  • Delete all cookies in your browser every week - it is easy enough to sign in again to web sites that require authentication. People who do not delete their cookies never see what sites are tracking them. It is easiest to do a 'delete all cookies' operation and not to try to save the 5 or 10 cookies out of thousands that are stored in your local browser data.
  • Keep a text file with all passwords in encrypted form - and, do not use the same password for different purposes.
  • Every time you use your super market's discount card (or possibly pay with a credit card), your purchases are permanently associated with you - do you care? maybe or maybe not.
I do use a lot of web services that track what I do (GMail, for example) but I make the decision to give up privacy vs. benefits on a service by service basis.

Coolness! good instructions for trying Rails 3.0pre

This is worth sharing: easy to follow instructions for installing Rails 3.0pre.

Thanks to Oscar Del Ben!

Yesterday I spent 90 minutes trying a fresh install of Ruby 1.9.1 and Rails 3.0pre with no luck, so thanks to Oscar for his writeup. BTW, I Used Ruby 1.8.7 when following Oscar's directions.

Thursday, December 03, 2009

New Amazon Web Services feature: boot from EBS

Awesome - in the past I have had to write my own code/scripts to manage attaching and EBS file volume to a new EC2 instance. This new feature will make it a lot easier to manage your own EC2 based services. If you would prefer a PDF with the complete documentation, then use this link.

This new feature will make using EC2 even easier for some applications - a welcome change.

That said, I have my own scheme for automatically mounting EBS volumes, assigning ElasticIP addresses, etc. and for some deployments I will continue to use temporary boot volumes and pull AMIs from S3.