Upgrading Chef 0.9 to 0.10 - some gotchas

There are a couple of problems I experienced with the upgrade of Opscode Chef from 0.9 to 0.10.

No DEB packages yet

Impatient as I was I wanted to upgrade before the Ubuntu deb's were released. So I uninstalled the deb install and did the ruby gem install, keeping my existing config & DB.  Perhaps I'm a bit naïve to think this should work :p

RabbitMQ not properly setup by chef-solo

Once the chef-solo bootstrap was complete I started up all the services and tried things out. It didn't work. Couldn't do the last step in the upgrade docs: knife index rebuild.

Turned out this was related to rabbitmq, which was also filling up the disk quickly with error logs.  The solution was to update the rabbitmq password: rabbitmqctl change_password chef testing.

Internal Server Error 500

Another error I keep getting on some servers is

File with checksum a73b7f6222549364ab0d6c4ed2442abf not found in the  repository (this should not happen) -  (Merb::ControllerExceptions::InternalServerError)"

It looks like there's an inconsistency between CouchDB and the filesystem cache.

The tips on chef bug #1397 explains how this can be fixed with a combination of Shef and rake install:

# shef
chef > require 'chef/checksum'
  => false 
chef > r = Chef::REST.new('http://localhost:5984/chef/_design/checksums/_view/', false, false)
 => #<Chef::REST:0x7f831db382b8 @redirect_limit=10, @cookies={}, @redirects_followed=0, @auth_credentials=#, @url="http://localhost:5984/chef/_design/checksums/_view/", @sign_request=true, @default_headers={}, @sign_on_redirect=true>
chef > r.get_rest("all")["rows"].each do |c| c["value"].cdb_destroy end

After doing this I did the rake install. During this process I had some difficulties that resulted in this error:

/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/openssl/buffering.rb:178:in `syswrite': Broken pipe (Errno::EPIPE)
        from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/openssl/buffering.rb:178:in `do_write'
        from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/openssl/buffering.rb:192:in `write'
        from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/protocol.rb:177:in `write0'
        from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/protocol.rb:153:in `write'
        from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/protocol.rb:168:in `writing'
        from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/protocol.rb:152:in `write'
        from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/http.rb:1557:in `send_request_with_body_stream'
        from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/http.rb:1527:in `exec'
        from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/http.rb:1048:in `__request__'
        from /Library/Ruby/Gems/1.8/gems/rest-client-1.6.1/lib/restclient/net_http_ext.rb:15:in `request'
        from /Library/Ruby/Gems/1.8/gems/rest-client-1.6.1/lib/restclient/request.rb:167:in `transmit'
        from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/http.rb:543:in `start'
        from /Library/Ruby/Gems/1.8/gems/rest-client-1.6.1/lib/restclient/request.rb:166:in `transmit'
        from /Library/Ruby/Gems/1.8/gems/rest-client-1.6.1/lib/restclient/request.rb:60:in `execute'
        from /Library/Ruby/Gems/1.8/gems/rest-client-1.6.1/lib/restclient/request.rb:31:in `execute'
        from /Library/Ruby/Gems/1.8/gems/rest-client-1.6.1/lib/restclient/resource.rb:72:in `put'
        from /Library/Ruby/Gems/1.8/gems/chef-0.10.0/lib/chef/cookbook_uploader.rb:134:in `uploader_function_for'
        from /Library/Ruby/Gems/1.8/gems/chef-0.10.0/lib/chef/cookbook_uploader.rb:25:in `call'
        from /Library/Ruby/Gems/1.8/gems/chef-0.10.0/lib/chef/cookbook_uploader.rb:25:in `setup_worker_threads'
        from /Library/Ruby/Gems/1.8/gems/chef-0.10.0/lib/chef/cookbook_uploader.rb:24:in `loop'
        from /Library/Ruby/Gems/1.8/gems/chef-0.10.0/lib/chef/cookbook_uploader.rb:24:in `setup_worker_threads'
        from /Library/Ruby/Gems/1.8/gems/chef-0.10.0/lib/chef/cookbook_uploader.rb:23:in `initialize'
        from /Library/Ruby/Gems/1.8/gems/chef-0.10.0/lib/chef/cookbook_uploader.rb:23:in `new'
        from /Library/Ruby/Gems/1.8/gems/chef-0.10.0/lib/chef/cookbook_uploader.rb:23:in `setup_worker_threads'
        from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/protocol.rb:135:in `map'
        from /Library/Ruby/Gems/1.8/gems/chef-0.10.0/lib/chef/cookbook_uploader.rb:22:in `each'
        from /Library/Ruby/Gems/1.8/gems/chef-0.10.0/lib/chef/cookbook_uploader.rb:22:in `map'
        from /Library/Ruby/Gems/1.8/gems/chef-0.10.0/lib/chef/cookbook_uploader.rb:22:in `setup_worker_threads'
        from /Library/Ruby/Gems/1.8/gems/chef-0.10.0/lib/chef/cookbook_uploader.rb:69:in `upload_cookbook'
        from /Library/Ruby/Gems/1.8/gems/chef-0.10.0/lib/chef/knife/cookbook_upload.rb:138:in `upload'
        from /Library/Ruby/Gems/1.8/gems/chef-0.10.0/lib/chef/knife/cookbook_upload.rb:74:in `run'
        from /Library/Ruby/Gems/1.8/gems/chef-0.10.0/lib/chef/cookbook_loader.rb:89:in `each'
        from /Library/Ruby/Gems/1.8/gems/chef-0.10.0/lib/chef/cookbook_loader.rb:88:in `each'
        from /Library/Ruby/Gems/1.8/gems/chef-0.10.0/lib/chef/knife/cookbook_upload.rb:72:in `run'
        from /Library/Ruby/Gems/1.8/gems/chef-0.10.0/lib/chef/knife.rb:391:in `run_with_pretty_exceptions'
        from /Library/Ruby/Gems/1.8/gems/chef-0.10.0/lib/chef/knife.rb:166:in `run'

To resolve this problem I removed the nginx proxy from the communications chain and that resolved the problem. The problem occurred in just a single recipe, and I believe the culprit was a 2.2MB file that was failing to upload through the proxy. Yes - I know chef should not be used with 2.2MB files.

After removing the proxy from the chain it was possible to do the rake install and all worked well.