29 Steps | Ruby, Rails, Web 2.0 development and design | UK

EC2 Server backup to S3

On September 8, 2010 By Chee Yeo

Since I’ve started using the EC2 service, I am aware of the hourly costs if my server is running overnight if I’m in the process of configuring it. Subsequently I’ve discovered I can actually back it up as a private AMI instance onto S3 and then restart it later when required.

In order to accomplish this, you would need 2 sets of information: your set of private key and certificate which is used to instantiate the EC2 server; this would need to be copied onto the running instance using scp; and the EC2 AMI tools installed. The AMI tools is required to bundle and upload the final image onto S3.

Once the above is in place, just issue the following command within the instance console:

  
ec2-bundle-vol -d /mnt -k /root/PrivateKey.pem  --cert /root/Certificate.pem  -u 012345678901 -a -e /var/mysql,/var/log/

The ec2-bundle-vol command is from the AMI tools. The -d option specifies where the image segments go; the -k switch specifies the private key used to create the AMI; the certificate is from your own amazon certificate and the -u identifier is your amazon account ID. The -a switch specifies to include all directories and the -e switch specifies a list of directories to exclude from the process; this can be used to exclude the directories such as where your certificate and keys are stored as well as log folders to save space on the resulting image.

Once it starts running, the command will generate a flatfile of the entire AMI onto /mnt and subsequently it breaks it down into various segments of fixed sizes within the /mnt directory. Once the process is complete an ‘image.manifest.xml’ will also be generated in the /mnt directory. This specifies the segments needed to recreate the AMI.

To upload the AMI into S3 you would need to create a new bucket first. Then use the ec2-upload-bundle command as follows:


ec2-upload-bundle -b yourbucketname -m /mnt/image.manifest.xml -a your_s3_access_key -s your_s3_secret_access_key

You should see the name of each image segment which has been uploaded into the S3 bucket successfully showing up.

Once that is complete, first check to make sure that all the segments and the manifest are in the bucket. Then you can terminate the AMI and log into your amazon console in the browser to register the AMI.

Navigate to the Amazon EC2 Management Console > AMIS > Register new AMI. Specify the bucket name followed by the manifest to register your AMI:

  
      yourbucketname/image.manifest.xml

If the above all goes well you should see your AMI registered and should be able to launch it by right clicking on it and selecting ‘launch’.

Tagged with: amazon • aws • backup • EC2 • S3

Setting up RVM on an EC2 Ubuntu server

On September 8, 2010 By Chee Yeo

RVM is a great tool for running various versions of Ruby on a single development machine. And especially useful to manage various groups of gems for each version of Ruby i.e. you can create gemsets which bind the particular version of Ruby to a specific group of gems so I can have a gemset for my RVM version of Ruby 1.9.2 which has the latest Rails 3 and associated gems and this can run in isolation from the other system gems without conflicts.

Trying to set it up on an older Ubuntu EC2 server for me is a challenge. The AMI image I used was an older image (Hardy 8.0.4) and already comes with ruby 1.8.6 pre-installed.

After may hours of swearing and some unusual errors (e.g. Cipher not found) and having to restart the installation a few times, I managed to get it right. This is what I did.

Firstly, you would need certain packages before you can build Ruby.

  
    $ apt-get update
    $ sudo apt-get install build-essential libxml2 libxml2-dev libsqlite3-dev locate libssl-dev openssl autoconf

Secondly, create a directory called ‘Downloads’ in your local directory and get Ruby 1.8.7. You would need at least Ruby 1.8.7 in order to install Ruby 1.9 upwards. I opted for the 302 patch of 1.8.7 as it fixes a vulnerability issue as mentioned on the main ruby homepage.


  $ mkdir Downloads
  $ cd ~/Downloads
  $ wget ftp://ftp.ruby-lang.org/pub/ruby/1.8/ruby-1.8.7-p302.tar.gz
  $ tar -xzxvf ruby-1.8.7-p302.tar.gz
  $ cd ruby-1.8.7-p302
  $ autoconf
  $ ./configure
  $ sudo make
  $ sudo make install

Next is to install the latest version of Rubygems. From my previous installation I managed to get RubyGems 1.3.7 installed(??) but just to make sure:


  $ cd ..
  $ wget http://rubyforge.org/frs/download.php/70696/rubygems-1.3.7.tgz
  $ tar -xzvf rubygems-1.3.7.tgz
  $ cd rubygems-1.3.7
  $ sudo ruby setup.rb
  $ cd ..

Now that should create a new ruby so if you do ruby -version it should show as 1.8.7 and likewise with gems –version, it should read 1.3.7.

The issue I ran into at this stage when I ran gem –version was a ‘Cipher is not a module error’. To fix this just change into your ruby-1.8.7 directory and do the following:


    $ cd ext/openssl
    $ ruby extconf.rb
    $ make
    $ sudo make install

You might need to exit the shell and open a new one And then try to require ‘openssl’ and it should return true if it is successful.

The next step is to get RVM installed and these are my steps below:


    $ mkdir .rvm
    $ cd .rvm
    $ git clone http://github.com/wayneeseguin/rvm.git
    $ cd rvm
    $ ./install

You would need to edit the .bashrc file as follows:

  
   # add this at the end of the file
   if [[ -s /home/user/.rvm/scripts/rvm ]] ; then source /home/user/.rvm/scripts/rvm ; fi

You would need to make sure there is no return statement inside your .bashrc file else rvm will not work properly. Also make sure the path to the rvm command above is correct. If you run which rvm and it shows up as a binary then you have installed it as binary mode and you will not be able to switch rubies in the terminal. The way to test this is

  
   $ type rvm | head -n1

and that should show up as rvm is a function’ then you know it is sourced correctly.

The next steps I did after setting up RVM is to install a different version of ruby such as 1.9.2.

After testing and configuring the above, I then set about creating different gemsets for each to avoid any conflicts between say the Rails 2 and Rails 3 gems.


  $ rvm gemset create rails2 rails3
  $ rvm 1.9.2-head@rails3
  $ gem install rails --version 3.0.0.
  $ rvm 1.9.2-head@rails2
  $ gem install rails
  $ rvm use 1.9.2-head@rails3
  $ rails -v

You would still need to open a new terminal session for rvm to take effect.

Tagged with: EC2 • ruby • RVM • ubuntu

Using a flash uploader in Rails 2.3x

On September 8, 2010 By Chee Yeo

For the Cuepoint App video ecoder app I’m currently building, I decided to switch from a server side progress bar which relies on nginx upload progress module to a more intuitive solution which is both cross browser compliant and also makes use of the new HTML 5 features. I also regard progress updates for file uploads to be a part of the application itself and not the server.

In my search I came across the Panda uploader which uses a hybrid method of auto detecting if the user browser supports HTML 5; if not it falls back to a flash solution using SWFUpload. Problem I ran into was SWFUpload does not include the form’s authenticity token which causes the uploads to crash with ‘Form authenticity token errors’ on upload.

To solve this I followed the article by the web fellas which uses a middleware solution to intercept the SWFUpload request and then appends the form’s authenticity token to it. The middleware, lives within the lib folder of the application and to call it within the Rails app, you would need to append it to the Rails middleware stack.

If you run rake middleware on the console of a Rails 2.3 x app you will see a list of middleware used by the framework. The trick here is to insert the flash middleware before the session middleware kicks in.

I put that within the config/initializers/session_store.rb:

ActionController::Dispatcher.middleware.insert_before(ActionController::Base.session_store, FlashSessionCookieMiddleware, ActionController::Base.session_options[:key])

What the above does is to intercept the request by using the FlashSessionCookieMiddleware and that passes the session key as a parameter into the middleware.

The middleware script is self explanatory. It first checks the http header for any Flash request and if a match is found, it is parsed and then collated into a HTTP_COOKIE header to be processed as an ordinary request by the Rails app. This is the middleware in its entirety:

  
require 'rack/utils'

class FlashSessionCookieMiddleware
  def initialize(app, session_key = '_session_id')
    @app = app
    @session_key = session_key
  end

  def call(env)
    if env['HTTP_USER_AGENT'] =~ /^(Adobe|Shockwave) Flash/
      params = ::Rack::Utils.parse_query(env['QUERY_STRING'])
      env['HTTP_COOKIE'] = [ @session_key, params[@session_key] ].join('=').freeze unless params[@session_key].nil?
    end
    @app.call(env)
  end
end

The missing piece is where and how to pass the session and authenticity token to the middleware and this is done in a helper like so:


  def new_asset_path_with_session_information
    session_key = ActionController::Base.session_options[:key]
    upload_file_videos_path(session_key => cookies[session_key], request_forgery_protection_token => form_authenticity_token)
  end

And then integrate it into your own forms:

  
   form_for(@video, :url => new_asset_path_with_session_information, :multipart => true)

Please consult the webfellas article for more precise details on the exact integration. I can verify that it works for me in a Rails 2.3.5 app.

The caveat is if you are calling the Flash uploader from within an iframe in a modal window, Rails does not pick up the authenticity token so you will still run into authenticity token errors. To overcome this I just escape protect_from_forgery_protection from within the controller but that is not the best option to do so. If anybody knows of how to overcome authenticity token errors in an iframe for a rails app, please feel free to comment to the article.

Tagged with: flash • rack • rails • upload

New Ruby Gem – Amaze SNS version 1.0.1

On July 8, 2010 By Chee Yeo

Amaze SNS rubygem

One of the things which I have always wanted to do is to contribute back to the open source community at large. For years I have been using and adopting other people’s work so its only fair to keep the cycle moving. Since it is my first attempt at creating a ruby gem, I decided to pick an area of usage which is relatively new as well as straight forward to maintain and develop.

This started back in May and Amazon AWS just launched their Simple Notification Service(SNS) for short. After reading through their API and from my earlier brushing with the S3 request protocol for the S3 IPhone App, I had gain more familiarity with how the AWS API calls work and so I made a gem called ‘amaze_sns’ which interfaces with the SNS service.

The Amaze SNS gem is in its stable release and its version is at 1.0.1. Please note – this is different from the amaze-sns gem which has since been yanked from rubygems as it is buggy and not advisable to be used.

To install the latest version, just do

sudo gem install amaze_sns

The source code can be found at http://github.com/cheeyeo/Amazon-SNS-Rubyand the actual rubygems page on http://rubygems.org/gems/amaze_sns

So far, it has received over 400+ downloads since May but that was mainly for the older version – amaze-sns – which due to a fatal flaw I realised did not work. Apologies for those of you who downloaded that earlier version – I’m still learning what else can I say.

Version 1.0.1 is confirmed to work and supports quite a few new nifty features if I may say so

You can now set attributes for each topic and the Policy attribute of a topic is JSON parsed so you can access it like a hash

EventMachine support is rolled back to the earlier stable version and you can still use your own code blocks to parse the response.

Sample code has been provided in the ‘examples’ folder.

Fixed some weird bug whereby the Topic actions does not return the response of an action due to the way the EventMachine reactor loop exits on dual core machines running ruby 1.8.7 ??

No doubt this will be an ongoing project as the API is still in a state of flux and so expect further updates as and when they arrive.

Further plans to develop an actual Rails app to showcase it in use in production is in the pipeline.

The eventual goal is to be able to wrap up the requests call within a Rack middleware so it can parse the requests and responses away from the core app.

I have added Pledgie support on the github page – I know it is only a small thing on the whole scale of things but any contribution you make will go towards allowing me to develop this gem further.

Please feel free to forward any links of projects you have used it in and I will showcase it on a dedicated page for it.

Enjoy!

Tagged with: amazon simple notification service • gem • github • open source • ruby • rubygems

IPhone video streaming take 1 – Compiling the segmenter on OS X 10.5

On June 6, 2010 By Chee Yeo

One of the main features I wanted to incorporate into Cuepoint App originally was the encoding of a video into a format suitable for streaming on the Iphone. After reading an article from http://www.ioncannon.net/programming/452/iphone-http-streaming-with-ffmpeg-and-an-open-source-segmenter/ I tried to compile the segmenter in the article on os x 10.5. After several attempts I gave up and only recently did I manage to get it to compile and function properly on the os x platform. Although the segmenter still throws out ‘mpegts < dts ‘ errors at least it did compile and generate the relevant ts and m3u8 files necessary to allow a video to stream on the IPHone simulator as a test bed.

Here are the steps I took:

First download the latest ffmpeg version from the svn repo. The revision number I used was 23197

Next download the segmenter from the article link above. Then edit the Makefile as follows:


gcc -Wall -I/
-g segmenter.c -o segmenter -lavformat -lavcodec -lavutil -lbz2 -lm -lz -lfaac -lmp3lame -lx264 -lfaad

And that was it. The binary would be compiled into /usr/bin.

In order to test the segmented files you would need to run the processed files within another web application and view it within the safari browser on the IPHone simulator. Below are some screenshots of it in action:

In the next follow up I hope to go into further detail on the process involved. Meantime, please checkout the link above and the apple site for documentation on the process.

The following command is adapted from the blog post mentioned previously and works really well for me:

ffmpeg -y -i test.m4v -f mpegts -acodec libmp3lame -ar 48000 -ab 128k -vcodec libx264 -s 320x240 -b 1000k -flags +loop -cmp +chroma -subq 5 -trellis 1 -refs 1 -coder 0 -me_range 16 -keyint_min 25 -sc_threshold 40 -i_qfactor 0.71 -bt 1500k -maxrate 1000k -bufsize 1000k -rc_eq 'blurCplx^(1-qComp)' -qcomp 0.6 -qmin 10 -qmax 51 -qdiff 4 -level 30 -g 30 -aspect 320:240 -async 2 output.ts

Tagged with: ffmpeg • iphone • m3u8 • segementer • video streaming

New Ruby gem AmazeSNS

On May 3, 2010 By Chee Yeo

Hi all,

just a short note that I have released a new plugin on my github account:

http://github.com/cheeyeo/Amazon-SNS-Ruby

I’m still in the process of wrapping up into a gem once the spec tests are passed so please bear with me.

The plugin is designed to integrate with the Amazon Simple Notification Service which is similar to the Pusher service as rolled out by the cool guys at New Bamboo. You can design your next generation of Real Time Web apps using SNS.

For the ‘uninitiated’, RTW (Real Time Web) relies on HTML5 websockets which are bi-directional transport streams between the browser and your app, so the moment you make an update to your app, say added in a new blog post,it can notify all the clients who are designed to listen specifically to that particular event and update the blog page in real time.

Similarly, SNS supports push actions through a ‘publish’ action which enables subscribers to a particular ‘topic’ to receive messages via email or JSON through SMTP and http respectively. So far, I have only tried out the SMTP delivery option and sadly it does not support HTML messages at this point.

One cool feature of the plugin is the performance and scalability of it when I started developing it. Originally it only used a single HTTPClient object connection. After having caught the EventMachine bug and async IO operations I was curious about whether a EM version of it would really speed things up.

I added in EM and ran some benchmarks and here are the results below:

100 requests were made to SNS service backend. As you can see, each httpclient request averages 0.75 secs whereas with EM it only takes 0.27 secs! That is less than half and by using EM in conjunction with EventMachine::HttpRequest or the em-http gem, that also supports Deferrable module in EM which means higher concurrency and parallel requests.

This is my first attempt at writing a gem or plugin so your comments and feedback are welcome as it is not perfect by any means.

I also want to thank the following gem authors: Pusher and Happening, from which AmazeSNS borrows some of its ideas from.

Tagged with: asynchronous • eventmachine • gem • github • plugin • ruby

RVM Rocks

On April 12, 2010 By Chee Yeo

I’ve learnt about RVM (Ruby Version Manager) through a contact of mine and decided to try it out for myself as to how easy it is to manage multiple versions of Ruby on one dev machine. And it does exactly what it says on the tin – it just works out of the box!

I already have Ruby 1.9 installed locally with the alias of ‘ruby19′ through a manual install but the problem with that approach is that you would have to reference that directly within your own Ruby scripts and to remember to call ‘irb19′ version of that. Add in multiple versions and you would have a management crisis at hand.

With RVM, it manages all these versions for you without any conflicts. All you need to change from one version to another is to type the following in terminal

rvm 1.9.1 # this switches to ruby 1.9.1 assuming you have it installed

Here’s what I did to install RVM on an OS X 10.5.8 :

Firstly, I installed the gem version of rvm:

sudo gem install rvm

You can run the bash script as documented on the RVM documentation site but I had permission issues on mkdir on here. The important thing to note is not to use sudo as that will present problems further down the road. Ideally the command you want to achieve is ‘rvm ‘ without needing to be root.

Once the gem is installed, I run the following command:

rvm-install

This will install rvm on your local machine, creating a binary at ‘/usr/local/bin/rvm’. Run ‘which rvm’ to make sure it is all working.

The fun part is to install a ruby version of your choice. I went for Ruby 1.9 but you can choose whichever version you want. The command is as simple as :

rvm install 1.9.1

This will fetch a copy of the source and compile it so it will take a while.

Once that is complete, open up a new terminal or tab and type the following:

 rvm list

This is my display on my local machine:


  rvm Rubies

ruby-1.9.1-p378 [ i386 ]

System Ruby
system [ ppc i386 ]

To switch back to your default system verison of ruby just type:

rvm system

In order to see the changes , you would need to switch to a new tab after the RVM and Ruby install or else it does not show up.

In addition I have to append the following two lines at the bottom of both my ‘.bashrc’ and ‘.bash_profile’ files:

 
  if [[ -s /Users/cheeyeo/.rvm/scripts/rvm ]] ; then source /Users/cheeyeo/.rvm/scripts/rvm ; fi

This adds the path to your rvm install on your home directory so you can access the rvm and rvm install commands.

Another quick tip: if you want to use a specific version of Ruby on startup, just set its default:


  rvm --default

More information and examples can be found on the RVM website: http://rvm.beginrescueend.com/

It is really worth trying out and you will be surprised how much time it saves you on managing multiple ruby versions.

Tagged with: multiple versions • ruby • RVM

Quick March 2010 update

On March 23, 2010 By Chee Yeo

First off, I did not make the initial commitment I promised myself which was to write one blog post every week so I’m going to take that back and just make blog posts when it is appropriate to do so. I think there is a world of difference in making blog posts for the sake of filling up space compared to talking about a specific issue.

Just a quick update on my progress. I have been busy doing commercial work for the past two months with various clients, one of which is helping Hermes Technologies to relaunch their web application, HiredHelp.ie , to make it suitable for deployment onto Heroku. One of the significant aspects of it is to redevelop their search capabilities to run on SOLR, rather than xapain as the former is supported by Heroku. That is still ongoing as we speak.

On another footnote, my own iphone application, S3 app has been officially launched and accepted by the app store. After several bug fixes and reviews the hard work has paid off and I was ecstatic to find out today that was the case. I want to thank the developers of ASIHTTP request library and Tapku library for the brilliant work they did on their open source projects, none of which this app will not have been what it is.

What’s next in the pipeline? I am working hard on pushing forward some of my own work which will be sold as products on the main site. One of which is a video encoding system and another a calendar application running on Redis DB, and it functions similarly to iCal but in the browser. Check this space for more details.

However back to the grind as they say!

Tagged with: 2010 • iphone • march • update

Phusion Passenger and memcache

On January 1, 2010 By Chee Yeo

One of my recently deployed Rails app uses memcache with Phusion Passenger. I read on the Phusion documentation about the possibility of corrupted commands issued to memcache through the spawned processes within Passenger.

I added the declarations below within the config/environment.rb file to overcome this issue:


      begin
         PhusionPassenger.on_event(:starting_worker_process) do |forked|
             if forked
               # We're in smart spawning mode, so...
               # Close duplicated memcached connections - they will open themselves
                 CACHE.reset
             end
         end
     # In case you're not running under Passenger (i.e. devmode with mongrel)
     rescue NameError => error
   end

The CACHE constant is set within the cache_fu plugin. All the declaration does is similar to that outlined in the Appendix of the Phusion documentation with the addition of the ‘begin..rescue’ block to counteract exceptions which may arise.

I understand from comments on the Passenger google group that cache_fu causes problems with memcached. If anyone has any further information on the above please feel free to comment.

Tagged with: apache2 • memcache • passenger • rails

New Year Resolutions 2010

On January 1, 2010 By Chee Yeo

Happy New Year 2010

I’m not a firm believer of new year resolutions as I tend not to be able to keep them for some reason or other. But I thought for 2010 I might try a different approach and adopt some to keep myself occupied while waiting for my next gig.

The list below is by no way complete ( in fact, a simple app could be built to manage it??):

Write more regular blog posts

At least once or twice a week. May not have to be technical but at least topical

List of technologies to look out for:

2010 is going to be an exciting time for the development of Rails 3 but also in these areas I’m particularly
interested in: cloud computing; collaborative map-reduce problem solving; HTML 5; new schemaless databases

Keep more up to date than before

I tend to spend too much time on projects so the idea is to speed up development so I can have
more free time to myself to keep up to date with the latest and greatest as well as for my personal life.
This means rethinking on my current way of working and applying the 80-20 principle

Be Faster,Better,Stronger

No pun intended on the Daft Punk song but to be better and faster at what I can do

Also a big Thank You to all those who supported 29 Steps. All my best wishes to you for 2010.

Blog of 29 Steps, a Ruby Rails company in UK

EC2 Server backup to S3

Setting up RVM on an EC2 Ubuntu server

Using a flash uploader in Rails 2.3x

New Ruby Gem – Amaze SNS version 1.0.1

IPhone video streaming take 1 – Compiling the segmenter on OS X 10.5

New Ruby gem AmazeSNS

RVM Rocks

Quick March 2010 update

Phusion Passenger and memcache

New Year Resolutions 2010

Write more regular blog posts

List of technologies to look out for:

Keep more up to date than before

Be Faster,Better,Stronger

Categories

Tags

Articles

Links