Amaze SNS rubygem

One of the things which I have always wanted to do is to contribute back to the open source community at large. For years I have been using and adopting other people’s work so its only fair to keep the cycle moving. Since it is my first attempt at creating a ruby gem, I decided to pick an area of usage which is relatively new as well as straight forward to maintain and develop.

This started back in May and Amazon AWS just launched their Simple Notification Service(SNS) for short. After reading through their API and from my earlier brushing with the S3 request protocol for the S3 IPhone App, I had gain more familiarity with how the AWS API calls work and so I made a gem called ‘amaze_sns’ which interfaces with the SNS service.

The Amaze SNS gem is in its stable release and its version is at 1.0.1. Please note – this is different from the amaze-sns gem which has since been yanked from rubygems as it is buggy and not advisable to be used.

To install the latest version, just do

sudo gem install amaze_sns

The source code can be found at http://github.com/cheeyeo/Amazon-SNS-Rubyand the actual rubygems page on http://rubygems.org/gems/amaze_sns

So far, it has received over 400+ downloads since May but that was mainly for the older version – amaze-sns – which due to a fatal flaw I realised did not work. Apologies for those of you who downloaded that earlier version – I’m still learning what else can I say.

Version 1.0.1 is confirmed to work and supports quite a few new nifty features if I may say so

• You can now set attributes for each topic and the Policy attribute of a topic is JSON parsed so you can access it like a hash

• EventMachine support is rolled back to the earlier stable version and you can still use your own code blocks to parse the response.

• Sample code has been provided in the ‘examples’ folder.

• Fixed some weird bug whereby the Topic actions does not return the response of an action due to the way the EventMachine reactor loop exits on dual core machines running ruby 1.8.7 ??

No doubt this will be an ongoing project as the API is still in a state of flux and so expect further updates as and when they arrive.

Further plans to develop an actual Rails app to showcase it in use in production is in the pipeline.

The eventual goal is to be able to wrap up the requests call within a Rack middleware so it can parse the requests and responses away from the core app.

I have added Pledgie support on the github page – I know it is only a small thing on the whole scale of things but any contribution you make will go towards allowing me to develop this gem further.

Please feel free to forward any links of projects you have used it in and I will showcase it on a dedicated page for it.

Enjoy!

Here are the steps I took:

1. First download the latest ffmpeg version from the svn repo. The revision number I used was 23197
2. Next download the segmenter from the article link above. Then edit the Makefile as follows:

   
   gcc -Wall -I/
   -g segmenter.c -o segmenter -lavformat -lavcodec -lavutil -lbz2 -lm -lz -lfaac -lmp3lame -lx264 -lfaad
   

And that was it. The binary would be compiled into /usr/bin.

In order to test the segmented files you would need to run the processed files within another web application and view it within the safari browser on the IPHone simulator. Below are some screenshots of it in action:

In the next follow up I hope to go into further detail on the process involved. Meantime, please checkout the link above and the apple site for documentation on the process.

The following command is adapted from the blog post mentioned previously and works really well for me:

ffmpeg -y -i test.m4v -f mpegts -acodec libmp3lame -ar 48000 -ab 128k -vcodec libx264 -s 320x240 -b 1000k -flags +loop -cmp +chroma -subq 5 -trellis 1 -refs 1 -coder 0 -me_range 16 -keyint_min 25 -sc_threshold 40 -i_qfactor 0.71 -bt 1500k -maxrate 1000k -bufsize 1000k -rc_eq 'blurCplx^(1-qComp)' -qcomp 0.6 -qmin 10 -qmax 51 -qdiff 4 -level 30 -g 30 -aspect 320:240 -async 2 output.ts

just a short note that I have released a new plugin on my github account:

http://github.com/cheeyeo/Amazon-SNS-Ruby

I’m still in the process of wrapping up into a gem once the spec tests are passed so please bear with me.

The plugin is designed to integrate with the Amazon Simple Notification Service which is similar to the Pusher service as rolled out by the cool guys at New Bamboo. You can design your next generation of Real Time Web apps using SNS.

For the ‘uninitiated’, RTW (Real Time Web) relies on HTML5 websockets which are bi-directional transport streams between the browser and your app, so the moment you make an update to your app, say added in a new blog post,it can notify all the clients who are designed to listen specifically to that particular event and update the blog page in real time.

Similarly, SNS supports push actions through a ‘publish’ action which enables subscribers to a particular ‘topic’ to receive messages via email or JSON through SMTP and http respectively. So far, I have only tried out the SMTP delivery option and sadly it does not support HTML messages at this point.

One cool feature of the plugin is the performance and scalability of it when I started developing it. Originally it only used a single HTTPClient object connection. After having caught the EventMachine bug and async IO operations I was curious about whether a EM version of it would really speed things up.

I added in EM and ran some benchmarks and here are the results below:

100 requests were made to SNS service backend. As you can see, each httpclient request averages 0.75 secs whereas with EM it only takes 0.27 secs! That is less than half and by using EM in conjunction with EventMachine::HttpRequest or the em-http gem, that also supports Deferrable module in EM which means higher concurrency and parallel requests.

This is my first attempt at writing a gem or plugin so your comments and feedback are welcome as it is not perfect by any means.

I also want to thank the following gem authors: Pusher and Happening, from which AmazeSNS borrows some of its ideas from.

I already have Ruby 1.9 installed locally with the alias of ‘ruby19′ through a manual install but the problem with that approach is that you would have to reference that directly within your own Ruby scripts and to remember to call ‘irb19′ version of that. Add in multiple versions and you would have a management crisis at hand.

With RVM, it manages all these versions for you without any conflicts. All you need to change from one version to another is to type the following in terminal

rvm 1.9.1 # this switches to ruby 1.9.1 assuming you have it installed

Here’s what I did to install RVM on an OS X 10.5.8 :

Firstly, I installed the gem version of rvm:

sudo gem install rvm

You can run the bash script as documented on the RVM documentation site but I had permission issues on mkdir on here. The important thing to note is not to use sudo as that will present problems further down the road. Ideally the command you want to achieve is ‘rvm ‘ without needing to be root.

Once the gem is installed, I run the following command:

rvm-install

This will install rvm on your local machine, creating a binary at ‘/usr/local/bin/rvm’. Run ‘which rvm’ to make sure it is all working.

The fun part is to install a ruby version of your choice. I went for Ruby 1.9 but you can choose whichever version you want. The command is as simple as :

rvm install 1.9.1

This will fetch a copy of the source and compile it so it will take a while.

Once that is complete, open up a new terminal or tab and type the following:

 rvm list

This is my display on my local machine:

  rvm Rubies

ruby-1.9.1-p378 [ i386 ]

System Ruby
system [ ppc i386 ]

To switch back to your default system verison of ruby just type:

rvm system

In order to see the changes , you would need to switch to a new tab after the RVM and Ruby install or else it does not show up.

In addition I have to append the following two lines at the bottom of both my ‘.bashrc’ and ‘.bash_profile’ files:

 
  if [[ -s /Users/cheeyeo/.rvm/scripts/rvm ]] ; then source /Users/cheeyeo/.rvm/scripts/rvm ; fi

This adds the path to your rvm install on your home directory so you can access the rvm and rvm install commands.

Another quick tip: if you want to use a specific version of Ruby on startup, just set its default:

  rvm --default 

More information and examples can be found on the RVM website: http://rvm.beginrescueend.com/

It is really worth trying out and you will be surprised how much time it saves you on managing multiple ruby versions.

Just a quick update on my progress. I have been busy doing commercial work for the past two months with various clients, one of which is helping Hermes Technologies to relaunch their web application, HiredHelp.ie , to make it suitable for deployment onto Heroku. One of the significant aspects of it is to redevelop their search capabilities to run on SOLR, rather than xapain as the former is supported by Heroku. That is still ongoing as we speak.

On another footnote, my own iphone application, S3 app has been officially launched and accepted by the app store. After several bug fixes and reviews the hard work has paid off and I was ecstatic to find out today that was the case. I want to thank the developers of ASIHTTP request library and Tapku library for the brilliant work they did on their open source projects, none of which this app will not have been what it is.

What’s next in the pipeline? I am working hard on pushing forward some of my own work which will be sold as products on the main site. One of which is a video encoding system and another a calendar application running on Redis DB, and it functions similarly to iCal but in the browser. Check this space for more details.

However back to the grind as they say!

I added the declarations below within the config/environment.rb file to overcome this issue:

      begin
         PhusionPassenger.on_event(:starting_worker_process) do |forked|
             if forked
               # We're in smart spawning mode, so...
               # Close duplicated memcached connections - they will open themselves
                 CACHE.reset
             end
         end
     # In case you're not running under Passenger (i.e. devmode with mongrel)
     rescue NameError => error
   end
 

The CACHE constant is set within the cache_fu plugin. All the declaration does is similar to that outlined in the Appendix of the Phusion documentation with the addition of the ‘begin..rescue’ block to counteract exceptions which may arise.

I understand from comments on the Passenger google group that cache_fu causes problems with memcached. If anyone has any further information on the above please feel free to comment.

I’m not a firm believer of new year resolutions as I tend not to be able to keep them for some reason or other. But I thought for 2010 I might try a different approach and adopt some to keep myself occupied while waiting for my next gig.

The list below is by no way complete ( in fact, a simple app could be built to manage it??):

Write more regular blog posts

At least once or twice a week. May not have to be technical but at least topical

List of technologies to look out for:

2010 is going to be an exciting time for the development of Rails 3 but also in these areas I’m particularly
interested in: cloud computing; collaborative map-reduce problem solving; HTML 5; new schemaless databases

Keep more up to date than before

I tend to spend too much time on projects so the idea is to speed up development so I can have
more free time to myself to keep up to date with the latest and greatest as well as for my personal life.
This means rethinking on my current way of working and applying the 80-20 principle

Be Faster,Better,Stronger

No pun intended on the Daft Punk song but to be better and faster at what I can do

Also a big Thank You to all those who supported 29 Steps. All my best wishes to you for 2010.

By applying the patches listed here and here, that solves the problem for me.

Please note that the changes have to be made to the ‘acts_as_xapian’ lib file within the vendors directory and then rebuild your index and restart your web app before seeing the changes take effect.

Also, if you are performing a numeric range search don’t prefix the values with ‘:’

For example, searching for ‘prices0..100′ is a numeric range search while ‘prices:low’ is a term search for specific attribute of that model.

I want to start off the new series of posts with interesting tidits I learnt while developing on these two projects.

On the HiredHelp website, we try to cache as much of the DB calls as possible using memecached with the cache_fu plugin. One of the obstacles we came across was if we had to cache the index action of say a list of babysitters with pagination added in, the subsequent paginated pages do not show. This is due to the fact that the cache is storing the page as ‘babysitters/index’ and not ‘babysitters?page=xxx’.

You will need to provide a ‘cached_path’ to the caches_action in the controller.

For example:

This is the ‘caches_action’ method within the controller:

  
      caches_action :index, :cache_path => :index_cache_path.to_proc
      

      def index
        @babysitters = Babysitter.cached(:list).paginate(:page => params[:page],:per_page => 10)
      end
   

  private
        def index_cache_path
          p = ''
          p = params[:page] if params[:page] and params[:page] != 1
          'babysitters/' + p
    end
   

The ‘cached’ method was provided by cache_fu and calls the method ‘list’ within the Babysitter model. The index_cache_path method simply appends the page params from the wii_paginate plugin to the end of the index url. So pages like ‘babysitters?page=2′ will be cached as ‘babysitters/2′ etc.

Now to expire the paginated pages, I wrote a custom method within my sweepers file:

  
  #within BabysitterSweeper
  

  def expire_cache
    total = (Babysitter.list.size.to_i)
    if total <= 10
      expire_fragment("babysitters/index")
    else
      pages = (total % 10) + (total /10) -1
      1.upto(pages) do |page|
        expire_fragment("babysitters/#{page}")
      end
      expire_fragment("babysitters/index")
      

    end
  end
  

The method computes the number of listings allowed per page as stipulated in the wii_paginate call in the index action. In our case it is 10. If the number of entries is less than 10, that means we only expire the index action. If not, we calculate the number of paginated pages there are. If we have say 22 entries that will be split into 3 paginated pages with 2 pages containing 10 entries and a third page with 2 entries. The ‘expire_cache’ method will expire the fragements labelled ‘babysitters/2′ etc from the index call such as ‘babysitters?page=2′.

  
     gcc -c -fPIC some_code.c -o some_code.o

      gcc -shared -Wl,-soname,mylib.so.1 -o mylib.so.1.0.1  some_code.o
   

The above essentially compiles native C code into object files using gcc compiler and then linking it into a library called ‘mylib.so’.

If you issue the command above on a MAC, you will get the following errors:

   
      /usr/bin/ld: unknown flag: -soname
      collect2: ld returned 1 exit status
    

Within OS X, the second command will not work as OS X treats shared libraries as dynamic libraries. The second line needs to be changed to:

   
      gcc -Wall -D_FILE_OFFSET_BITS=64 -fPIC -arch ppc -arch i386 -arch x86_64 -dynamiclib -o mylib.dylib calc_mean.o
   

Then move the shared library mylib.dylib into ‘/usr/lib/’:

  
    sudo cp -v mylib.dylib /usr/lib/