Provision & Automate free Amazon EC2 instances with Hosted Chef

1. Sign-Up with Amazon via this link and they will offer one year of usage (micro instance) to help you understand and evaluate EC2. Read their terms of usage carefully and make sure to only provision the free instances if you don't want to get dinged on costs due to ignorance or carelessness on your own part.
2. Now the free instance image provided by Amazon happens to run RHEL4 but the out-of-the-box Chef-Client requires a higher version of that OS! Luckily some, if not all, of the very same issues have already been tackled in a post on the opscode's help website. And here's a gist of my experience with it as well. In case it is difficult to follow, let me highlight the fact that the modified file that allows you to properly finish the chef-client installation process on the AWS image for Linux was found here: CHEF-2515
3. Some Notes:
   • Either command works well:
     • knife ec2 server create -x ec2-user -i /Users/xxx/.ec2/ec2.pem --template-file ~/chef-repo/.chef/bootstrap/centos5-gems-modified.erb
     • knife ec2 server create --ssh-user ec2-user -i ~/.ec2/ec2.pem -d centos5-gems-modified
   • I find it crucial to configure the appropriate Amazon security group as part of the instance creation otherwise its ends up using the "default" group. To do this, add it to your "knife ec2 server create" command with "-G groupName" and remember that it can be a comma separated list of security groups "-G secGroup1,secGroup2" if you want. Example:
     • knife ec2 server create --ssh-user ec2-user -i ~/.ec2/ec2.pem -d centos5-gems-modified -G elasticsearch

EC2 Bare Minimum

1. install ec2 tools
   cd ~/.ec2
   ssh -i /Users/XXX/.ec2/ec2.pem ec2-user@ec2-XXX.compute-1.amazonaws.com
   sudo yum install screen (why?)
2. For moving data to & from the EC2 instance, simply use scp which runs on top of ssh like so:
3. A lot of images will flunk all ssh connection attempts! It seems like this has nothing to do with the permissions on the pem file that you use to connect to them or opening up port 22 by default in their security policies. They just don't work even after all that, all the attempts spit out:
   
   • either, Permission denied (publickey,gssapi-keyex,gssapi-with-mic)
   • or, Net::SSH::AuthenticationFailed
   So don't waste your time on them! Here's a small list of useless ones that you should avoid:
   • ami-31d41658, Red Hat Enterprise Linux 6.1, x86_64
   • ami-8f4083e6, RHEL-5.6-Starter-EBS-x86_64-11-Hourly2

Import data from Amazon RSS feeds into Solr

1. For this example, lets use the RSS feed for new products that have been tagged as blu-ray, here's the URL:
   
   • http://www.amazon.com/rss/tag/blu-ray/new/ref=tag_rsh_hl_ersn
   • If you are using Firefox, you may not get a chance to view the feed in its raw xml format as the browser tends to actually interpret and present most of the RSS feed in a user-friendly fashion. This is not very desirable for developers or adminstrators. In order to view the raw content of the feed, you can simply view the same URL via Feed-Proxy: http://persistent.info/cgi-bin/feed-proxy?url=http%3A%2F%2Fwww.amazon.com%2Frss%2Ftag%2Fblu-ray%2Fnew%2Fref%3Dtag_rsh_hl_ersn
   • Refer to the following links for info on how to get the RSS feeds you want from Amazon:
     • http://www.amazon.com/gp/help/customer/display.html?nodeId=200202840
     • http://www.amazon.com/gp/tagging/rss-help.html
2. Before we import any data into Solr, lets take a moment to understand the format of the data from the RSS feed. Here's a sample item from the RSS feed:
   
   1. There are 5 basic fields per item: title, guid, link, pubdate and description.
   2. If we take a closer look at the HTML CHUNK inside description, we will find more information like:
      
      • the image URL
        
      • the price for a new item
        

        Buy new: $10.99
        

      • the price of used items starting from a lowerbound:
        

        
          22 used and new
          from $6.35
        
        

      • In fact, the HTML Chunk is like a complete webpage, it even has a separate description section inside of itself!

        ...

      • All this is rather messy and unpredictable (sometimes its there, sometimes its not, sometimes its a class, sometimes its an id, sometimes there are duplicates) but we must try to make the best of it.
3. Having understood the complexities of our source of data, we are now ready to configure the Data Import Handler (DIH) for Solr.
   
   1. Navigate to the directory which has the out-of-the-box sample core for configuring RSS feeds. And edit the rss-data-config.xml file for importing data to look as follows:
      

      cd /trunk/solr/example/example-DIH/solr/rss
      vi conf/rss-data-config.xml
      

4. In order to grab and add the price from the description:
   
   1. add the price as a field to the schema.xml file
      

      cd /trunk/solr/example/example-DIH/solr/rss
      vi conf/schema.xml
      

   2. add the RegexTransformer to the chain of transformers in the rss-data-config.xml file
      

      cd /trunk/solr/example/example-DIH/solr/rss
      vi conf/rss-data-config.xml
      

      
         ...
      
      

   3. add the regex to find and extract the price
      

      cd /trunk/solr/example/example-DIH/solr/rss
      vi conf/rss-data-config.xml
      

      
         ...
         
         
      
      

      Please keep in mind that this is NOT the best regex that you can use. This one simply grabs the last set of digits with a dollar sign in front of them in the HTML, so it may grab the used price instead of the new price! Feel free to come up with a better regex for yourself.
   4. If there are any items where the regex is incapable of pulling the price due to malformed HTML or any other reason, you can choose to either skip those items and not add them to Solr. OR you can introduce your own dummy price for them. Here's how you may do so:
      
      • add the use of the ScriptTransformer function you'll define into the chain of transformers in the rss-data-config.xml file
        

        cd /trunk/solr/example/example-DIH/solr/rss
        vi conf/rss-data-config.xml
        

        
           ...
        
        

      • add the following script to rss-data-config.xml file if you want to skip the items where a price wasn't available or couldn't be deduced
        

        cd /trunk/solr/example/example-DIH/solr/rss
        vi conf/rss-data-config.xml
        

        
        
        
        ...
        
        

      • add the following script to rss-data-config.xml file if you simply want to inject a dummy price
        

        cd /trunk/solr/example/example-DIH/solr/rss
        vi conf/rss-data-config.xml
        

        
        
        
        ...
        
        

5. The following can be added to pull out the image URL:
   

   
      ...
      
      
   
   

6. The following can be added to pull out the date (SimpleDateFormat was used as a reference to devise a format string to parse the incoming dates from the feed such as Mon, 12 Sep 2011 21:14:23 GMT):
   

   
      ...
      
      
   
   

7. Start the Solr server:
   

   cd /trunk/solr/example
   java -Dsolr.solr.home="./example-DIH/solr/" -jar start.jar
   

8. Navigate to the following URL
   

   http://localhost:8983/solr/rss/admin/dataimport.jsp?handler=/dataimport
   

9. Click the Full Import with Cleaning button to import data from Amazon RSS feed into Solr.
   

Junk / Errata

url="http://www.amazon.com/rss/tag/blu-ray/new/ref=tag_rsh_hl_ersn"
regex=".*Buy new.*span.class..tgProductPrice...(\d*.\d*)"
stripHTML="true"