Wednesday 18 May 2016

Create and Alter Sequence in PostgreSQL

Create and Alter Sequence in PostgreSQL

To activate autoincrement property in PRIMARY KEY field for a table having huge records present in postgresql. We need to create sequence and assign it to the corresponding table.

Creating a sequence :
  1. Assume that we have a table called [test_table] with an unique column called [id]
  2. Generate sequence
  3. run command > create <sequence name> , i.e. create test_table_id_sequence;
  4. After the sequence’s already created, we can call NEXTVAL(‘<Sequence name>’) to generate a new value automatically.
  5. Link the sequence to the unique column
  6. ALTER TABLE <table name> ALTER COLUMN <column name> SET DEFAULT NEXTVAL(<created sequence name>);
    => ALTER TABLE test_table ALTER COLUMN id SET DEFAULT NEXTVAL('test_table_id_sequence');
Now the sequence is created and linked to the desired table .
If the table is empty or max id is 0 then we dont have to set value for the sequence .
If the table contains records then we have to set intial value . Neither it will start the value from 1.

Alter the sequence now :

Selecting maximum id from the table to run
SELECT MAX(id) FROM your_table;
=> SELECT MAX(id) FROM test_table;

The value should be higher than the record.
SELECT nextval('your_table_id_seq');
=> SELECT nextval('test_table_id_seq');

If it's not higher... run this set the sequence last to your highest pid it.
SELECT setval('your_table_id_seq', (SELECT MAX(id) FROM your_table));
=> SELECT setval('test_table_id_seq', (SELECT MAX(id) FROM test_table));
If your tables might have no rows
-- false means the set value will be returned by the next nextval() call

SELECT setval('your_table_id_seq', COALESCE((SELECT MAX(id)+1 FROM your_table), 1));
=>SELECT setval('test_table_id_seq', COALESCE((SELECT MAX(id)+1 FROM test_table), 1));

Tuesday 21 October 2014

Process monitoring with God in Rails

God: A Ruby framework for process management.

The core issue we face before using the explanation of the use of process managment tools/gems:

While I have implemented sidekiq to monitor the threads in our application for sending emails in certain time frames. The sidekiq stops if server reboots and the queued processes goes in a indefinite period of time without execution. So this needs the sidekiq should be restarted automatically when it stops.

God is a framework which can automate the process.

FEATURES
  • Config file is written in Ruby
  • Easily write your own custom conditions in Ruby
  • Supports both poll and event based conditions
  • Different poll conditions can have different intervals
  • Integrated notification system (write your own too!)
  • Easily control non-daemonizing scripts
Installation

Add gem 'god' in Gemfile .
Run $ bundle install

Or

Run $ [sudo] install gem god


Version

$ god –version

God Learning Curve

Let us start with basic examples, how to write codes to execute the processes using God.

Let us create a new file god_demo.rb in application folder.

Write the following code in the same file

loop do
puts 'Hello'
sleep 1
end

Now we'll write a god config file that tells god about our process. Place it in the same directory and call it god_demo .god:



God.watch do |w|
w.name = "god_demo"
w.start = "ruby /full/path/to/god_demo.rb"
w.keepalive
end

God configuration should start with a God.watch block. A watch in god represents a process that we want to watch and control. Each watch must have, at minimum, a unique name and a command that tells god how to start the process. The keepalive declaration denotes god to keep this process alive. If the process is not running when god starts, it will be started. If the process dies, it will be restarted.

To run god, we give it the configuration file we wrote with -c. To see what's going on, we can ask it to run foreground with -D:

There are two ways that god can monitor your process. The first and better way is with process events. Not every system supports it, but those that do will automatically use it. With events, god will know immediately when a process exits. For those systems without process event support, god will use a polling mechanism. The output you see throughout this section will show both ways.

This is the example in http://godrb.com/

# Events

I [2011-12-10 15:24:34] INFO: Loading simple.god
I [2011-12-10 15:24:34] INFO: Syslog enabled.
I [2011-12-10 15:24:34] INFO: Using pid file directory: /Users/tom/.god/pids
I [2011-12-10 15:24:34] INFO: Started on drbunix:///tmp/god.17165.sock
I [2011-12-10 15:24:34] INFO: simple move 'unmonitored' to 'init'
I [2011-12-10 15:24:34] INFO: simple moved 'unmonitored' to 'init'
I [2011-12-10 15:24:34] INFO: simple [trigger] process is not running (ProcessRunning)
I [2011-12-10 15:24:34] INFO: simple move 'init' to 'start'
I [2011-12-10 15:24:34] INFO: simple start: ruby /Users/tom/dev/mojombo/god/simple.rb
I [2011-12-10 15:24:34] INFO: simple moved 'init' to 'start'
I [2011-12-10 15:24:34] INFO: simple [trigger] process is running (ProcessRunning)
I [2011-12-10 15:24:34] INFO: simple move 'start' to 'up'
I [2011-12-10 15:24:34] INFO: simple registered 'proc_exit' event for pid 23298
I [2011-12-10 15:24:34] INFO: simple moved 'start' to 'up'

# Polls

I [2011-12-07 09:40:18] INFO: Loading simple.god
I [2011-12-07 09:40:18] INFO: Syslog enabled.
I [2011-12-07 09:40:18] INFO: Using pid file directory: /Users/tom/.god/pids
I [2011-12-07 09:40:18] INFO: Started on drbunix:///tmp/god.17165.sock
I [2011-12-07 09:40:18] INFO: simple move 'unmonitored' to 'up'
I [2011-12-07 09:40:18] INFO: simple moved 'unmonitored' to 'up'
I [2011-12-07 09:40:18] INFO: simple [trigger] process is not running (ProcessRunning)
I [2011-12-07 09:40:18] INFO: simple move 'up' to 'start'
I [2011-12-07 09:40:18] INFO: simple start: ruby /Users/tom/dev/mojombo/god/simple.rb
I [2011-12-07 09:40:19] INFO: simple moved 'up' to 'up'
I [2011-12-07 09:40:19] INFO: simple [ok] process is running (ProcessRunning)
I [2011-12-07 09:40:24] INFO: simple [ok] process is running (ProcessRunning)
I [2011-12-07 09:40:29] INFO: simple [ok] process is running (ProcessRunning)




Wednesday 8 October 2014

Custom routing for restricting sidekiq pages in rails.

After integrating sidekiq with rails . It provides the web access to manage the processes/jobs . User can view Scheduled jobs from "Scheduled" tab.

Here is the screenshot of the sidekiq page.  This will be more easier to understand with this page.


Here we can see the jobs are being listed in the table. These jobs are in queue to be scheduled.

These listed jobs have some individual checkbox to select each jobs. We can delete the jobs at any point. Also deschedule from the site . Below is the "Delete" button to delete the jobs .


If we will view it to pubic it will be miss-utilized.

What we can do?

We can restrict the url from routes. Below is the routes for sidekiq.

require 'sidekiq/web'

Myapp::Application.routes.draw do
     mount Sidekiq::Web, at: '/sidekiq'
end

This is the general route for sidekiq. All the routes are generated from the same route configurations.

If we will do  $rake routes   in our rails console, we will get the following routes
 sidekiq_web     /sidekiq           Sidekiq::Web

How to restrict?

Example 1-

We can use a lambda call for the specific user group or emails. For example :

authenticate :user, lambda { |u| u.is_super_admin? || u.email == 'your@email.com'} do
    mount Sidekiq::Web, at: '/sidekiq'

end

Here it is restricting through the user group as if the user is super admin then it will show to the user.
Also it can be opened to a specific user email.

Example 2

Through Restful Authentication

Checks a User model instance that responds to admin?

#lib/admin_constraint.rb
class AdminConstraint
    def matches?(request)
        return false unless request.session[:user_id]
        user = User.find request.session[:user_id]
        user && user.admin?
    end
end

#config/routes.rb
require "admin_constraint"
mount Sidekiq::Monitor::Engine => '/sidekiq', :constraints => AdminConstraint.new



Also we can follow all the examples explained in
 Sidekiq wiki : https://github.com/mperham/sidekiq/wiki/Monitoring

Thanks :)

Wednesday 24 September 2014

Getting labels from fedex

Here we will start on getting rates from fedex with rails..

 require 'fedex' 
fedex = Fedex::Shipment.new(:key => 'xxx', 
       :password => 'xxxx', 
       :account_number => 'xxxx', 
       :meter => 'xxx', 
       :mode => 'production') 

Generating a shipping label with PDF Format

To create a label for a shipment:

label = fedex.label(:filename => "label_dir/label.pdf",
:shipper=>shipper,
:recipient =>recipient,
:packages =>packages,
:service_type =>"FEDEX_GROUND",
:shipping_options=> shipping_options)

shipper is the test shipper we defined in the previous blog . Recipient is the
same test recipient also packages .

Generate a shipping label in any available format

Change the filename extension and pass a label_specification hash. 

For example:

example_spec = {
:image_type => "EPL2",
:label_stock_type => "STOCK_4X6"
}

label = fedex.label(:filename =>"my_dir/example_epl2.pcx",
:shipper=>shipper,
:recipient =>recipient,
:packages =>packages,
:service_type =>"FEDEX_GROUND",
:shipping_options=> shipping_options, 
:label_specification => example_spec)

Generate shipping labels for multi-package shipments (MPS)

Customer can have multiple packages for a single pick-up, destination and payer 

can be combined into a single MPS shipment. The first label will provide a
master tracking number which must be used in the subsequent calls for
the remaining packages in the shipment.

Parameters for the first label:

label = fedex.label(
:filename => file_name,
:shipper => shipper,
:recipient => recipient,
:packages => packages,
:service_type => service_type,
:shipping_details => shipping_details,
:shipping_charges_payment => shipping_charges_payment,
:customs_clearance_detail => customs_clearance_detail,
:mps => {:package_count =>
package_count, :total_weight => total_weight, :sequence_number =>
'1'}

)

Parameters for labels 2 through 'n':

fedex.label(
:filename => file_name,
:shipper => shipper,
:recipient => recipient,
:packages => packages,
:service_type => service_type,
:shipping_details =>shipping_details,
:shipping_charges_payment =>shipping_charges_payment,
:customs_clearance_detail =>customs_clearance_detail,
:mps => { :master_tracking_id =>{
:tracking_id_type => 'FEDEX', 
:tracking_number=>tracking_number},
:package_count =>package_count,
:total_weight => total_weight,
:sequence_number => 'n'      }   )


Tracking a shipment

To track a shipment:

results = fedex.track(:tracking_number=> "1234567890123")

# => [#<Fedex::TrackingInformation>]

# Pull the first result from the returned array

tracking_info = results.first

tracking_info.tracking_number

# => "1234567890123"

tracking_info.status

# => "Delivered"

tracking_info.events.first.description

# => "On FedEx vehicle for
delivery"

Deleting a shipment
If you dont want to use a fedex label, you should delete the shipment. Fedex will not charge you .

For deleting fedex shipment :
fedex.delete(:tracking_number => "1121123123345")

Address verification with fedex


address = {
:address     => "5 Elm Street",
:city        => "Littleton",
:state       => "CO",
:postal_code => "80125",
:country     => "USA"
}

address_result = fedex.validate_address(:address => address)


# =>  #<Fedex::Address:0xb2e60ac
@changes=["NORMALIZED", "HOUSE_OR_BOX_NUMBER_NOT_FOUND"],
@score=0, @confirmed=false, @available=true, @status="UNDETERMINED",
@residential=false, @business=false, @company=nil, @street_lines="5
ELM ST", @city="LITTLETON", @state="CO",
@province_code="CO", @postal_code="80125",
@country_code="US", @options={:score=>"0",
:changes=>["NORMALIZED",
"HOUSE_OR_BOX_NUMBER_NOT_FOUND"],
:residential_status=>"UNDETERMINED",
:delivery_point_validation=>"UNCONFIRMED",
:address=>{:street_lines=>"5 ELM ST",
:city=>"LITTLETON", :state_or_province_code=>"CO",
:postal_code=>"80125", :country_code=>"US"},
:removed_non_address_data=>nil}>


address_result.residential

# => false

address_result.score

# => 0

address_result.postal_code

# => "80125"

These are the operations working for fedex with this gem. Hope this will help to understand the integration of fedex with rails . Thanks









Thursday 18 September 2014

Fedex API integration in rails

Introduction to Fedex
FedEx Corporation is an American global courier delivery services company headquartered in Memphis, Tennessee.


With FedEx Web Services, we can integrate FedEx Express®, FedEx Ground® and FedEx Freight® shipping services into our business systems, retail website or order management system. FedEx Web Services is the next generation shipping API for integrating software applications with FedEx Systems to create shipping labels, facilitate returns, track shipments, obtain rate quotes and generate reports.

Fedex API integration in rails 4

We can integrate Fedex API with gem 'fedex' in rails .

It uses Fedex API Shipment version 13. This version uses the Non-SOAP Web Services so there is no need to download the Fedex WSDL files, note however that you will need to apply for development/production credentials.

Installation

Include gem 'fedex' in Gemfile.rb .

Run $bundle install .

How to use?
Basically we use some shipping destination where to ship and the sender's destinations from where we want to ship. We need to provide package informations . The package informations consists of

Package Dimensions with parameters

  • height
  • length
  • width
These informations need to give in inches or centimeters .

Package Weight parameter
  • weight with lbs

We will send a request to the fedex API to get rate from fedex. Later we will get a request to get label from fedex. The label will give a report with bar code with valid fedex tracking number.

Using Rails gem how will acomplish this will described here .....

Defining a Test Shiper

shipper = { :name => "Test Fedex Sender",
:company => "Home",
:phone_number => "121-333-2332",
:address => "Main Street",
:city => "Littleton",
:state => "CO",
:postal_code => "80125",
:country_code => "US" }

Defining a test Recipient

recipient = { :name => "Test Fedex Recipient",
:company => "Home",
:phone_number => "555-555-5555",
:address => "Main Street",
:city => "Boulder",
:state => "CO",
:postal_code => "80304",
:country_code => "US",
:residential => "false" }

Defining the packages; multiple packages in a single shipment are also allowed (all the dimensions must be integers).

packages = []
packages << {
:weight => {:units => "LB", :value => 2},
:dimensions => {:length => 10, :width => 5, :height => 4, :units => "IN" }
}
packages << {
:weight => {:units => "LB", :value => 6},
:dimensions => {:length => 5, :width => 5, :height => 4, :units => "IN" }
}

By default the packaging type will be “YOUR PACKAGING” and the dropoff type will be “REGULAR PICKUP”

Initializing shipping options

shipping_options = {
:packaging_type => "YOUR_PACKAGING",
:drop_off_type => "REGULAR_PICKUP"
}

Creating a Fedex::Shipment object using your FedEx credentials; mode should be either production or development depending on what Fedex environment you want to use.

Include fedex in your ruby file
require 'fedex'

Initializing Fedex::Shipment

fedex = Fedex::Shipment.new(:key => 'xxx',
:password => 'xxxx',
:account_number => 'xxxx',
:meter => 'xxx',
:mode => 'development')




Getting Shipment Rates

rate = fedex.rate(:shipper=>shipper,
:recipient => recipient,
:packages => packages,
:service_type => "FEDEX_GROUND",
:shipping_options => shipping_options)


This will provide a response as
# Complete response
$ <Fedex::Rate:0x1019ba5f8
@total_net_charge="34.03",
@total_surcharges="1.93",
@total_billing_weight="8.0 LB",
@total_taxes="0.0",
@rate_type="PAYOR_ACCOUNT_PACKAGE",
@total_base_charge="32.1",
@total_freight_discounts=nil,
@total_net_freight="32.1",
@rate_zone="51">

$ rate.total_net_charge => "34.03"


Fedex provides multiple total values; total_net_charge is the final amount we are looking for.


We will then find the suitable way to pay the fedex price and continue shipping.

In the next blogs we will read about the fedex label, cancelling fedex shipment, verifying address with fedex.









Tuesday 9 April 2013

A Ride To Rails 4


        Most of the time we think about a better version of rails for which our application can run smoothly. In the mean time rails has been upgraded to 4 new versions. For rails developer who has been acquainted with all earlier 3 versions will be easy to understand the later versions of rails . Let us discuss some new concepts in rails 4. :)

  1. The most important changes that Rails 4 needs is Ruby 1.9.3 at a minimum.
  2. Vendor/Plugins directory folder has been removed from Rails 4.

Deprecations


Many things has been deprecated and moved to separate gems. Like active record deprecator s finder. Rails 4 uses “activerecord-deprecated_finders” gem for finding old active record finders .

Deprecated finders


Rails 2/3 Model.find([:all/:first/:last]) has been deprecated.

# find_all_by_name
Rails 3 Model.find_all_by_name('name') is deprecated.
In Rails 4, Model.where(:name, 'name')

#find_last_by
Rails 3 Model.find_last_by_name('name') is deprecated.
In Rails 4, Model.where(:name, 'name').last

#find_or_create_by
Rails 3 Model.find_or_create_by_name('name') is deprecated.
In Rails 4, Model.where(:name, 'name').first_or_create

#find_or_create_by..!
Rails 3 Model.find_or_create_by_name!('name') is deprecated.
In Rails 4, Model.where(:name, 'name').first_or_create!

#find_or_initialize_by
Rails 3 Model.find_or_initialize_by_name('name') is deprecated
In Rails 4, Model.where(:name, 'name').first_or_initialize

Normal Eagerly evaluated Scopes declarations have been deprecated


Scopes require now a callable object. In Rails 4 all scopes must use a callable object such as a Proc or lambda.
For example, In Rails 3 we can declare scope as

Class Comment <ActiveRecord::Base
scope :recent_comment, where(:published_at, Time.now-2.weeks)
end
but it is deprecated in Rails 4. And the new decalration as below

Class Comment <ActiveRecord::Base
scope :active, -> where(:status, 'active')
end

Identity map removed


active_record identitymap configuration has been removed from Rails4. If you are upgrading Rails 4 from Rails 3, then please be sure to remove/comment the line from config.application.rb

// config.active_record.identity_map = true


ActiveRecord::SessionStore


Storing sessions in databases is an interesting feature in many of the applications. As many cases it does not have much influences like a normal cokiee store. So ActiveRecord::SessionStore has been removed from Rails 4. We can use it by adding gem 'activerecord-session_store' in gem file.

ActiveResource


ActiveResource which provides a ORM to REST web services is no longer exist in Rails 4. To get the funcationality in your app include gem 'activeresource'

authenticity_token


authenticity_token is no longer included in forms in Rails 4. While submiting non-ajax forms in rails, authenticity_token was required in back end to prevent cross-site forgery attacks.

New Features

Postgress Array Support


We all know ProstgreSQL supports array datatypes. That means we can store array elements in a column in PostgreSql. But normally in many of the databases they are stored as strings.

Active Record with Rails4 will support array datatypes for PostgreSql. Also it will support MACADDR, INET, CIDR datatypes.

Strong Parameters


Rails 4 has introduced mass assignment protection of attributes in controller. This process can be achived by the “strong parameters” in Rails 4. It is the tool to fight against the mass assignment vulnerabilities.

Let us consider one example incase of Rails 3, when we are supposed to submit a form to create a user we do as,

    <form method='post' action='/users'>
       <input type='text' name='name' />
       <input type='submit' />
   </form>

When we submit the form it will go to create a user record. For this we need to mass assign the name attribute in model as 'attr_accessible', neither it will raise an error. To protect the name attribute in model we will do something like this ..

    Class User<ActiveRecord::Base
       attr_accessible :name
    end

Then in controller we will do this to create the user

    def create
      @user = User.new(params[:user])
      @user.save
      .....
    end

In Rails 4 there are new options to make the attribute secure and dynamically set them as attr_accessible in controller side.

   Class UsersController< ApplicationController::Base
      def create
         @user = User.new(user_params)
         @user.save
         .....
      end

     private
    def user_params
       params.require(:user).permit(:name)
    end
  end

We can test it in our 3.2.x applications by installing the strong_parameters gem in app. :)

Turbolinks


Turbolinks makes the following links in your web application faster. Instead of letting the browser recompile the JavaScript and CSS between each page change, it keeps the current page instance alive and replaces only the body and the title in the head.

Turbolink works like PJAX. But there are many fuctionalities difference in both the two.
Turbolinks uses pushState, but instead of replacing only parts a page it loads a complete website from the server and replaces the <title> and<body> in the currently loaded DOM. By default it applies this to all links on a page. So unlike PJAX you don't have to mark links and containers to support in place reloading, Turbolinks will handle that for you.

In Rails 4 bydefault gem 'turbolinks' is being added in the gem file. Hope it will be better to slow down the page responses and the sight will run faster :)

Live Streaming


Rails 4 will support to live streaming. Live stream is the effect to stream content to end user's browser directly from the server with long polling in the browser.

Live streaming has now got much popularity now a days. It helps to watch score reports of various games, sports, watching seminars from famous people hosting through internet, viewing results, and lot more.

The PATCH verb


PUT in REST is defined for updating a record in rails. PATCH will work as same . It will refer to the update action of the controller. But we need to post the form with the parameter “_method=PATCH”. Also routing will help to redirect to the update action in the controller.

 PUT, semantically-speaking, should contain a full representation of an entire object to be updated. But in Rails 4 PATCH will send a new bit of rescource about an object on the server not a full representation of it.

The same PUT method will also work in case of Rails 4.

Concern in Routing


In Rails 3 if we have a same nested resource for a resource, we include the nested resource each time in a parent resouce. For example

    resouces :projects do
        resources :comments
    end

    resources :tasks do
        resources :comments
    end

Rails 4 introduced concern to cut down the duplicacy of common resource routing declaration. Here it defined as

    concern :commentable do
        resources :comments
    end

    resources :projects, concerns: :commentable
    resources :tasks, concerns: :commentable


Thanks 
Debadatt Pradhan
Mindfire Solutions, Pvt. Ltd