Super Spread Sheet S³

Or little computing tricks and hacks

Too many ssh authentication failures

Edit August 6, 2016

The solution I gave back in August 1, 2016 does not work. That rule will simply not offer any keys.

I found this post which refers to the pitfalls of ssh-agent, and it decribes that it is a problem having too many keys. It depends on the setting of the server to allow a specific number of keys offered before refusing access.

There is also a ssh argument force using a specific key.

Original post

In  my previous post Managing multiple ssh keys, I describe how to set up different ssh keys to different server and have the ssh command automatically discover which key to user for which (user, server) combination.

However, when I had a new server, if the key was not in the ~/.ssh/config file (se previous post), I would experience the follow:

ssh username@server
Received disconnect from server_ip port 22:2: Too many authentication failures
Connection to server closed by remote host.
Connection to server closed.

Looking around there were many explanations:

  1. That there is a limit on the number of keys offered to the server (true)
  2. That I ( could check what is happening by adding the -v flag to the ssh command (true)
  3. That I should add the line IdentitiesOnly yes to every definition in the ~/.ssh/config file (true but not enough)

I still had the problem.

The final solution was to add the line IdentitiesOnly yes to the /etc/ssh/ssh_config under Host *.

The enlightenment came from this post.

 

 

 

 

Advertisements

Rails: When join tables have no class

I was revisiting an old project of mine, where I had brilliantly created a join table. I had forgotten, though that I did not create a Rails model, because I did not need one. So coming back to the project, I wanted to count the number of rows in that join table in the command line, rails c.

Surprise! I cannot find the the class name, so I cannot access the table directly there…

Instead, I ran psql database_name:

# \dt;
# SELECT * FROM database_name;

to get the list of all the rows in the table.

The join table I created, joins the ids of other two tables: article and category. In was of the models one find the following statement: has_and_belongs_to_many.

To access the information in the table in the rails console, where c is an object of catergory:

c.article_ids

which returns an array of the article ids which are associated with that category. The other direction also works. Given a an object of article:

a.category_ids

returns an array of categories to which the article belongs to.

When in doubt, read the docs:

http://guides.rubyonrails.org/active_record_querying.html

http://guides.rubyonrails.org/active_record_querying.html#joining-tables

Adventures with Ubuntu in a MacBookPro9,2

My daughter came to me one day: “Mum, my Mac is kaput”. When upgrading, the computer just hang. After all the diagnosis possible, we figured it was a disc crash.

A mac techie acquaintance took the computer and after checking it up told us that he could replace the disc and … install Ubuntu! Yes please!

After getting the computer back I started playing with it, connect to the wifi, install this, download that, why is the Ubuntu version only 14.04? Let’s upgrade. After all we are only a few days from the next release 15.10. Hmm, there are errors. We need to reboot, yes, no, ahhh. Kernel panic…

… After some research I had some interesting findings.

There seems to be a tie between the Mac model and the Ubuntu release. This page shows the recommended Ubuntu release to the specific MacBookPro hardware model. They recommend the latest LTS when the user is not sure of the release to install. I was reticent to leave 14.04LTS, but looking at this wikipedia page, I was reassured that this particular version’s support runs until 2019-04! By then this Mac should be history!

To install according to the Mac’s model, first find out the hardware type by typing the following:

sudo dmidecode -s system-product-name

The output in my case:

MacBookPro9,2

And there is where I noticed that 15.04 was not going to work. So I proceeded to reinstall 14.04LTS from a usb stick and that was like a breeze, only after reading how to boot from a usb stick in a Mac:

Insert the Ubuntu LiveCD into your Mac and Shutdown. Restart the Mac and hold the Option Key. When the boot selector screen comes up, choose to boot from the CD.

The full installations instructions can be found here, but I just followed the section “Single-Boot: Ubuntu Only”.

All good except that the wireless card did not seem to be set up. But it was working before so it can be done. I did get scared when I clicked on the MacBookPro9-2/Utopic Unicorn link, and it read that wireless was not supported. But Utopic Unicorn is 14.10. And I have 14.04 Trusty Tahr.

Roughly these are the steps to follow to set up the wireless connection.

Identify the wireless chipset

This can be done in a couple of ways:

  • lspci | grep Network
  • lspci -vvnn | grep -A 9 Network

From the commands I learned that

  • The Chip ID is BCM4331,
  • The PCI-ID is 14e4:4331, and
  • Kernel driver in use is bcma-pci-bridge

Find the drivers for the chipset

This guide contains a full description of specific drivers supporting Broadcom BCM43xx Chipset. And there are a different instructions that one could follow. IN my case the chipset was supported by more that one driver but what worked for me was the section b43 – No Internet access:

  1. Install the b43-fwcutter package.
    cd /media/pool/main/b/b43-fwcutter/
    sudo dpkg -i b43-fwcutter* 
    
  2. Download the firmware file from here unto a computer with internet connection.
  3. Copy the file to your working directory (yes, using a usb stick). In a terminal use b43-fwcutter to extract and install the firmware:
    tar xfvj broadcom-wl-5.100.138.tar.bz2
    sudo b43-fwcutter -w /lib/firmware broadcom-wl-5.100.138/linux/wl_apsta.o
    
  4. Restart the computer or reload the b43 module by switching between drivers. I did the later.
    First unload all conflicting drivers (this includes removing the driver you’re trying to install):

    sudo modprobe -r b43 bcma
    sudo modprobe -r brcmsmac bcma
    

    Then load the driver to use:

    sudo modprobe b43
    

And by magic I now have a wireless connection, and life is good again!

Related links

  1. https://en.wikipedia.org/wiki/List_of_Ubuntu_releases#Ubuntu_15.10_.28Wily_Werewolf.29
  2. https://help.ubuntu.com/community/MacBookPro
  3. https://help.ubuntu.com/community/MactelSupportTeam/AppleIntelInstallation
  4. https://help.ubuntu.com/community/WifiDocs/Driver/bcm43xx
  5. http://www.lwfinger.com/b43-firmware/broadcom-wl-5.100.138.tar.bz2
  6. http://askubuntu.com/questions/338210/broadcom-sta-wireless-driver
  7. https://en.wikipedia.org/wiki/Station_%28networking%29
  8. https://help.ubuntu.com/community/WifiDocs/WirelessCardsSupported

git: comparing with remote branches

I have a Rails app which is deployed in Heroku and its source is in bitbucket. In Heroku I have in fact two instances: staging and production. When I came back after a break on the project, I wanted to compare what was deployed or committed where, as I knew that a JavaScript bug had prevented me to have a full deployment.

git works with branches so the comparison I want to make, takes place between branches, regardless of where they are located.

To lists local and remote branches, run the command:

git branch -a

The output for my app looks something like this:

  ...
  master
  remotes/origin/master
  remotes/heroku/master
  remotes/staging/master
  ...

The first line is the local working branch, the second in the remote master branch in bitbucket, the last two lines are the production and staging branches in heroku.

I can simply run git diff with the name of the two branches to get a detail description, line by line, of the differences:

git diff remotes/heroku/master remotes/staging/master

To get the list of only the files that are different, use the following command:

git diff --stat --color master remotes/heroku/master

Sources:

Dear App, why are you giving me the wrong date?

Localization has always amazed me. But I also giggle when developers get it wrong. I know for experience that it is not obvious. How does the app know what language do you speak if you leave in Montreal, Switzerland or Belgium, where there are more than one official language? And that is when when the user wants to use the app in one of the official languages.

Added to that are the timezones and daylight savings shifts in one place throughout the year. I am originally from Colombia, where even the notion of seasons is summarized as the rain and dry seasons. Don’t ask me when each is supposed to be! Change of time in Summer? Forget it. “Wow, are you really going to sleep now, aunty, here we are going to have lunch!” The internet brings these issues to the fore and forces the need for solutions. The infrastructure is there to be used.

The last time I stumbled upon timezones was when implementing the events section of my Rails app. The first surprise was to find that the time I saved was shifted. The reason made perfect sense. The canonical storage timezone in the database is UTC and the display can be done in a timezone of your choice. Cool. I adjusted the configuration and added to the config/application.rb file the following line:

    config.time_zone = 'Eastern Time (US & Canada)'

That worked just fine in my local development platform: the storage was done in UTC and the display in EST/EDT depending on the time of the year, with the following code:

    event.my_date.to_formatted_s(:long_ordinal)
    event.my_date.zone %>

But when I deployed to heroku, the initial problem came back. I realized that it had something to do with the local timezone of the machine, somehow. As would only make sense, the heroku server is running in UTC time. However, even if the default timezone read EST, the actual saving into the database was done simply by striping the timezone: 17h00 EST became 17h00 UTC.

I could not find a reason, but that does not matter. The point is to make the app work standalone, independently of where it is deployed.

I tried several paths:

1. I added an extra line to the config/application.rb as follows:

    config.active_record.default_timezone = 'Eastern Time (US & Canada)'

However, that line just made the previously saved my_date fields disappear.

2. I then used the use_zone command to create a block, where the default timezone would be defined:

Time.use_zone("Eastern Time (US \& Canada)") {
        time_entered = Time.new(
          params[:date]['year'].to_i,
          params[:date]['month'].to_i,
          params[:date]['day'].to_i,
          params[:date]['hour'].to_i,
          params[:date]['minute'].to_i,
        )
        @event.the_date = time_entered
      }

This was a step forward, the time_entered was produced in the correct timezone but the assignment and then saving in the database, simply stripped off the timezone information.

3. After much trial and error, the solution was to create an object with the UTC offset. This offset had to be dependent on the entered date, as daylight savings for that date is what matters.

The final solution is:

      Time.use_zone("Eastern Time (US \& Canada)") {
        time_entered = Time.new(
          params[:date]['year'].to_i,
          params[:date]['month'].to_i,
          params[:date]['day'].to_i,
          params[:date]['hour'].to_i,
          params[:date]['minute'].to_i,
        )
        the_offset = Time.zone.parse(time_entered.to_s).utc_offset
        new_time = Time.new(
          params[:date]['year'].to_i, 
          params[:date]['month'].to_i,
          params[:date]['day'].to_i,
          params[:date]['hour'].to_i,
          params[:date]['minute'].to_i,
          0,
          the_offset
        )
        @event.my_date = new_time
      }

The code for viewing the date, did not change.

If you have any further comments or explanations, please send them this way.

Sources

Python’s lists: referencing versus copying

I finished the on-line class Introduction to Interactive Programming in Python from Rice University and it was a lot of fun.

asteroids

This is a screen shot of the last game RiceRocks, which is a simplified version of the arcade game Asteroids. My ship has blown up a few asteroids. 😉

Although the course caters for beginners, I learned a great deal of things. For example, I had never programmed an animated game, so working out this kind of interactive programming was great.

In this post I want to put to the fore an issue which comes up very often. In the RiceRocks game, there was a class Sprite, to create rocks and other objects. The sprite image is based on an image object whose center is a list:

[ x, y ]

The sprite instance has an image_center element which corresponds to the image center. It is created as follows:

self.image_center = info.get_center()

and the method in the Image class

    def get_center(self):
        return self.center

is returning the center as list. The result is that the center is a reference to the same list. So the sprite.image_center and image.image_center point to the same list.

This is not a problem for what is called in the game static images. But explosions are dynamic, which means that the image is actually a tile of a sequence of the explosion, and tiles are shown one at a time in consecutive frames as a function of the center of each tile.

In my implementation, I create a sprite for the explosion of a rock, so the rock sprite is replaced by the explosion sprite. As the time advances, I change the center of the explosion sprite to display the new frame. This worked only for the first explosion…

As I was changing the image_center of the explosion sprite, I was also changing the center of the original image. The solution was to create the sprite with a copy of the list:

self.image_center = info.get_center()[0:]

Google maps in a Rails app

My Rails app has Events which have location which of course needs to be shown in a map. With the postal address of a location, its latitude and longitude can be calculated through a geocoding conversion. With them, the location can be plotted in a map.

The Google Geocoding API does just that.

For example, by entering the following URL: https://maps.googleapis.com/maps/api/geocode/json?address=800+Boulevard+Rene-Levesque+O+Montreal in a browser, a JSON string is retrieved which includes the latitude and longitude of an office building located at “800 Boulevard Rene-Levesque, Ouest” in Montreal, Canada, where one of the events took place:


{
   "results" : [
      {
...
         "geometry" : {
            "location" : {
               "lat" : 45.5009512,
               "lng" : -73.5675947
            },
...
}

However, this is one of the last steps. Let’s start with the Rails side.

Include a map in a Rails app

Before being able to use the geographical coordinates just retrieved, we need to tell Rails to include the map. This requires some JavaScript. First, I want to include a fixed map.

I added to the app/views/events/show.html.erb the following:
script

The first script brings in the necessary tools to draw the map. It can be included in the app/views/layouts/application.html.erb as follows:

in_app_erb

The second script delays the map related scripts until the page if fully loaded. Here is a good explanation.

The third section defines how the map will be displayed. The CSS portion defining these two elements is:

#map-container {
   height: 400px;
   border-radius: 16px 16px;
   border-color: #fff;
   border-style: solid;
   box-shadow: 2px 2px 10px #B1B1B1;
   margin-top: 25px;
   border-width: 7px;
 }

#map-canvas {
   height: 384px;
   width: 100%;
 }

The script to initialize the map is as follows:

# app/assets/javascripts/gmap.js

function initialize() {
    ## Initialize the map parameters
    var center = new google.maps.LatLng(45.5009512, -73.5675947)
    var mapOptions = {
        center: center,
        zoom: 16
    };
 
    ## Initialize the map object and attach it to the element with id 'map-canvas'
    var map = new google.maps.Map(document.getElementById('map-canvas'),
        mapOptions);

    ## Initialize the marker and attach it to the previously created map
    var marker = new google.maps.Marker({
        position: center,
        map: map,
        title: "Here! Ici!"
    });
}

These scripts and code snippets produce the following map with a marker centred in downtown Montreal:

map-1

The map according to the event address

Having this in place, next is to feed the geographical coordinates corresponding to the event’s address to the initialize function to display the map. However, when I changed the function to

  function initialize(lat,lon)

and

  google.maps.event.addDomListener(window, 'load', initialize(lat,lon));

I had the following error:

 Uncaught TypeError: Cannot read property 'offsetWidth' of null

After looking high and low and bearing with my inexperience with JavaScript, I found the answer here: simply move the div defining the map-canvas and map-contianer in the show.html.erb file, above the script calling the initialize function.

Latitude and longitude from a postal address

I decided to calculate the latitude and longitude only when the postal address (map_address) is created or updated. In the controller this corresponds to the update and create methods. I wrote a function and included it in app/controllers/events_controller.rb as follows:

  private
  def lat_lon(address)
    # Replace spaces by + signs, escape any other characters and
    # convert the string into a url object.
    url = URI.parse(
        'https://maps.googleapis.com/maps/api/geocode/json?address=' +
        address.tr(' ', '+')
      )
    # Make the request to retrieve the JSON string
    response = open(url).read

    # Convert the JSON string into a Hash object
    result = JSON.parse(response)
    lat = result['results'][0]['geometry']['location']['lat']
    lon = result['results'][0]['geometry']['location']['lng']
    return lat, lon
  end

The preliminary test suite worked fine, but I missed one test. As you may know, Montreal is a bilingual city and addresses are also found in French with non-ASCII characters. Browsers tend to be more lenient and flexible, and accept characters which otherwise would throw an exception in a strict conversion into a URI element. So, https://maps.googleapis.com/maps/api/geocode/json?address=800+Boulevard+René-Levesque+O+Montréal will work just fine, having the address equal to “800 Boulevard René Levesque, Montréal” as an input to the lat_lon method above, will throw an error:

URI::InvalidURIError: bad URI(is not URI?)

The solution was to use the URI method escape to transform all the unsafe characters including the space.

Final version of code

For the sake of completeness, here are the revised scripts:

# app/controllers/events_controller.rb
require 'net/http'
class EventsController < ApplicationController
  ...
  def create
    ...
    @event.lat, @event.lon = lat_lon(@event.map_address)
    ...
  end 

  ...

  def update
    ...
    @event.lat, @event.lon = lat_lon(params[:event]['map_address'])
    ...
  end

  ...

  private
  def lat_lon(address)
    # Escape any non_ASCII characters and convert the string into a URI object.
    encoded_url = URI.escape(
      'https://maps.googleapis.com/maps/api/geocode/json?address=' + address
    )
    url = URI.parse((encoded_url))

    # Make the request to retrieve the JSON string
    response = open(url).read

    # Convert the JSON string into a Hash object
    result = JSON.parse(response)

    # Extract the latitude and longitude and return them
    lat = result['results'][0]['geometry']['location']['lat']
    lon = result['results'][0]['geometry']['location']['lng']
    return lat, lon
  end
# app/assets/javascripts/gmap.js

function initialize(lat,lon) {
      var center = new google.maps.LatLng(lat, lon)
      var mapOptions = {
        center: center,
        zoom: 16
  };
  var map = new google.maps.Map(document.getElementById('map-canvas'), mapOptions);

  var marker = new google.maps.Marker({
      position: center,
      map: map,
      title: "Here! Ici!"
    });
}

script-final

Feedback

Leave any questions or comments at the bottom! I would love hearing from you!

Helpful links

Rails: Different storage for carrierwave assets depending on the environment

I am using carrierwave for storing images, one (or more) per field of a table in a database of the my Rails app. Given that heroku does not allow assets to be uploaded by the user, or dynamically for that matter, I had to use storage in the cloud. Mind you, not a bad thing. I chose AWS, s3 services to store my bucket.

Carrierwave has an extensive explanation on setting up the right configuration variables to allow the storage on the cloud. (Carrierwave in github)

The first problem I had was running the development and staging (or production) sites at the same time: Updates in one were seen in the other in the following case:

  1. Create a field with id=1 in production. The image associated with it, is stored in the cloud with id=1.
  2. Create a field with id=1 in development. the image associated with it, overrides in the cloud, the previously created image.

My solution to solve this problem was to create a second bucket, and add in the file

config/initialize/carrierwave.rb

the setting as follows:

# config/initialize/carrierwave.rb
...
  if Rails.env.production?
    config.fog_directory = 'bucket-0'
  elsif Rails.env.development?
    config.fog_directory = 'bucket-1'
  end

The next problem was that the test suite run was affecting the development data in the same way as described before. In fact, I wanted the images for the test suites simply to be stored locally and then deleted. The first approach was to include in the same file as above the following:

# config/initialize/carrierwave.rb
...
  if Rails.env.test? || Rails.env.cucumber?
    config.storage = :file
    config.enable_processing = false
    config.root = "#{Rails.root}/tmp"
  else
    config.storage = :fog
  end

  config.cache_dir = "#{Rails.root}/tmp/uploads"

And to delete after finishing the test, add to the file:

spec/spec_helper.rb

the following:

# spec/spec_helper.rb
RSpec.configure do |config|
  config.after(:all) do
    if Rails.env.test?
      FileUtils.rm_rf(Dir["#{Rails.root}/tmp/uploads"])
    end
  end
end

That only seemed to work. I think that the display was using cache data. Only recently did I notice the problem when reworking some aspects of the app. The images in development were being overwritten by the test suite.

I had to add the following to the uploader file

# app/uploaders/image_uploader.rb
...
  if Rails.env.test?
    storage :file
  else
    storage :fog
  end

Et voilá!

When system upgrades break gem dependencies

After recently upgrading to Ubuntu 15.04 (almost ready for the next release!), my gem dependencies were broken.

... /lib/mysql2.rb:31:in `require': 
Incorrect MySQL client library version! 
This gem was compiled for 5.5.43 but the client library is 5.6.24. (RuntimeError)

Neither bundle nor bundle update changed the gem. Only uninstalling the gem forced the correct update:

gem uninstall mysql2
bundle install

Solution taken from http://stackoverflow.com/questions/10725439/incorrect-mysql-client-library-version.

Restarting unicorn after boot using mina and nginx in Rails

My teammate did a wonderful job at setting up a droplet in Digital Ocean to host a Rails app. Most of the details are here.

All of the reference to directories are based on having exactly the set up described in that blog post.

However, we noticed that unicorn was not being restarted after booting, having to do so by hand. Not quite a problem if you are on developing/staging states, but in full production a real problem. Except, we could not use a simple command line, because unicorn has to be started by mina.

Looking at the config/deploy.rb file, I noticed the following:

    to :launch do
      invoke :'unicorn:restart'
    end

so the command line to start unicorn should be:

mina unicorn:restart

Actually the command only works if in the correct directory: /home/deployer/my-app-name/current as follows:

bundle exec mina unicorn:restart

So now, all I had to do was to turn that command into a script called at boot. By the way, the droplet uses Ubuntu.

After a few attempts, I created a file called /etc/init.d/my_unicorn, you have to have root privileges for this, with the following format:

#!/bin/sh -e
# upstart-job
#

echo "starting unicorn after reboot"
exec sudo -u deployer sh -c "cd /home/deployer/my-app-name/current && /home/deployer/.rbenv/shims/bundle exec mina unicorn:restart"

(Looking back “-u deployer” might not be needed but I did not test it.)

By just adding that script, I was able to run

$sudo service my_unicorn start

But, it was not being called at boot yet. I needed to add new service (i.e. my_unicorn) to startup. For that, the following command is needed:

sudo update-rc.d my_unicorn defaults

And that should have worked on reboot, but it wasn’t.

Every time mina is called, it asks for authentication, and after 3 failed attempts it quits with an error. The solution was to create a ssh key and add it to the authorized_keys file, the same way you do for sites like github.

And that is it, if I haven’t forgotten anything!