Listing and counting huge number of files in Linux

I have a directory with humongous number of files in it. Humongous is more than more than 8 million in this case. File sizes are very small to small. An automated process creates the files. After the process finishes writing the files, which by the while takes quite a while as you can imagine, I need to know how many files were created. So, first I tried

$ ls -l | wc -l

and the machine went into thinking mode. It remained in that mode till I killed the process. So I tried something else:

$ find . -name "*" | wc -l

Here also it went into thinking modes, but returned within seconds with the answer: 8167080. find in this case was faster by a few thousand times.

So, if you have a directory with a lot of files, your friend will be find, not ls. E.g. if you want to list the files whose name starts with foo -

$ find . -name "foo*"

You can learn a lot of interesting things by looking at top and ps kind of tools' output while ls is running.

On Motivation

A very interesting short (10 min) video on motivation with great presentation.

Of course the reason it is interesting is partly because it is counter-intuitive and partly because the rampant implementation of the faulty motivational reward system in the knowledge industry. This also helps explain the motivation behind the Open Source phenomenon. Kind of.

Netconnect on Ubuntu

I had a very good experience with Ubuntu's Lucid Lynx. I was trying to get my Reliance Netconnect USB stick to work on mu Ubuntu laptop. It's much simpler than I initially thought. Even simpler than setting it up on Mac OS or Windows.

Here are the steps:

1. Put the stick in the USB port
2. Open System -> Preference -> Network Connections
3. Click on Mobile Boadband tab
   
4. Click Add
5. On the New Mobile Broadband Connection window, select HUAYWEI TECHNOLOGIES HUAWEI Mobile from the mobile broaband device dropdown (your device name may be different)
   
6. Click Forward. Select India from the Country List
7. Click Forward. Select Reliance from the Provider list (Tata Photon+ and Tata Plug2Surf are also listed there)
8. Click Forward. Click Done.
9. Select Reliance Connection in the Mobile Broadband tab and click Edit
10. In the Edit windows Mobile Broadband tab use:
    • Number: #777
    • Username: <your device's mobile number>
    • Password: <your device's mobile number>
11. Click Apply

You are done. Now, your Network Connection icon on top right panel will show and entry for Reliance Connection. Click on it to connect.

The jouney begins - Part II

The planning can be clubbed in three broad groups -

1. Legal and Financial


2. Possessions


3. Relations

For Legal and Financial, the main thing was the immigration aspect. For us, it was securing some long-term Indian visa. For some of my friends who went back, the main issue was to keeping the Green Card alive. Paramita and kids had PIO. I had a 6-month Indian multiple-entry visa that I obtained in haste in August. First thing was to apply for my OCI. Kudos to San Francisco Consulate, the process was very clean and streamlined. It took just 3 weeks to get the OCI in hand. Foolishly I didn't keep a copy of the application number. Otherwise, I could have kept track of the process in detail over internet. I did that when I applied OCI for Paramita and the kids. It also took 3 weeks. For them it was a change from PIO to OCI.

A few people has asked about the practical difference between PIO and OCI. It seems the biggest practical difference is in case of OCI one doesn't need to go and register as Foreign National every six months with the Indian Government (through outposts in most major cities).

The only complaint that I have against the OCI application process is that websites of different consulates as well as the website of the ministry have conflicting information. Some are up-to-date, but a lot are dated. Depending on which website you hit, your application process may be erroneous. Like in our case, the right amount of fee for changing PIO to OCI was misinformed. That took away some precious time and money.

After the visa, it was time make sure we get paid in India and pay taxes. It was time for a PAN card. Permanent Account Number or PAN in India is equivalent to a Tax Identification Number or TIN in US.

The PAN application process is also quite streamlined, it seems. But for people residing outside India, like us, the main inconvenience is the to obtain the attestation for the address-proof. Regular notarization doesn't work. One needs to go to the nearest Indian Consulate to get this attested. It may not be too much of a hassle if one has ample time in hand. But for our case, it was definitely a big inconvenience.

I found a service on the net, pancardnri.com. They actually run a few more websites with different domains essentially providing the same service through the parent company nriinvestindia.com. The extra service they provide is to get your address proof attested. The rest is basically creating a Demand Draf on your behalf, submit the form and mail you back the acknowledgment etc. The price seems to be steep for the work done, but basically you pay for the convenience. I availed their service and received my PAN card in mail within the promised time.

The jouney begins

It all started on a March or April morning when I received a call from India. Will I be interested in applying for a position in MyCompany Bangalore? Of course, the name of the company is not MyCompany. But, that's besides the point here.

In last 4-5 years me and Paramita have explored the idea of going back to India in a semi-serious mode. That generally involved talking to people vising US from India, especially my Brother-In-Law, who comes at least once a year in summer.  That would be followed by some more phone calls to friends in India who had made the r2i journey at different times in past.

My position was that if I return, there is no point returning to any city other that Kolkata. But as time went by, I came down from that position since with the kind of career profiles we have, it's difficult to get the right kind of opportunity in Kolkata. There are a very few product companies in Kolkata. We both have spent most our working lives in product componies. If not Kolkata, Bangalore was the natural next choice.

When the call came I agreed to explore it with an open mind.

It was a long process. There were 4-5 telephone conversation (interview) over telephone. Then there was one in-person interview at MyCompany's local office. It was August by the time the in-person interview happened. My father suddenly passed away in August. I rushed back to India.

I came back with a  changed perspective. I was no more on an exploratory path. I was convinced that we should make the move. Now. MyCompany or not. My kids are 5 and 2. This is high time to make the move, if at all, for them too.

After I returned from India, me and Paramita had some deep discussions and decided we will try to make the move now. I won't say that Paramita was totally convinced, especially when approaching the issue from an emotional angle,  but she agreed to give it a serious try.

In early September, I started negotiating on the offer with MyCompany. Paramita started preparing her resume. We started thinking about the logistics. From that point onwards it became a hectic affair. I just joined a new company. Paramita had her fully-loaded job. Megh's school session had already started. We had to keep up our regular life and added this planning and execution, mostly planning at that stage, of R2I.

Resume parsing score

I found this on Web. Being a programmer, I found it funny, but quite true.

 

[Source: http://www.hanovsolutions.com/resume_comic.png]

Wordpress and Unicode

You type some non-English character in Wordpress or Drupal editors. You save it only to find those characters have become ????. If this describes your problem, then you have an issue with Unicode. You need to turn on Unicode support for Wordpress and Drupal. It will require some change of code. Basically, both Wordpress and Drupal supports Unicode. However, by default, the databases are not configured to store unicode characters.

Wordprss

For Wordpres, the modification is straight-forward.

1. Open up ‘wp-config.php’ from the root directory of your WordPress installation.


2. Comment out the following lines by adding ‘//’ at the very beginning of the following two lines:
   define('DB_CHARSET', 'utf8');
define(’DB_COLLATE’, ”);

So that section should now look like this:

//define('DB_CHARSET', 'utf8');
//define(’DB_COLLATE’, ”);

[Source: http://hansengel.wordpress.com/2007/10/09/wordpress-unicode-and-s/]

Drupal

For Drupal, the change is a bit more involved:

1. Go to to the root directory of your Drupal installation.


2. Save the following code in a file named collate_db.php


3. 
   

   <?php
   # Do not change anything below this ( :-) <- Rewwrite Editors note.)
   require_once("includes/bootstrap.inc");
   require_once("includes/database.inc");
   require_once("includes/database.mysql.inc");
   $connect_url = 'mysql://user:pwd@server/database';
   $active_db = db_connect($connect_url);
   $sql = 'SHOW TABLES';
   if ( !( $result = db_query( $sql ) ) ) {
         echo '<span >Get SHOW TABLE - SQL Error: ' . $result . '<br>' . "</span>\n";
      }
   
   while ( $tables = db_fetch_array($result) ) {
      echo $tables[0];
   # Loop through all tables in this database
     $table = $tables[key($tables)];
   
     if ( !( $result2 = db_query("ALTER TABLE %s COLLATE utf8_general_ci", $table) ) ) {
          echo '<span >UTF SET - SQL Error: <br>' . "</span>\n";
   
            break;
             }
   
     print "$table changed to UTF-8 successfully.<br>\n";
   
     # Now loop through all the fields within this table
     if ( !($result2 = db_query("SHOW COLUMNS FROM %s",$table) ) ) {
            echo '<span >Get Table Columns Query - SQL Error: <br>' . "</span>\n";
   
            break;
             }
   
     while ( $column = db_fetch_array( $result2 ) )
     {
        $field_name = $column['Field'];
        $field_type = $column['Type'];
   
        # Change text based fields
        $skipped_field_types = array('char', 'text', 'enum', 'set');
   
        foreach ( $skipped_field_types as $type )
        {
           if ( strpos($field_type, $type) !== false )
           {
              $sql4 = "ALTER TABLE $table CHANGE `$field_name` `$field_name` $field_type CHARACTER SET utf8 COLLATE utf8_bin";
              $result4 = db_query($sql4);
   
              echo "---- $field_name changed to UTF-8 successfully.<br>\n";
           }
        }
     }
     echo "<hr>\n";
   }
   ?>
   

   
   


4. Look at the red line in the code in red (5th line). You have to replace that part with your installation-specific information.


5. Open the file sites/default/settings.php under the same Drupal root directory. Copy the rest of the line of
   

   $db_url =

   
   and paste it after
   

   $connect_url = 

   
   in the above code.


6. Save collate_db.php


7. Open a browser window and request the collate_db.php from your browser.


8. If everything is fine you would see something similar to this:
   access changed to UTF-8 successfully.
   ---- mask changed to UTF-8 successfully.
   ---- type changed to UTF-8 successfully.
   

   ---

   <--- Lines not shown --->
   

   ---

   watchdog changed to UTF-8 successfully.
   ---- type changed to UTF-8 successfully.
   ---- message changed to UTF-8 successfully.
   ---- variables changed to UTF-8 successfully.
   ---- link changed to UTF-8 successfully.
   ---- location changed to UTF-8 successfully.
   ---- referer changed to UTF-8 successfully.
   ---- hostname changed to UTF-8 successfully.
   

   ---



9. Delete collate_db.php for security.

You are done.

[Source: http://www.urbannatives.net/localtreechild/..._ci_and_tabl ]