Sunday, December 6, 2009

The Philippines and the International Date Line


Here's an interesting factoid regarding the Philippines taken from wikipedia:

The Philippines, as part of the Viceroyalty of New Spain, long had its most important communication with Acapulco in Mexico, and was accordingly placed on the east side of the date line, despite being at the western edge of the Pacific Ocean. 00:01 Tuesday in London was 17:21 Monday in Acapulco and about 08:05 Monday in Manila. During the 1840s, trade interests turned to China, the Dutch East Indies and adjacent areas, and the Philippines was changed to the west side of the date line. Monday, 30 December 1844 (ending up as a 365-day year, despite being a leap year) was followed by Wednesday, 1 January 1845.

That gives me a great idea for a short story. Whatever happened to 31 December 1844? All sorts of neat plot possibilities.

Wednesday, November 25, 2009

The NFL and How To Lie With Statistics


(In case you need a refresher on what happened, here's a summary.)

This blog post is a response to this article: Defending Belichick's Fourth Down Decision.

For the record, I disagree with that article and I think that it's a great example of how to mislead with statistics.

A number of points:

First, using the probability collected from the league as a whole is a fallacy of false analogy. The problem is right there in front of us. The probabilities are based on the league as a whole. The false analogy here is thinking that just because the numbers crunched for all teams who have attempted a 4th and 2 will apply to the Patriots. It's reducing the complexity of football down to a mere throw of the die. The oversimplification is so drastic and untenable. There is absolutely no way you can characterize a complicated situation like that by looking at the statistics of the league as a whole. Yet people read the article and believe the logic behind it.

It also misleads about what Belichick's decision process. In the end, only Belichick knows what went on in his mind. But to imply that Belichick churned the numbers in his head and made this call based on it is a total red herring. The message was loud and clear, especially to Belichick's defense. Coach Belichick did not trust his defense. Period. Do you think he would have made the same decision if he had his defense from 2002?


Secondly, the numbers used are very dubious. To quote the article:

With 2:08 left and the Colts with only one timeout, a successful 4th-and-2 conversion wins the game for all practical purposes. A conversion on 4th-and-2 would be successful 60 percent of the time. Historically, in a situation with 2:00 left and needing a TD to either win or tie, teams get the TD 53 percent of the time from that field position.

There are a couple of things wrong with this.

  • The article qualifies the 53% with the fact that this was taken from the sample space of situations with 2 minutes left needing a TD to win or tie. But the 4th down statistic is taken without such a qualification. This already skews the numbers we need. This leads me to suspect that the 60 percent number was cherry picked to make the case stronger. Alas, I don't have any way to confirm this.

  • The 4th down probability of 60% is probably too high. First of all, this takes into account both runs and throws. Given that the Pats chose to throw, it was likely that their actual probability was lower. Add to that the fact that the Colts are playing at home with the crowd on their side. Add to that the Colts have a defense that's unbeaten at this point in the season. Throw in the unquantifiable factors. The Colts defense took it personally as a "diss" by Belichick. Add to that the fact that the Pats defense morale will take a hit, not just for this game but for the rest of the season.


So, let's recalculate the probabilities using some revised numbers. The two key probabilities here are the probability of converting the 4th down and the probability of the Colts scoring from that field position.

Here's the table if you change that probability:

60% -> 79% WP
50% -> 74% WP
40% -> 69% WP
30% -> 63% WP

And that is assuming that the Colts, at home can only score 53% of the time from the 34 yard line with the game on the line.

So let's change that value assuming that the 4th down conversion is at 50% (giving the Colts an advantage at home).

53% -> 74% WP
60% -> 66% WP
70% -> 62% WP

So if we assume a coni flip probability for 4th down conversion, and give Manning a 70% probablity of converting from that field position, then the probability of a Pats win is only 62%.

Since as we already noted that the numbers quoted in the article are NFL league-wide, then it's pretty safe to assume that the actual probability for both are higher than what is quoted in the article.

It's closer to a coin flip. It's certainly not the 79% winning probability that the article would like you to believe that the Pats had.

IMNSHO, it's almost 100% certain that Belichick screwed up this game and the rest of the season big time.

Wednesday, November 18, 2009

pythonic perl 5: Use scalar

To illustrate this point, let's look at the different ways one can get the length of a list in Perl.

Let us say we have an array/list @mylist

  1. To get the length of @mylist, simply use @mylist in a scalar context. Remember that in Perl, there are 3 contexts: void, scalar and list. Using a list in a scalar context will return the length of the list.

  2. Sound easy? Well, yes and no. When reading through code in practice, it's quite painful to have to try to figure out whether something is being used in which context. And in fact, such errors will occur sooner or later. Which brings us to this technique. We use the scalar function to force the variable to be evaluated in a scalar context. And this I present to be the pythonic perl way to do it. The question you want to ask at this point is why bother typing extra characters if it's going to be evaluated in a scalar context anyway? There are a number of reasons. First it clarifies in the programmer's mind that this is exactly what you want to do. Secondly, it makes the code a LOT easier to read 6 months later when you have to go in and fix a bug.
  3. The third way has nothing to do whether to use scalar or not and uses the fact that the following is always true: scalar(@mylist) == $#mylist + 1 . My personal preference is to use scalar over this. Some people might prefer this. This is indeed a good alternative to using scalar and is preferable to the first method.


To summarize, the following will return the length of @mylist:


$len1 = @mylist;
$len2 = scalar( @mylist );
$len3 = $#mylist + 1;


How to initialize a hash from a list in Perl

Given the following code:


@mylist = qw( little tech tips );



We want to construct a hash with the elements in @mylist as the keys. We can use the x (that's lowercase x) operator for this. Let's say we will initialize the values to cool.

The code will look like this:



@myhash{ @mylist } = "cool" x scalar(@mylist);



One thing to note. The call to scalar is actually not necessary since you're using @mylist in a scalar context with the x operator. But doing it this way makes it clear and less error prone.

Thursday, November 12, 2009

Simple way to do alternation with grep (command line)

I know that the | character is supposed to be used for alternation, but I just can't seem to get it to work with command line grep. The easy way I've found to do alternation is to use -e switch like so:


grep -e someregex1 -e someregex2 file.txt

Using more than one -e on the command line implies alternation.

Tuesday, November 10, 2009

Open Source and Free on Windows XP

I have a brand new previously owned computer with a fresh install of Windows XP. (I bought it at my company's fire sale.) As I am a bit of a software geek, I love installing new software on my PCs.

Here's a list of open source and free (as in beer) software that I like to install on fresh Windows XP machines.

First up is the browser. Internet Explorer just does not hack it for me. There are a lot of security issues with version 6. A lot of the defects have been addressed in versions 7 and 8 though, but at which point I've gotten used to the following little known browser.

For the longest time, my browser of choice is K-Meleon, a lightweight browser that uses the Gecko rendering engine. This is the same rendering engine used by the folks at Mozilla. K-Meleon though is not as heavy (hence lightweight), and has a lot of features builtin. That is, no need to screw around with plugins although you can add Mozilla plugins with a little bit of work. K-Meleon has all the goodies that Mozilla does (tabs, plugins). There are a couple of things that I've grown to like about K-Meleon. First, the search box and address box have been integrated into one box. Secondly, On a right click, you can send a new tab to either the foreground or background. This one is pretty much a K-Meleon only feature that I haven't found anywhere else.


I still use K-Meleon occassionally but it has mostly been replaced by Chrome.

What is good about Chrome? What's not to like? It has the basic features we've come to expect of browsers (tabs, builtin search, extensions, themes). The layout is great. It is fast. The builtin incognito mode is very useful. It's very, very stable. And best of all it is open source. I'm hoping for a Linux port of Chrome to show up.





Up next: Open Source and Free Office/Productivity Software on Windows XP.

Wednesday, November 4, 2009

Python backticks

Backticks in Python is an operator that converts the enclosed expression into a string.

For example:

>>> print `2+3`
5
>>> print `[ 1,2,3 ]`
[1, 2, 3]



Note that I intentionally changed the formatting in the list example to show that the backtick operator evaluates the expression and prints a representation. The spacing after the comma in the output is different from how I had it formatted in the print statement.

Monday, October 26, 2009

how to remove an element in a Perl array/list

splice does the trick:

@a = qw( A B C D E F G );
splice( @a, 3, 1 );


This will remove 'D' from the @a array. (Remember that indexing starts at 0.

how to reload a file in vim/gvim

It took me a while to discover this but to reload a file in vim/gvim simply do the following:


:e


Done!

Saturday, October 24, 2009

A basic probability question concerning throwing a die


Someone asked this question on my Facebook today:

If a fair sided die is thrown 6x, what is the probability of it landing 3x on the number 6?



My answer:
[BTW, C(n,k) refers to combination. While x^y refers to exponentation.]

Suppose you number the throws 1 to 6:
C(6,3) => which of the 6 throws hits a 6. (There should be only exactly 3. That's why C(6,3).

Then you have to figure out the number of ways the other 3 non-6 throws could come out:
=> 5^3
since they can be anything from 1 to 5.

Multiply those 2 things to get the total number of ways to throw 3 sixes with 6 throws.

But that doesn't answer the problem completely. To get the probability, you need to divide that by the total number of ways to throw a 6 sided die, which is 6^6.

So the complete answer is: (C(6,3)*5^3) / 6^6


So this might be useful for someone preparing for one of the acronym standardized tests like the GRE or the GMAT. :)

Monday, October 19, 2009

Problems with connecting to Xming from a remote system

My setup is this:

Main desktop running Windows XP (it's company provided so I can't just blow the OS away and install Linux). So I installed Xming on it so I can forward X connections from my secondary PC which is running Arch Linux.

I do the proper incantation on the Linux box as:

export DISPLAY=winxpipaddress:0.0

I check that the display is indeed 0.0 on my WinXP box. But I get the following error message:

AUDIT: Tue Oct 20 :31:12 2009: 5636 Xming: client 4 rejected from IP ipaddress_of_archlinuxbox}

After some searching, find the X0.hosts file in the Xming main directory and add the Arch Linux IP. And everything works just fine.

Try Try::Tiny for easier Perl 5 exception handling

Perl 5 exception handling is not an easy thing. For one thing, it makes use of the eval construct. Any Perl code that uses eval for exception handling directly moves a notch closer to a "write-only" mode. Basically, you have to figure out whether the eval means dynamic code compilation or exception handling.

This is the reason that there are is a good number of exception handling libraries on CPAN.

There are a number of worthy contenders for the exception handling crown, but for minimalist reasons, I like Try::Tiny. First of all, it doesn't try to do too much. Secondly, it covers the cases where using eval/if becomes difficult in practice. The use of $@ pretty much dominates the difficulty in using eval/if. In a nutshell, $@ must be localized to prevent clobbering previous values. At the same time, all die statements should have the proper effect.

Try::Tiny is currently at version 0.02. For the few times that I will need to write in Perl, this is definitely a the top of my list for exception handling.

Tuesday, October 13, 2009

vim 101: how to enable line numbers

To enable line numbers, simply enter the following command in command mode:

:set number

To turn off line numbers, do the following:

:set nonumber


Saturday, October 10, 2009

how to enable incremental search in vim

It's not that hard. All you do is:

:set incsearch


The key is to actually know the words to google for. Basically the installation of gvim on OpenSuse 11.1 did not have incremental search enabled. So I searched for smart search and somehow got the right phrase I needed. Once I knew that this feature meant incremental search, it was easy.

So for those who are unclear on what incremental search is according the vim help:


While typing a search command, show where the pattern, as it was typed
so far, matches. The matched string is highlighted. If the pattern
is invalid or not found, nothing is shown. The screen will be updated
often, this is only useful on fast terminals.







EMC rolling out a compute cloud?

The Register recently broke the news that EMC will be rolling out a compute cloud that will be competing with Amazon's EC2 compute cloud service. I suppose this is to be expected after EMC rolled out its own storage cloud service, Atmos.

It should be noted that EMC bought then spun off virtual computing company, VMWare. That turned out to be a great decision on their part. So it's no surprise that they came out with cloud storage and computing solutions. I'm definitely watching this with great interest.

Friday, October 9, 2009

How to find the absolute path of a script in Python

(Or for you perl-heads, how to do FindBin in Python. Thanks to Naveed for giving me the idea for this little tip).

In Python, you'll need os.path and sys
. So it will look like something like this:


import os
import sys

myabspath = os.path.abspath(os.path.dirname(sys.argv[0]))

Wednesday, October 7, 2009

How to serve your hg repository using http

Mercurial (hg, get it? ;) is one of a number of distributed version control systems. So this is an easy way to host an hg repository. So from the command line, cd into the repository you want to host and simply type:


hg serve

The repository will be located at http://localhost:8000/. If you want to access it from another computer you'll have to use the IP address or host address like so:

hg clone http://yourpc.example.net:8000/

Very cool.

Monday, October 5, 2009

Python 101: How to iterate a dictionary/hash



For those who don't know, python dictionary == perl hash.

The most common idiom uses the iteritems function:

mydict = { "a" : "Alpha", "b" : "Beta", "c" : "Charlie" }
for key, value in mydict.iteritems():
print key, value

The iteritems function returns an iterator that gives us key and value.
If for some reason you only wanted the keys, you can use iterkeys:

mydict = { "a" : "Alpha", "b" : "Beta", "c" : "Charlie" }
for key in mydict.iterkeys():
print key, mydict[key]


Or if you wanted the values only, you can use itervalues:

mydict = { "a" : "Alpha", "b" : "Beta", "c" : "Charlie" }
for value in mydict.itervalues():
print value





Like Python? Here are some links to some more articles on Python:
  1. Little Python SQLite tutorial
  2. Little Python optparse module tutorial
  3. Little python dictionary hack

Sunday, October 4, 2009

how to start a simple HTTP server using Python

In python 2.6, just type the following command in the directory where the files to be served are:

python -m SimpleHTTPServer

In Python 3.0, SimpleHTTPServer has been merged into http.server, so change the argument to -m to http.server.SimpleHTTPRequestHandler

how to set system date/time on Linux

To set the system time on Linux, use the date command. For example to set the date/time to October 4, 11:05 pm, do the following:

date 10042305


Note that this retains the current year. To change the year to 2009, use 100423052009

Tuesday, September 29, 2009

python: How to add a directory to the module search path

It's easy to add a directory to the module search path. (Note: I like running stuff and testing new things in the interpreter. This tidbit enables me to test out modules I'm writing without having to modify the PYTHONPATH environment variable. The python interpreter rocks.)


import sys
sys.path.append( 'newpath' )

Wednesday, September 23, 2009

python optparse mini tutorial / how to

Here's a little tutorial on how to use the optparse module in the Python standard library.


  1. So far I've only been using optparse.OptionParser so I do the
    following:


    from optparse import OptionParser


  2. Define a usage string. I've been using:


    usage = "Usage: %prog [options]"

  3. Define a parser: parser = OptionParser( usage = usage )

  4. Set the defaults: parser.set_defaults( somevar = defaultvalue )

  5. Add options:

    1. Example of command line options that have arguments:


      parser.add_option(
      "--someopt",
      action="store",
      dest="someoptname",
      help="help info for --someopt" )


    2. Example of command line option that acts as a switch/toggle:


      parser.add_option(
      "--turniton",
      action="store_true",
      dest"turnitonflag",
      help="help for turniton" )


    3. Another useful argument is metavar - helps you to name the variable being addressed by the option. The default is ok.



  6. Get the options and arguments:


    (options, args) = parser.parse_args()


  7. At this point it all depends on what you want to do. Let's say we want to access the options defined above:

    • To access someoptname:


      someoptname = options.someoptname


    • To access turnitonflag and store in a differently named variable:


      myturnitonflag = options.turnitonflag


    • You get the picture.


  8. You can use parser.error( "some message to indicate what went wrong" ) to convey to user that the options didn't work.













Monday, September 21, 2009

pythonic perl 4: Finding an element in a list

In Python, you can do this:


a = "findme"
mylist = [ "no", "notthis",
"gettingwarmer",
"findme", "toomany",
"not here" ]

if a in mylist:
print "Found it!"
else:
print "D'oh! Didn't find it."


True to its roots, there is more than one way to do this in Perl. The more common ways would be to use an array (the semantic equivalent of a Python list) and grep.

But the most Pythonic way I found would be not to use an array but to use a hash. Note that the mapped values are arbitrary and are not really used. In fact, you probably don't need to define any values since we are using exists and not the defined function.


$a = "findme";
%myhash = (
"no" => 0,
"notthis" => 1,
"gettingwarmer" => 2,
"findme" => 3,
"toomany" => 4,
"not here" => 6,
);

if (exists $myhash{$a)) {
print "Found it!\n";
} else {
print "D'oh! Didn't find it.";
}


More Perl articles:

Sunday, September 20, 2009

perl 101: use FindBin

FindBin does one simple thing. It allows you to determine the full path of the Perl script you're running. This allows you to build relative paths to other parts of the filesystem that you might need.

More Perl articles:

Thursday, September 17, 2009

how to do pythonic perl 3: To carp and croak is better

Use carp and croak instead of warn and die.
The primary reason is that the carp module produces more useful information, especially when used inside modules. That is, the user is more interested in where the offending code was called from instead of where your module crapped out.

Find the Carp module here on CPAN.

Thanks to chorny for pointing out useful resources that led me to the Carp module.
It was the Perl::Critic module that led me to this particular bit and I will be doing more tips from the Perl Best Practices book by Daman Conway

Wednesday, September 16, 2009

how to do pythonic perl 2: Explicit > Implicit

As a followup to my earlier post, this is an even better way to do it, IMNSHO.

Don't use $ARG or $_ at all. Assign to a variable that explicitly names the variable you expect.

Because the Pythonic way is: Explicit is better than implicit.

Let me just say now that if you don't like the Pythonic way, that's cool. I'm not here to beat you over the head with it. So please give me the same respect I give you and just move along if you don't find this blog to your liking. If mocking other people is more to your liking, go do it somewhere else.

Tuesday, September 15, 2009

how to do pythonic perl: $ARG is not argh

Well, it had to happen sooner or later. I'm forced to maintain someone else's Perl program. I really don't like Perl. (And as I mentioned before, don't take it personally Perl-mongers.)

So here's my series of articles on how to make Perl more Pythonic. (I know this is heresy and blasphemy in some circles, but hey, it's a big world out there.)

One of my primary gripes is the use of $ variables. One in particular. $_ which is the default input and search space variable.

In fact, a lot of perl programmers will leave it out since it is implied by default. Which makes for totally unreadable code unless you know the secret handshake and secret password.

So, my first step to wading in the Perl pool is to use the alternative form of this variable:

$ARG

So whenever you feel like using $_ or using $_ implicitly, please. Please use $ARG.


:)

Stay tuned for more "how to do pythonic perl" articles.

Update: Here's the next parts to the Pythonic Perl series:

vim: enable line numbers in printout

If you want to print line numbers like I sometimes do, use printoptions to control this.
Do the following to enable line numbers in the printout:
:set printoptions=number:y


If you wanted to enable line numbers in vim (on the screen and not on the printout), see this post: vim 101: how to enable line numbers in vim

Sunday, September 13, 2009

vim: how to split a line

In vim/gvim, you can join two lines using the J command. So I was wondering if there was a simple command to split a line.

After much searching, I found that there really isn't one, but the replace command r, can do so by placing the cursor on the whitespace you want to split and doing the following:


r[Enter]


where Enter means press the enter key.

This works pretty well for splitting a line without having to enter insert mode.
It's a little tip, but it's the little things in life that make life a little brighter. :) Obviously it has to have whitespace at the point of the split to make this work. Otherwise, going to insert mode is probably faster.

Friday, September 11, 2009

python dictionary little hack

The situation is this: Let's say I have a dictionary that maps to dictionaries.
I want to get a list of all the keys in the secondary dictionaries, take out duplicates and return as a list.

So let's say we have something like this:
d1 = { 1 : "aaa", 2 : "bbb" }
d2 = { 22 : "yyy", 33 : 'zzzzz' }
d3 = { 141 : 'xyz', 123 : 'abc', 231 : 'lmnop' }
myd2d = {
'a' : d1,
'b' : d3,
'c' : d2 }
The desired answer is
[1,2,22,33,141,123,231]
Not necessarily in that order. The straightforward way to do it would be:
result = []
for k,v in myd2d.iteritems():
for mykey in v.keys():
result.append( mykey )
I suppose the other way might be more pythonic:
r = sum( (v.keys() for k,v in myd2d.iteritems()), [] )

Thursday, September 10, 2009

how to un-highlight vim searches

The problem is basically this. If you have hlsearch set, then after you've searched for an expression, it remains highlighted. (I believe this is the default behavior in gvim. In vim on the console, I've seen the opposite.)

I got into the habit of turning off the highlight by searching for a bogus expression like


/asdfasdasdf


Turns out you can do this:


:nohlsearch


to turn of the highlighting.

If you do:


:set nohlsearch


then it turns off highlighting of searches compeltely. :)

Where to find Google Apps Standard Edition

So I was looking for the standard (read: free ;) edition of Google Apps. I did it already once and it was pretty easy. I had a hard time looking for the standard edition since the landing page only seemed to advertise the premium (read: give Google your money ;) edition.

After a little bit of googling (ironic isn't it?), I found it here.

I did notice one change. The number of users allowed went from 200 to 50. :-/ Oh well. :) It's still pretty awesome.

Up next: how to set up Google Apps Standard Edition

Wednesday, September 9, 2009

how to iterate through a hash in Perl

Yes, I'm a Perl newb. And I really don't like Perl much. But I still have to read through Perl code and sometimes modify it. So here's how a bit of Perl 101 for myself.
Here's how to iterate through a hash in Perl:


foreach my $key (keys %myhash)
{
print "key: $key value: $myhash{$key}\n";
}


I like this construct because it's pretty clear what's happening.

Ad: Learning Perl, 5th Edition

how to flatten a list of lists in Python

Say we have:

mylist = [ [ 1, 2, 3, 4 ],
[ 'bb', 'uuu', 'bbbb' ] ]


There are different ways of flattening mylist:

  1. The first is the obvious way:

    result = []
    for x in mylist:
    for y in mylist:
    result = result.append( y )


  2. The next one is my preferred way using sum and a list comprehension:

    result = sum( mylist, [] )


  3. The third way is an obfuscation of the previous one

    from itertools import chain
    result_iterator = reduce( chain, mylist )

    Or if you want a list instead of an iterator:

    result = [ x for x in reduce( chain, mylist ) ]



If anyone can think of anything else, I would like to hear about it.

Wednesday, September 2, 2009

vim negative match using negative look-ahead assertion


The second part to the mildly popular vim regex negative match post. This time we'll use the zero-width negative look-ahead assertion feature in vim.

I'll show 3 different ways this can be used.


  1. The first one is in my opinion, the most useful. Suppose you want to find all instances of foo which are NOT followed by bar. For example, we want to match foobear but not foobarstool. The vim search command for this is:

    /foo\(bar\)\@!


    So the first part is easy. We want to match foo, so that starts the regex. Next we want to indicate that we don't want bar following foo, so we put bar and the negative look-ahead assertion multi indicator
     \@! 

    The negative-look ahead assertion multi applies only to an atom though. So if we didn't put the parentheses around bar, the assertion would only apply to the letter r. So that's why we need the parentheses.

  2. The second way to do use it is simply apply the negative look-ahead assertion to an atom without a match in front of it like so:

    \(bar\)\@!

    Try it and you'll see that it is pretty useless as it will match most everything. (Exercise: how can you get this expression to not match a certain spot? :)

  3. The third way to use this can mimic the

    :v/somepattern/p


    command described in an earlier post. So do this:

    /^\(\(.*sommepattern.*\)\@!\).*


    This kind of mimics the global command. I'm not entirely certain if the behavior is the same on blank lines (ie. lines withonly the newline/CR character on it). This one is mostly academic though as I would use the global v command anyway. :)








Wednesday, August 26, 2009

project announcement: bingcode on Google Code: searchfiles.py

I've started a new open source project on Google Code called bingcode. It is a collection of python scripts that I've created. Nothing spectacular, just thought I'd start getting some stuff out there.

The first python script committed to bingcode is searchfiles.py. It is a simple utitily script to search through a list of files with a list of words. It's still in a very basic form. It simply prints out the list. I will improve the output in the next commits, but I thought it was best to get something out there.

Please leave any comments and feedback.


PS. It is called bingcode because that is my name. I've been using the name Bing longer than Microsoft. So there.

Monday, August 24, 2009

how to strip trailing characters in Python

Oddly enough, you use the rstrip command. :)

Simply import the string module:

import string

Suppose you have a variable oldstring that has trailing spaces, then do this to the string:

newstring = string.rstrip( oldstring, ' ' )

Done.

Monday, August 17, 2009

how to get information about a video file using ffmpeg

For ffmpeg, it's simple and easy. Simply:

ffmpeg -i yourvideo.avi

and it will print out some basic information about your video like such:


FFmpeg version SVN-r18866, Copyright (c) 2000-2009 Fabrice Bellard, et al.
configuration: --enable-nonfree --enable-libfaac
libavutil 50. 3. 0 / 50. 3. 0
libavcodec 52.29. 0 / 52.29. 0
libavformat 52.32. 0 / 52.32. 0
libavdevice 52. 2. 0 / 52. 2. 0
libswscale 0. 7. 1 / 0. 7. 1
built on May 10 2009 23:45:03, gcc: 4.1.2
Input #0, avi, from 'yourvideo.avi':
Duration: 01:32:47.92, start: 0.000000, bitrate: 1057 kb/s
Stream #0.0: Video: msmpeg4, yuv420p, 352x288, 25 tbr, 25 tbn, 25 tbc
Stream #0.1: Audio: mp3, 44100 Hz, mono, s16, 112 kb/s
At least one output file must be specified

Thursday, August 13, 2009

how to print in vim without using the GUI

The command for printing in vim is hardcopy:


:[range]ha[rdcopy][!] [args]


range defaults to the whole file.
On Windows, the print dialog pops up unless you use !.
I hardly ever use args so I don't know anything interesting about it. :)

If you want to print line numbers like I sometimes do, use printoptions to control this.

Do the following to enable line numbers in the printout:


:set printoptions=number:y


Easy!

Monday, August 10, 2009

what is a UUID?

I recently just installed a not so recent Ubuntu distro (8 point something...) and found the fstab to be a little more cryptic than I remembered. Basically instead of the standard /dev/sda* that I knew, it listed a:

UUID=somelonghexadecimalstringwithdashes

After a bit of search, I found that this was called a Universally Unique Identifier. In short, it allows us to identify the disk partition with the knowledge that it won't be mistaken for something else. At least that's what the wiki page tells me. :) So I'm not really sure why this works, but at least I now know what that thing is.

To list the UUIDs for your devices, just run:

sudo blkid

Here's the relevant link on the Ubuntu help pages on fstab.

You learn something new everyday, eh? :)

Sunday, August 9, 2009

how to print to string in Python

Here's how to do it. You have to use the StringIO module.


import StringIO

datastring = StringIO.StringIO()
...
print >>datastring, " Hello string."

Thursday, August 6, 2009

why i hate reading perl code (aka why I hate Perl)

After going through line after line that looks like this:


my $targetFooThingie =
"$object{fooBarDir}/$fooType/$config_layerThingieName[$layer]";


my nose starts to bleed....
I hate the embedded variable thingie (aka variable interpolation). And the total abuse of punctuation marks just makes things worse.)


PS. perl zealots and fundamentalists, go away. This is my blog and I'll post whatever I want. Thoughtful and insightful comments are most welcome. (Not that I've ever gotten a comment here. Too insignificant of a blog.... ;)


Added 13 August 2009

Given the comment by chorny, I feel the need to explain this point.

It's not the string interpolation per se that I find nosebleed inducing. Like he said, string interpolation in some form is present in all major scripting languages. What I find totally annoying is the way Perl does it. IMNSHO, it was a bad idea to to it The Perl Way as shown above. It makes a simple string concatenation operation doubly difficult in terms of deciphering what's going on.

As I don't know Ruby or PHP, I can not comment on those languages. And given that it looks like PHP is probably a clone of Perl, I probably won't be learning that language any time soon. And I hear that there are a lot of Ruby fundamentalists/zealots given that it's the latest Cool Language, i probably won't be in a rush to learn it either.

Wednesday, August 5, 2009

apt search for packages on Ubuntu / Debian

Sure there's a gui for doing this, but sometimes you're adminitering or working on a Debian/Ubuntu box and you need to figure out the package name for that software package you need.

The command is

apt-get search

Useful options are:

--installed
--names-only


The first option searches only the installed packages. The default is to search both installed and non-installed packages.
The second option searchs only the names. The default is to search the name and description.

Tuesday, August 4, 2009

how to print the current stack in Python

a little screen shot of the source code. (Why a screen shot? Well because i wanted to post some purty pictures... :)

stacktrace-src

Wednesday, July 29, 2009

Travelling Salesman Problem is NP-complete

It's been a while since I was last in a classroom studying computability and complexity.

This is one of the most interesting results and it's worth remembering.

The TSP is described succinctly on wikipedia:

Given a list of cities and their pairwise distances, the task is to find a shortest possible tour that visits each city exactly once.


The key is that you have to visit each city exactly once and that it should be the shortest such tour.

Find a lot more about the TSP on wikipedia. And if you don't know what NP-complete is, here is the wikipedia page.

Why is this important? As software engineers and programmers, we should be aware of the limits of what we can do with our craft.

little tidbit on polymorphism in C++

Here's another interview question that I've come across:

Polymorphism in C++ is supported with the use of dynamic binding with the use of virtual functions.

There are two things to do when you want to use dynamic binding:

  1. Only member functions that are declared virtual are dynamicallly bound.
  2. The call must be made through a pointer or reference to a base class.


When these conditions are met, the function call is dynamically bound. Otherwise, the function is resolved at runtime. And dynamic binding is how C++ supports polymorphism. (Although I believe templates support a different kind of polymorphism, dynamic binding polymorphism is what people mean when they simply say polymorphism...)

Tuesday, July 28, 2009

how to check CPU and memory usage on a Linux box

I was asked this question once on a job interview. The interview was supposedly for a C++ developer role but they were considering me for a Linux/DB admin job. (Not really sure why. :)

I answered top because that was the only command I ever used on my Linux boxen. And after the interview, I remembered the uptime command.

So I finally decided to figure out what a good answer would have been in addition to those two commands.
And the winner is:


vmstat


The most important fields here are:


  • free - amount of physical memory not in use

  • id - percent of CPU time that is idle
  • wa - percent of CPU time spent waiting for IO

Tuesday, July 21, 2009

vim sort lines

This is actually very easy. All you do is:

:sort

If you only want to keep unique lines, do the following:

:sort u

Done!


Wednesday, July 15, 2009

Using ffmpeg to convert video files for the PSP

Well first you have to install ffmpeg. I will save that for another post.

Let us assume that your ffmpeg is capable of encoding for the PSP platform. The command to convert to a PSP format is:

ffmpeg -i yourfile.avi -f psp -r 29 -b 768K -ar 24000 -ab 64k -s 320x240 M4V00001.MP4

To be honest, I'm not really sure what the parameters mean. (Again, save for another post :). But this command works for me. One thing of note is the -s parameter which refers to the size. The PSP format needs both dimensions to be divisible by 16. You can use -s 368x208 for widescreen videos.

To create the thumbnail file:

ffmpeg -i yourfile.avi -f image2 -ss 5 -vframes 1 -s 160x120 M4V00001.THM

The -ss parameter refers to which frame to use. I believe the units for this is seconds.

Stay tuned for more PSP/ffmmpeg posts. I will also post how to convert using mencoder

Monday, July 13, 2009

google front page!

The admittedly simple post, String Concatenation In Bash, has made it to the front page of google when searching for "bash string concatenation."

Here's proof. w00t. Not bad methinks. Not that I get a lot of traffic on this blog but I like to keep it going for personal satisfaction. :) And to save (and share) a lot of the little things that I learn along the way.

subversion: backing up your repository

Here's a simple way to do it:

svnadmin dump /path/to/repo/ | bzip2 -c > datetoday-repo.bz


Monday, July 6, 2009

vim paragraph reformat (justify or align)


I like editing my plaintext files (and some of my source code) at

:set textwidth=78

The problem is that vim does not automatically justify or realign when adding/deleting text. The simple way to do it is to select the lines to format and use:

gq


Done!
Note that since gq is an operator that you can use other ways to select the text to format, but visual mode is the most intuitive for me.

You could for example do this:

gq21j

The downside is that you'd have to know the line count. A more useful text movement command to use with gq is ) (which moves you to the end of the current paragraph. So you'd do this:

gq)

Friday, July 3, 2009

how to eject stuck cd/dvd in linux

I'm using Opensuse 11.1. And it refuses to eject a DVD data disc I have inserted. This is the method I used to eject it.

First, find which processes are accessing it using the lsof command (lsof stands for list open files.) On OpenSuse 11.1, I used:

lsof /dev/sr0

This gave me the PID of the "offending" process. Which I was able to kill. Then eject as normal.

Tuesday, June 30, 2009

Project: python graphics render package

Once upon a time, back in my grad school days, I was really into computer graphics. I haven't really done much CGI since then. So I've decided to get back into computer graphics as my hobby.

Why Python? Well because Python is da shiznit. I plan to build a renderer. It will be a learning experience so I will recreate the path I took in grad school. I will be making design decisions along the way. At this point, I don't know what to reuse and what to invent from scratch. But I do know that I want to implement most of the algorithms in Python. Maybe if this project is successful, I will rewrite the performance critical parts in C. But that is a loooooooong way off. One step at a time. :)

Monday, June 29, 2009

python graphics link-o-rama


  1. Tkinter is the standard default GUI for Python. The documentation is here.

  2. gnuplot.py as a package that interfaces to gnuplot.

  3. PyQwt is a set of bindings for the Qwt class library which is written in C++.

  4. PLplot is snother scientific plotting package. It has bindings for Python and other languages.

little link-o-rama 1

I like making lists. So here you have a list of links of useful stuff!

1. A list of 23 starter kits for web designers.

2. 53 digital photography tuotorials.

3. Who isn't on facebook these days? 20 facebook tips.

4. 10 ways to get cables under control

5. The couch potato to 5k running plan. The best little tech tip I can give you is, keep your body well maintained. :)

Thursday, June 18, 2009

vim: case insensitive search


vim/gvim has a default case sensitive search which is what I want most of the time.

There are times however when I need case insensitive searches. There are 2 ways to do this:


  1. First you can do

    :set ignorecase

    This sets the searches to ignore the case. Use :set noignorecase to revert to case sensitive search.


  2. The problem with that is if (like me) you only want to ignore case just for one search. It's a lot of keystrokes just to do this for one search. This takes care of that case. Use \c anywhere in the search pattern to ignore case. This does not touch the global setting of case.


Easy as pie. :)



Ad: Vim books

Ad: Regex books

Thursday, June 11, 2009

python sqlite trick / tidbit

I found myself doing this a lot in Python when using sqlite3:



conn = sqlite3.connect( 'somedb' )
cursor = conn.cursor()
cursor.execute( 'SELECT * from sometable' )
for row in cursor:
do something with row


I found it easier and much more fun to use list comprehensions:



conn = sqlite3.connect( 'somedb' )
for row in [x for x in \
conn..execute( 'SELECT * from sometable' )]
do something with row


We even dispense with the cursor altogether. After discovering this, I found SQL to be a lot less painful.








python: how to flatten a tuple

Basically the problem was that I wanted to create a flat tuple from a tuple and a single value like such:

val = 3
tup = ( 'a', 3.14, "zzz" )


I wanted this:

( 3, 'a', 3.14, "zzz" )

not this:

( 3, ( 'a', 3.14, "zzz" ) )


It turns out that this is exactly what the '+' operator does for tuples.

In general, adding two tuples creates a bigger flat tuple from the elements of each.
For this problem the answer then was:


(val,) + tup

Easy. :)


Note:

(val,) results in a tuple with a single element.



Ad: Python books on Amazon

opening a unicode file in vim/gvim

This was the easiest way I found:


:e ++enc=utf-16 filename

python: finding the maximum element in a list

Actually it is very easy, as I discovered. Python has a built-in function, max. So it is simply:

maximum = max( mylist )

That easy. The problem that I had to solve involved a list of tuples. And I had to find the maximum of the first element. There's a very cool way of using either generator expressions like so:

maximum = max( (x[0] for x in list_of_tuples ) )

You can also do it with list comprehensions as such:

maximum = max( [x[0] for x in list_of_tuples] )

Now the big question is, which is better? I have no clue. Stay tuned for the answer. I will try to find out.

Wednesday, June 10, 2009

python tidbits

1. How to change the working directory in python:

import os
os.chdir( )


2. How to add a directory to the module search path from within the interpreter. (Note: I like running stuff and testing new things in the interpreter. This tidbit enables me to test out modules I'm writing without having to modify the PYTHONPATH environment variable).

import sys
sys.path.append( )


Monday, May 25, 2009

String Concatenation in Bash

I needed this for a small script I was doing and it turns out it's very easy.

$a="zig"
$b="zag"
echo $a$b
--> zigzag

Or this:

echo $a"ziggity"$b
--> zigziggityzag

Tuesday, May 5, 2009

little python sqlite tutorial



Sqlite is an embedded database engine that has bindings in a lot of popular programming languages (Perl, C++ and most notably Python). Sqlite3 is part of the standard Python library so there's no need to muck about with installation.

Here's my mini tutorial to using sqlite3 in Python:

1. First thing you have to do is import the module so:

import sqlite3

2. Next is to connect to a 'database' which is in reality a single file in the filesystem. Note that this will be created in the working directory of your current process (in this case the Python interpreter). Let's use an absolute path to make it clear:

conn = sqlite3.connect( '/yourpath/exampledb' )

3. Once you have a connection, you probably will create a cursor. You need a cursor object to use its execute method to perform SQL commands:

cur = conn.cursor()

4. Run your SQL commands. Note that like most database engines, sqlite3 supports its particular dialect of SQL. Check it out here: SQL As Understood by SQLite.

Here are some examples:

- to create a table:

cur.execute( '''CREATE TABLE books (title text, author text)''' )

- to insert a row of data:

cur.execute( '''INSERT INTO books VALUES ('Judas Unchained', 'Hamilton, Peter' )''' )

5. Most importantly, to save your data, issue the following command:

conn.commit()

6. To close the database:

conn.close()


That's it! :)

Here are some more Python links:


Here are some online docs:
Python docs for the sqlite3 Python bindings: DB-API 2.0 interface for SQLite databases

SQLite homepage is here.








Sunday, April 26, 2009

encrypting from the command line

This can be done from Linux, or any of the BSDs that have the openssl tool.

To encrypt

openssl enc -bf -in infile -out outfile.enc

enc is the encrypt command
-bf tells openssl to use the blowfish cipher
-in and -out are self-explanatory.
(I use the enc file extension to classify files that have been encrypted)

To decrypt:

openssl enc -f -bf -in infile.enc -out plainfile

-d add this flag to indicate that we are decrypting.

Note that you probably should choose a really passphrase.

Wednesday, April 22, 2009

vim: trailing spaces

Part of our coding standards is the absolute prejudice against trailing whitespace.

I often run the following vim regex to look for such creatures:

/\s\+$

Even better yet, add the following to your vimrc to show you where these pesky creatures live:

syntax match ToDo /\s\+$/


Note: I used ToDo out of personal taste. Some people might prefer to use Error. :)

PS. Related post: vim: How To Highlight Unwanted Spaces


vim: how to undo the undo

One of the first things you learn in vim is how to undo changes using u. (Duh!)

The quetion is, how do you undo the undo if you've accidentally pressed u?

Well the answer is:

:redo

or shortens to:

:red

(note, in case you haven't figured it out yet, the bits and pieces here are not earth shattering. Since nobody really even reads this blog, I really only use it for myself.)

Thursday, April 16, 2009

vim: seeking %

I use vim for my text editing needs. And I've recently been coding a lot in Python. Python, as you may well know, is a "{}"-less language. It depends on semantic whitespace to mark out blocks of code.

I find this to be very intuitive. But the one place I miss the braces is when I need to find the beginning (or end) of a block. Normally one can use the "%" command in vim to do this in C code. Alas no such luck for Python. (Remember the semantic whitespace? :)

(Note: I'm misusing the term "command" here. The more accurate description is cursor motion command, but that seems rather long. ;)

I found a little hack to get around this obstacle. Just make sure that Python blocks don't have empty lines in them. The aptly named "{" and "}" commands take you to the beginning and ending of a paragraph respectively. (Just make sure there are no empty lines.)

python global pre version 3

A snippet from the Python tutorial:

A special quirk of Python is that – if no global statement is in effect – assignments to names always go into the innermost scope. Assignments do not copy data — they just bind names to objects. The same is true for deletions: the statement del x removes the binding of x from the namespace referenced by the local scope. In fact, all operations that introduce new names use the local scope: in particular, import statements and function definitions bind the module or function name in the local scope. (The global statement can be used to indicate that particular variables live in the global scope.)

This may save a beginning Pythonista lots of subtle bugs later on. :)
nonlocal is a new language feature in version 3 that I will write about some time soon.

Wednesday, April 15, 2009

distroaddict: Qimo 4 Kids

Found a new Linux distro, Qimo 4 Kids. It claims to be appropriate for kids aged 3 and up. I've been thinking of coming up with a new distro, and this seems to be an underserved niche of the distro market. Stay tuned for a review of Qimo.

vim regex negative match

I know how to negate a single character match using the

[^x] - for a single character
[^a-z] - for a range of characters

I needed to be able to do a negative match (that is, match everything EXCEPT the regex pattern) for a project.

The easiest way I found that worked for my little problem was using the

:v/someregex/

command. This is similar to the :g// command except that it returns everything except the regex matches.

I think that you might also be able to use the negative lookahead regex pattern. I will save that for a future article.

Update: The negative look-ahead assertion articles is here!

Wednesday, April 8, 2009

gimp mini tip 1

I'm a learn as I go gimp user.

When fooling around with layers and transparencies once, I found on the web how to erase/unerase.

To erase simply use the "eraser" tool (duhhh).
To unerase, use the eraser tool while holding down ALT.

Simple!

simple webscraper in python

Found this on the python lists:

import os
site = 'website.com'
X = os.popen('lynx --dump %s' % site).readlines()


Obviously, you need lynx on your system. Links and elinks can probably be suitable replacements.

Coolness.

Sunday, April 5, 2009

hex editor for Linux

I needed a hex editor to search through a binary file. The best one I found was the Bless hex editor (http://home.gna.org/bless/).

I also tried tweak, hexcurse, vche and khexedit.

tweak, hexcurse and vche are good because they can be used on the console. Bless seems better than khexedit. Bless made it easy to view the binary data in different format, so that's what made it work for that particular use. I'm sure I can find a use for the other editors eventually, especially the console ones.

Looking for Linux books?

Wednesday, April 1, 2009

using links to download ftp files nonanonymously

How's that for a very specific tip?

Well I didn't want to emerge ftp in my gentoo installation and links was there. So I took it as a challenge to figure it out. I seem to remember being able to do that a long time ago. Found the answer here.

In short do this:

links ftp://username:passwd@hostname.sample/directory

In other words, it's not very secure. :) But sufficient for my internal network ftping needs...

Of course ncftp r0xx0r, so this is a very limited use tip.

opensuse 11.1

I love trying out new distros and it's been a while since I tried installing Suse. Back when I tried it, it was still called Suse. Now it's opensuse and they're at version 11.1.

Highlights include:

- KDE 4.1.3
- pretty fast graphics even using a software OpenGL implementation
- software installation is pretty slick (Yast is pretty good)
- (btw, pure_ftpd is a lot faster than vsftpd. not sure which is more secure, but I only use it internally anyway...)

Up next, PCBSD...

figuring out the MAC address

I needed to find out the MAC address on a Windoze XP box. On Linux, one simply types ifconfig. The equivalent ipconfig on WinXP did not do the trick.

Turns out you have to supply the /all as such:

ipconfig /all

Little things. (This IS the little tech tips blog after all. :p)
That raspberry goes out to a readership on none. ;)

Thursday, March 26, 2009

Visual Studio - symbols not loaded

I had a problem that I could not set any break points in the debugger because Visual Studio could not load the symbol file (pdb). After messing around with the debugging settings on Visual Studio 2005 (and wasting a good part of a day), I decided to restart Visual Studio.

Then it just started working.

Thanks Microsoft for yet another example of "oh that's fixed when you restart" engineering!

NOT.

Wednesday, March 25, 2009

Software Find List 1

I'm starting a new series of posts which basically lists new software that I've found and am actively trying out:

1. Gentoo distro - tried this once a long time ago. Now seriously considering using it as a basis for a console only distro.

2. JMEC64 - A Commodore 64 emulator written in Java. Found out you had to compile it which leads to number 3 on the list...

3. Netbeans IDE - wanted to run #2, so ended up here.

4. Git - the latest and greatest in the version control race. After I just made the transition to subversion from cvs, something new and shiny pops up. :)

Sunday, March 22, 2009

hexdumps in Python

I once had to decode hex strings that were in ASCII to the proper hex values.

I found this on the web:

map( ord, codecs.decode( '030109ff', 'hex' ) )


The more pythonic way would then be:

[ ord(x) for x in codecs.decode( '030109ff', 'hex' ) ]

Cool.

Monday, February 23, 2009

how to delete blank lines using sed

Easy one liner for this:

sed '/^$/d' < inputfile

cr/lf linebreak madness

I get these log files in my work that are supposed to be text files. Supposed to be. For some bizarre reason the CRs are there (represented by ^M) while the linefeed character seems to be missing. So doing this:

sed 's/^M//g' < inputfile.txt > outputfile.txt


does NOT work.

After much searching and experimenting, this works:

perl -ne ' s/:/\n/g; print ' inputfile.txt > outputfile.txt


I have yet to find a sed version that works. Stay tuned.

PS. Check out the wikipedia entry on this for other ideas: http://en.wikipedia.org/wiki/Newline