Nikoloogle Lindbloogle

Scala blunder: appending to a Seq that is a List

2011-10-21T15:50:00.000+02:00

I recently made a mistake in a loop reading lines from a file, doing some string manipulation and adding the result to a collection. A seemingly trivial Scala script just refused to halt. My mistake is illustrated by the following two toy examples, adding integers to a Seq and a Vector, respectively:

var x1 = Vector[Int]()
for(i <- 0 to 100000) { x1 = x1 :+ i }

var x2 = Seq[Int]()
for(i <- 0 to 100000) { x2 = x2 :+ i }

One of the above for loops runs about 38,648 times slower than the other one (according to a single, somewhat sloppy benchmark using Scala 2.9.1).The explanation, I believe, is that the Seq turned out to be backed by a List. Lists hate being appended to (:+), and this hatred manifests itself in bad performance. Good to know if you want a program to be impressingly slow.

By the way, this made me think of another one:

var s1 = ""
for(i <- 0 to 100000) s1 = s1 + i

var s2 = ""
for(i <- 0 to 100000) s2 = s2.concat(i.toString)

I don't know why you'd want to create a string like the above, but the version using + is about four times slower than the one using concat (Scala 2.9.1).

Programming Scala without... anything

2011-09-28T13:51:00.000+02:00

You don't need all that fancy, modern stuff. A keyboard, a terminal window and the scala command are all you need:

$ scala -e 'println(io.Source.fromFile("freq_list.txt").getLines().map(_.split("\t")(0).toInt).sum)'
71213401

(Prints the result of summing the frequency numbers found in the first tab separated field of file freq_list.txt. The result turned out to be 71213401.)

When the programs get longer, you better stay focused.

Testing Scala 2.9.0 (RC2) parallel collections: four extra key strokes, double speed

2011-04-29T19:20:00.003+02:00

We have just tried the new parallel collections that you can find in Scala 2.9.0.RC2.

By adding .par at a few places, the software we tested ran almost twice (1.9 x) as fast on a two core processor. Running the same code on a four core processor was, as expected, quicker (2.7 x), but not four times as fast. That's quite a performance boost, with close to zero programming effort.

The software we've tested validates (electronic) pronunciation dictionaries, where each entry has an orthography, a phonetic transcription and some other stuff. The program runs a large number of quality checks to find problems (faulty transcriptions, inconsistencies, etc) that are hard or impossible for a human lexicographer to find. It runs hundreds or even thousands of validation rules, using regular expressions and other string processing, on a hundred thousand or more dictionary entries.

The software runs a sequence of validation rules on each input entry. The validation rules are independent of each other, suitable for running in parallel. The rules, living in a Seq, are applied in sequence in a call to map(...). By calling .par.map(...) on the Seqs holding the validation rules, a multi-core processor is now able to perform the validation in parallel (par returns a parallel version of a collection).

Apart from using parallel collections at the point where the validation rules are run, we also run the main loop, reading the input lexicon data, using a parallel collection. Adding parallel collections at different places (the outermost loop and inside the validation) seems to add to the performance gain.

An initial problem that we had, was that the Scala 2.9.0.RC2 API documentation fooled us to believe that foldLeft would, just like map, run in parallel. That appears to be incorrect. We had to change calls to foldLeft into calls to map (followed by an additional foldLeft to aggregate the result). I don't know if I've misunderstood the documentation, or if parallel foldLeft is pending.

Anyway, double speed, or more, with zero effort. It sounds too good to be true, but this quick test suggests that it works like a charm.

And now I want more cores.

Interview with Maxime Lévesque, author of Squeryl

2010-09-15T10:20:00.000+02:00

Squeryl is a great Scala database API. On its website, it is describe like this: "A Scala ORM and DSL for talking with Databases with minimum verbosity and maximum type safety".

Preparing an introduction to Squeryl for a Swedish computer magazine, I sent a number of questions to Maxime Lévesque, the man behind Squeryl. The answers were so interesting, that I asked his permission to post them here:

Could you describe yourself in a few words?

I'm a dad, a programmer, a hobbyist bass player and percussionist.

I'm the kind of programmer who prefers to write libraries and frameworks to writing applications. If I was in the construction industry I'd probably be making bricks, mortar and nails rather than houses.

Do you develop Squeryl as part of your work, or is it a hobby?

Squeryl started as a hobby, only later did I start using it in a commercial project.

What are the most important features of Squeryl? Why should you use it?

The main reason to use Squeryl in an application, in my opinion, is to have the data access code validated by the compiler. I've seen many projects where the database schema stops evolving after a lot of code has been written against it. Ugly workarounds are sometimes chosen because there isn't enough time to investigate the repercussions of a schema change or conduct all the testing required.

Strongly typed languages are good for "deterministic refactoring". A data access layer needs to be refactorable, as any part of a system does. Perhaps to an even greater extent, because in a sense, bad design decisions get persisted with the data.

A developer needs all the help he can get from tools such as compilers and IDEs. Hard work and discipline don't scale. Why rely on it when you can have automated validation?

Reusability is another big one. Squeryl queries are composable, reusable pieces of code. A query that encodes a particular piece of application logic needs only be written once, and reused anywhere it is needed. I'm a big believer in the DRY principle (Don't Repeat Yourself).

Low verbosity would be another strength. I dislike APIs or frameworks that require you to write more than you should.

What's the story behind Squeryl?

In 2005 I wrote an ORM for dotNet. I was in need of one at the time and I couldn't find a decent one that exploited generics and annotations, so I wrote my own. By the time I considered publishing it, LINQ came out, and instantly obsoleted my ORM (and all other ORMs except HaskelDB in my opinion).

A few years later I started to write a query DSL in Java, and at every step, I got bitten by language limitations. Every time I worked around them, the solution became a bit more ugly and verbose. I then discovered Scala, and started experimenting with writing a statically typed query DSL. I was amazed by the expressivity of the language.

The fact that it was possible to write Squeryl as a library (i.e., without a compiler plug-in) speaks a lot about the potency of the language. The first two attempts were abandoned when they reached a critical level of inelegance. They were Squeryl's pre-history.

Squeryl is in fact my third attempt at a Scala ORM. When I became confident that a fourth rewrite wouldn't be necessary, I published it on GitHub.

If Squeryl didn't exist, what would you use?

If Squeryl didn't exist, I'd have a look at ScalaQuery or Circumflex. I only have a superficial knowledge of them, but I would surely try them out before going to any of the Java based ORMs.

If you are to demo Squeryl (e.g., to a Java programmer), do you have a favourite example?

Here's a one liner that says a lot :

val avgHeight: Option[Float] = 
  from(people)(p => compute(avg(p.heightInCentimeters)))

Apart from the shortness of the code, we can see a few implicit conversions at work. The compiler "knows" that the sum query can translate into a 32 bit floating point value, but it also "knows" that it is an Option[], because the avg aggregate function is not guaranteed to return something (the table can be empty). In fact it won't compile if you try to refer to it as a (non Option[]) Float.

Where has Squeryl turned up? Who uses it?

I haven't made any survey, it's on my todo list, but I've exchanged emails with developers that are building systems with Squeryl in fields ranging from finance to bioinformatics.

I read something about Lift...?

Ross Mellgren from the Lift team has written an integration module that is part of Lift 2.1 (release candidate).

What's on the roadmap?

High on my priority list is free text search (backed by Lucene). Longer term I'd like to add things like support for sharding and extending the DSL to exploit the geospatial capabilities of databases like Postgres, Oracle and H2.

Is it of any importance that Squeryl was written i Scala? Or was this merely a coincidence?

Without Scala there wouldn't be a way to have strongly typed queries on the JVM without having verbosity that reaches a caricatural level. Not only wouldn't there be Squeryl, but there wouldn't be anything like it.

When Java came out I was impressed with all the features it had built in: serialization, RMI, garbage collection, portability. It was in its time a game changing technology. Today I have the same impression of Scala: the level of static validation that it gives you, all this with minimal verbosity. If I could say just one thing to qualify it, I'd have to say: game changing.

So the answer is yes, Scala made Squeryl possible. I expect a lot of interesting Scala DSLs will get written in many domains in the coming years. I have a few other DSLs I'd like to write myself.

Any particular advice for someone beginning with Squeryl?

I would just copy an example from the Squeryl site, and modify it gradually so that it becomes your own schema. And most importantly, don't hesitate to ask questions in the discussion groups. I'm often impressed by the quality of the answers given by the community.

Thanks a lot for the great answers!

Using the Scala REPL to tell the difference between ЕКАТEРИНБУРГ and ЕКАТЕРИНБУРГ

2010-05-06T15:46:00.013+02:00

Sometimes, one runs into UTF-8 strings with characters from different code blocks. This is problematic in cases where the fonts look the same, but the characters are different. The Scala REPL is handy for finding out what Unicode block each character in a string belongs to. Let's use "ЕКАТEРИНБУРГ" and "ЕКАТЕРИНБУРГ" as examples:

scala> "ЕКАТEРИНБУРГ" == "ЕКАТЕРИНБУРГ"
res0: Boolean = false

scala> import java.lang.Character.UnicodeBlock
import java.lang.Character.UnicodeBlock

scala> "ЕКАТEРИНБУРГ".foreach(c => println(c +"\t"+ UnicodeBlock.of(c)))
Е CYRILLIC
К CYRILLIC
А CYRILLIC
Т CYRILLIC
E BASIC_LATIN
Р CYRILLIC
И CYRILLIC
Н CYRILLIC
Б CYRILLIC
У CYRILLIC
Р CYRILLIC
Г CYRILLIC

scala> "ЕКАТЕРИНБУРГ".foreach(c => println(c +"\t"+ UnicodeBlock.of(c)))
Е CYRILLIC
К CYRILLIC
А CYRILLIC
Т CYRILLIC
Е CYRILLIC
Р CYRILLIC
И CYRILLIC
Н CYRILLIC
Б CYRILLIC
У CYRILLIC
Р CYRILLIC
Г CYRILLIC

scala>

The REPL exposed one of the seemingly identical strings to be an unhealthy mix of Latin and Cyrillic characters. Thanks, REPL.

A tiny Scala case class to clean up user input

2010-04-11T00:31:00.053+02:00

We needed some cleaning up of user input entered into a text field. We ended up with a Scala case class that cleans up its constructor string argument a bit, by removing multiple whitespace characters and trimming it. It behaves like this:

scala> Text("    a       a      ") == Text("a a")                                                                
res0: Boolean = true
scala> Text("   a      a       ").text == Text("a a").text                                                      
res1: Boolean = true
scala> Text("  a     a    ").text          
res2: java.lang.String = a a

The code looks like this:

case class Text(private var _text: String) {
 val text = _text.trim.replaceAll(" +", " ")
 _text = text
}

Since the input string, var _text, is private, we can manipulate it a bit, without making it possible for others to tamper with. I'm not sure if this is the obvious way to do it, but it seems to work as intended.

We tried a similar version that did not work:

// Doesn't work
case class BrokenText(private var _text: String) {
  _text = _text.trim.replaceAll(" +", " ")
  val text = _text
}

This version does not work since Text.text will return the original string, not the cleaned up one:

scala> BrokenText("    a      b      ")
res0: BrokenText = BrokenText(a b)
scala> res0.text
res1: String =     a      b    
scala>

Why the second version doesn't work? Beats me. (But I'm sure the answer will turn out to be obvious.)

Update: See the two anonymous comments below: one answering my question above, the other one suggesting a neater way of handling it. Thanks.

Scala: Getting into performance trouble, calling head and tail on an ArrayBuffer

2010-02-04T09:50:00.003+01:00

Update: The performance problem described below will be remedied in the final release of Scala 2.8. See martin's comment.

====================================

Recently, I wrote the following two different versions for doing the same thing (compute frequencies):

// Version 1  --- Don't do this, lousy performance
// Scala 2.8
def freq[T](seq: Seq[T]): Map[T, Int] = { 
 import annotation._
 @tailrec
 def freq(seq: Seq[T], map: Map[T, Int]): Map[T, Int] = {
   seq match {
     case s if s.isEmpty => map
     case s => {
       val elem = s.head
       val n = map.getOrElse(elem, 0) + 1
       freq(s.tail, map + (elem -> n ))
     }
   }
 }
 freq(seq, Map())
}

// Version 2 --- 260 times faster than Version 1 on some input
def freq[T](seq: Seq[T]): Map[T, Int] = {
 val freqs = collection.mutable.HashMap[T, Int]()
 for(elem <- seq) {         
   val n = freqs.getOrElseUpdate(elem, 0)         
   freqs.update(elem, n + 1)     
 }     
 // Return immutable copy of freqs     
 Map() ++ freqs
}

When comparing the two versions, it turned out that for some input, Version 1 was about 260 times slower (after JVM warm-up). The performance difference surfaced when both versions were called with the following different inputs:

val linesList = io.Source.fromPath("testfile.txt").getLines().toList
val linesSeq = io.Source.fromPath("testfile.txt").getLines().toSeq

Version 1 called with linesSeq as input, performes horrlibly compared to when called with linesList. On my own, I couldn't figure out why, but helpful and knowledgeable people at #scala solved my problem in a few seconds. The explanation appears to be that 1) The default implementation of Seq is an ArrayBuffer, and 2) Calling head and tail on an ArrayBuffer is costly. The same operations are cheap on a List. That's why Version 1 above is a performance trap.

A possible way of getting better performance, is to change the inner, two argument, freq method to use List, instead of Seq:

// Version 1.b  --- Somewhat better
// Scala 2.8
def freq[T](seq: Seq[T]): Map[T, Int] = { 
 import annotation._
 @tailrec
 def freq(seq: List[T], map: Map[T, Int]): Map[T, Int] = {
   seq match {
     case s if s.isEmpty => map
     case s => {
       val elem = s.head
       val n = map.getOrElse(elem, 0) + 1
       freq(s.tail, map + (elem -> n ))
     }
   }
 }
 freq(seq.toList, Map())
}

Better yet --- in Scala 2.8 --- is to scrap the entire method, and call groupBy(identity).mapValues(_.length) directly on the Seq...

Counting Strings and Things in Scala (2.8)

2010-02-01T22:26:00.041+01:00

I often need to count the frequencies of strings ("words", typically). Below are a few Scala snippets for counting strings and things. (Don't miss the last one.)

First try

Let's start with a method for counting string frequencies in a list:

// Scala 2.8
def freq(wds: List[String]): Map[String, Int] = { 
  import annotation._
  @tailrec
  def freq(wds: List[String], map: Map[String, Int]): Map[String, Int] = {
    wds match {
      case l if l.isEmpty => map
      case l => {
        val elem = l.head
        val n = map.getOrElse(elem, 0) + 1
        freq( l.tail, map + (elem -> n ) )
      }
    }
  }
  freq(wds, Map())
}

It takes a list of strings, and returns a map (hash table) with a frequency count for each unique string. The one argument freq method contains an embedded two argument freq method. The second method recursively consumes elements of the list, incrementing the frequency count of the second accumulator argument. The two argument method is initialised with an empty map (at the end of the one argument method, freq(wds, Map()).

In each recursion, a new, immutable word frequency map is produced, with the incremented frequency count. The

import annotation._
@tailrec

part tells the compiler to check whether it can optimize the tail recursive call or not. (The Scala compiler can optimize a special case of tail recursion.)

If the recursion makes you dizzy, you can use a mutable HashMap instead:

def freq(wds: List[String]): Map[String, Int] = {
  val freqs = collection.mutable.HashMap[String, Int]()
  for(w <- wds) {     
    val n = freqs.getOrElseUpdate(w, 0)     
    freqs.update(w, n + 1)   
  }   
  // Return immutable copy of freqs   
  Map() ++ freqs 
}

Second try

You'll soon find out that the above code is limited, since it only accepts List input. There is a more general concept, Seq, that will make it possible to call freq with different kinds of sequences (lists, listbuffers, arrays):

// Scala 2.8
def freq(wds: Seq[String]): Map[String, Int] = {  
  import annotation._
  @tailrec
  def freq(wds: Seq[String], map: Map[String, Int]): Map[String, Int] = {
    wds match {
      case l if l.isEmpty => map
      case l => {
        val elem = l.head
        val n = map.getOrElse(elem, 0) + 1
        freq( l.tail, map + (elem -> n ) )
      }
    }
  }
  freq(wds, Map())
}

Third try

One day you find yourself relocated from the word counting department to the character counting department. A string is a sequence, but of Chars, not Strings. The code above will not help you count character frequencies. Here is an attempt at generalising the code further, to make it able to count the frequencies of any thing, T, not just String:

// Scala 2.8
def freq[T](seq: Seq[T]): Map[T, Int] = {  
  import annotation._
  @tailrec
  def freq(seq: Seq[T], map: Map[T, Int]): Map[T, Int] = {
    seq match {
      case s if s.isEmpty => map
      case s => {
        val elem = s.head
        val n = map.getOrElse(elem, 0) + 1
        freq(s.tail, map + (elem -> n ))
      }
    }
  }
  freq(seq, Map())
}

Here's the more general non-tail-recursive version:

def freq[T](seq: Seq[T]): Map[T, Int] = {
  val freqs = collection.mutable.HashMap[T, Int]()
  for(elem <- seq) {     
    val n = freqs.getOrElseUpdate(elem, 0)     
    freqs.update(elem, n + 1)   
    }   
  // Return immutable copy of freqs   
  Map() ++ freqs 
}

Hooray.

Last try (shamelessly lifted from someone at #scala)

But... you can still do better than this. A while ago, someone on the #scala irc channel (unfortunately, I don't remember this persons name) answered a question on how to associate each integer in a sequence with the number of times each integer occurred (or something like that). It turns out that, in Scala 2.8, it is possible to write a frequency counting thing even more compactly:

def freq[T](seq: Seq[T]) = seq.groupBy(x => x).mapValues(_.length)

It's so short, that it's almost not worth defining a method/function for it. You can simply call .groupBy(x => x).mapValues(_.length) directly on your Seq. (Or groupBy(identity).mapValues(_.length), which is the same thing.)

Double hooray.

Benchmarking is tricky, but a small test indicates that the last, most beautiful, version is also the quickest, and that the recursive ones using only immutable maps (in some situations) are quite slow.

Beware! scala.swing.TextField proclaims EditDone when it isn't

2009-11-22T16:33:00.023+01:00

Update: Forget about EditDone. See Update below!

scala.swing.TextField is a basic GUI component that can be used for
letting the user input a line of text. When listening to this component, one can react to an EditDone event:

// Inside some GUI component ...
val textField = new TextField(20)
contents += textField
listenTo(textField)

reactions += {case EditDone(`textField`) =>
  println("Ok, searching DB for input "+ textField.text)
}
//...

Fine. Whenever the user (me) hits the Enter key, the message, "Ok,
searching for DB input ...", simulating a database search, is printed.

However, what happens when some unrelated software product suddenly
pops up a window while the user (me) is still inputting text into the
TextField? I tell you what: The evil, non-sentient contraption prints
the simulated search message --- just as if I had hit Enter.

When the TextField loses focus, it emits an EditDone event. But I'm
not done editing. I've only typed "a". I was about to type
"abecedarian". Now the silly thing will search the database for all
words containing the letter "a". I never told it to do that. This
happened just because some other, unrelated, ill-behaving program
grabbed the focus.

Of course, the focus may also be lost because the user voluntarily
changes windows (for instance, in order to Google for "abecedarian").

As far as I can tell, there is no sane way to tell an EditDone event
produced by the user (me) hitting Enter from an EditDone event
produced because the TextField component lost focus. This cannot be
right.

(A while ago, I asked about this on the Scala-user list. Not one single
answer from one single soul in the entire Universe. It feels lonely.)

(I'm using Scala 2.8.)

Update: Forget about EditDone.

What you should do, is not to listen to the TextField, but to TextField.keys. This way, you'll be able to catch a KeyPressed event, and check if the key pressed was Enter. Simple.

It's a bit tricky to figure out, however, since it's not in the TextField Scala docs (you'll have to find your way to scala.swing.Component). This is how it could look:

import swing._
import event._

//...

// Inside some GUI component ...
val textField = new TextField(20)
contents += textField

listenTo(textField.keys)

import Key._
reactions += {case KeyPressed(`textField`, Enter, _, _) =>
  println("Ok, searching DB for input "+ textField.text)
}
//...

Thanks to Ingo Maier for explaining this.

Source's getLines in Scala 2.8 now strips line end

2009-11-16T15:10:00.018+01:00

In Scala 2.8 (not yet officially released), scala.io.Source has been updated.

When reading lines from a file, you do not longer need to trim the lines, since newlines are removed by default. The code to read lines from a file using Source may now look something like this (where fName is a file name (a string)):

val lines = io.Source.fromPath(fName) getLines()

If you want to specify the input file encoding to be UTF-8, you could try this:

val lines = io.Source.fromPath(fName)("UTF8") getLines()

When you look at the API documentation, you'll find that fromPath takes a Codec as a second implicit parameter. Through some mysterious conversion (or "implicit conversion"), you can call it with a string ("UTF8") instead, as in the example above.

Anyway, no more Source.fromFile(fName).getLines.map(_.stripLineEnd). Someone is improving Scala!

Cracker: "Tired of coding Perl"

2009-08-17T11:05:00.004+02:00

Perl is doomed.

Even Cracker is tired of it. The evidence is found in a song on their latest album, where they sing "I'm tired of coding Perl, tired of V.B.A."

Take a look some 30 seconds into the video. The guy browsing the mod_perl Developer's cookbook is not happy. The "Turn on, tune in, drop out with me" video is here.

Perl is doomed.

Scala case classes don't have auxiliary constructors?

2009-07-29T16:00:00.008+02:00

The lesson of today, is that Scala case classes don't appear to have auxiliary constructors.

In Scala, auxiliary constructors may be added to a class by defining a "this" method:


scala> class AClass(s1: String, s2: String) {
  def this(s: String) = this(s, "default")
}
defined class AClass

scala> new AClass("hey")
res0: AClass = AClass@187b5ff

Look what happens when you try the same trick on a case class:


scala> case class ACaseClass(s1: String, s2: String) {
  def this(s: String) = this(s, "default")
}
defined class ACaseClass

scala> ACaseClass("hey")
:7: error: wrong number of arguments for method apply: (String,String)ACaseClass in object ACaseClass
ACaseClass("hey")
^

The attempt at adding an auxiliary constructor compiles, but results in a runtime error.

Update: Oops, yes the can have auxiliary constructors --- see comment below, by jkriesten, straightening things out!

Update: Paul (see comment below) points to the following discussion on this topic http://www.scala-lang.org/node/976.

Printing the Unicode code points of UTF8 characters (Scala)

2009-06-09T21:53:00.024+02:00

Sometimes it is useful to be able to print the Unicode code point of a UTF8 character. (For instance, when you need to check if you mistakenly use a similar looking character instead of the one you're supposed to use.)

Using Scala's RichString's format method, you can create a string of a zero padded, four digit, hexadecimal Unicode number, for example of the 'ä' character, like this:

scala> "%04X".format('ä'.toInt)
res0: String = 00E4

scala>

Here's a related example, printing a tab separated list of some IPA (phonetic) characters and their Unicode code points in a format suitable for using in Scala/Java strings:

scala> "ɸβfvθðszʃʒʂʐçʝxɣχʁħʕʜ"\
.map(c => "%s\t\\u%04X".format(c, c.toInt))\
.foreach(println)
ɸ \u0278
β \u03B2
f \u0066
v \u0076
θ \u03B8
ð \u00F0
s \u0073
z \u007A
ʃ \u0283
ʒ \u0292
ʂ \u0282
ʐ \u0290
ç \u00E7
ʝ \u029D
x \u0078
ɣ \u0263
χ \u03C7
ʁ \u0281
ħ \u0127
ʕ \u0295
ʜ \u029C

scala>

(The line terminating backslashes in the Scala code are added to indicate the fact that the above is a one-liner that doesn't fit the page. Remove these and the newlines if you want to run the code in the Scala shell.)

Knowing the codepoints can be useful, e.g. when you don't want to or can't input non-ASCII characters into your code:

scala> var v = "\u0278"
v: java.lang.String = ɸ

scala>

In Java, it looks similar, but you have to cast your chars to ints:

String.format("%04X", (int) 'ä'), etc.

The perils of changing the case of UTF8 strings

2009-03-24T09:26:00.033+01:00

Below are a few examples of what happens to some just slightly exotic UTF8 strings when up-cased and then down-cased again. The German ß (Eszett) doesn't have an uppercase variant, and becomes two characters. The Greek Sigma has one uppercase variant, but two different lowercase versions: one word final (ς); one for other positions (σ) (explaining my not-so-very-amusing joke in an earlier post).

In the table below, you'll find two other Greek lowercase characters that don't like to be up-cased, ΰ and ΐ. These two characters ultimately become six (see the length columns).

Last, the Turkish variants of <i>, always trusty when it comes to creating confusion (in a computer). The last but one row is interesting, since the original string is severely damaged. In the last row, the proper locale ("tr") is used, and the same string ends up in a much better condition.

The table was generated using Scala (thus Java) strings. The column EqIgnoreCase reports the result of comparing the original string and the up-cased and then down-cased version of that string using Scala's/Java's equalsIgnoreCase. The two rightmost columns present the length of the string before and after changing the case up and down again.

Orig	UpCase ↑	UpDown ⇅	EqIgnoreCase	OrigLen	NewLen
ß	SS	ss	false	1	2
ςσ	ΣΣ	σς	true	2	2
ΰΐ	Ϋ́Ϊ́	ΰΐ	false	2	6
iİıI	IİII	iiii	true	4	4
iİıI	İİII	iiıı	true	4	4

The lesson? Nothing special. That you can do terrible things to strings. That changing the case of strings may be an irreversible operation. That if you are to normalize some text into either lower or uppercase, you might need to decide what's most suitable for a given language. That it might be a good idea to keep the original strings after normalization. That using the correct locale might help. That I'm not a graphical designer (the table is hideous).

Scala: Reversing a string by up- and then downcasing it

2009-03-08T00:14:00.005+01:00

Did you know that you can reverse a string by merely upcasing it and then downcasing it again? Here's an example:

scala> val s = "ςσσ"
s: java.lang.String = ςσσ

scala> s.toUpperCase.toLowerCase == s.reverse.toString
res0: Boolean = true

scala>

If you don't believe me, just copy and paste the two lines of code above into the Scala interpreter, and see it for yourself.

The Firebird database: Problem handling UTF8 characters

2009-03-05T11:26:00.014+01:00

The 'Latin capital letter I with dot above', İ (Unicode 0130), strikes again! This innocent looking Turkish character seems to be reliable when it comes to breaking software that should be able to handle UTF8. (See also this post for a Java example.)

This time it breaks the Firebird database (in my case, v2.1.1 on a 64-bit Debian system). Downcasing some random characters in a database configured to handle UTF8 works fine:

SELECT LOWER('AӴЁΪΣƓ') FROM RDB$DATABASE

returns the expected string, aӵёϊσɠ.

However, when you throw in the trouble-making İ, everything blows up:

SELECT LOWER('AӴЁΪΣƓİ') FROM RDB$DATABASE
*** IBPP::SQLException ***
Context: Statement::Fetch
Message: isc_dsql_fetch failed.

SQL Message : -104
Invalid token

Engine Code    : 335544849
Engine Message :
Malformed string

Slightly different input, generates a different error message:

SELECT LOWER('İA') FROM RDB$DATABASE
*** IBPP::SQLException ***
Context: Statement::Fetch
Message: isc_dsql_fetch failed.

SQL Message : -802
Arithmetic overflow or division by zero has occurred.

Engine Code    : 335544321
Engine Message :
arithmetic exception, numeric overflow, or string truncation

There is an item on the Firebird user list, but without any answers so far.

Update: As mariuz points out in a comment below, this defect now seems to be fixed in an upcoming version. See this bug tracker item.

Book: Real World Haskell (not much real world so far :)

2009-01-06T13:06:00.020+01:00

I've just started to read Real World Haskell (the paper book). It seems like a nice book (except for a few irritating and confusing typos/mistakes at the start of the book).

However, I've read more than 100 pages so far, and still not a sign of any of the "real world" stuff promised by the title. I still don't know much or anything about IDE:s, how to compile the code, scripting, any practical details on how to structure your code into modules, or anything in that direction. So far, mostly (sometimes rather long-wined) discussions on specific (list) functions. One of the examples, end up in a conclusion that might be paraphrased as "by the way, don't use the function we've discussed the last few pages; in real world settings it doesn't work too well".

In the real world, you run into both needles and haystacks , occasionally, but that doesn't help making sense of

isInAny3 needle haystack = any (isInfixOf needle) haystack

And one more real world example of the kind zip3foobar "quux" and I may start losing interest... or just start screaming.

Well, the upcoming chapters have promising titles, so I guess I just have to keep reading. And I guess you have to start with the basics. Still, over 100 pages, and mostly foobars so far...

The book is available on-line.

Scala for small throw-away scripting tasks

2008-12-18T23:33:00.027+01:00

I've come to use Scala for tiny scripts to be thrown away after doing some small task. Typically this involves processing a few files, comparing some textual data, maybe extracting some fields of tab-separated files, etc. The kind of things that Perl used to be the obvious choice for.

Although lacking Perl's simplified syntax for iterating over all lines in files, Scala works quite nicely for small tasks.

For example, today I had to extract from a file all lines of four or more characters including only upper-case characters, and capitalize the output:

scala.io.Source.fromFile(args(0))
.getLines.map(_.stripLineEnd).filter(_.matches("[A-Z]{4,}"))
.map(_.toLowerCase.capitalize).foreach(println)

Not exactly a thing of beauty, but it only took a minute and it works. And it reminds me a bit of a classic Unix command line pipeline.

A few things on my wish-list to make Scala even better for small scripts:

A nicer way of setting the output character encoding (currently you have to do something like Console.setOut(new java.io.PrintStream(Console.out,true,"UTF8")))
It would be great if Source.getLines could remove the new line character of each line
A better name for RichString.stripLineEnd (for some reason, it is totally impossible for me to remember the name of this method)
Maybe scripting support in the Scala Netbeans plugin? (Currently, I think the plugin wants you to put your code in a class/object)

Scala: Reading a tab separated file into a Map (first attempt)

2008-12-12T12:01:00.013+01:00

Below is my first attempt, in Scala, at reading a tab separated file into a map, where the first and second fields of the input file make up the key-value pairs.

There are probably better ways of doing it, but the following seems to work:

val keyValuePairs = scala.io.Source.fromFile(inputFileName, "UTF8")
       .getLines.map(_.stripLineEnd.split("\t", -1))
       .map(fields => fields(0) -> fields(1)).toList

val map = Map(keyValuePairs : _*)

The keyValuePairs:_* stuff is a way to call a variable length argument, the constructor of (the immutable) Map, with a list (keyValuePairs).

I'm pretty sure that there are neater ways of doing it. Furthermore, the above snippet does not do any sensible error checking or input validation (such as skipping empty line, for instance).

Intelligent Software: Netbeans (or JUnit?) can count to three!

2008-12-11T16:26:00.009+01:00

I just noticed a (very) small detail in Netbeans. I was adding some unit test, when I noticed that Netbeans can count to, at least, three.

When running a JUnit test suite of only one test, you get the message "The test passed". After adding another test, the message is "Both tests passed", then "3 tests passed", etc. (Well, of course, given that the tests pass.)

Now, that's what I call (artificial) intelligence.

Here's an unrelated article on counting to three (and more).

Scala: Beware of inadvertently shadowing variables

2008-12-09T22:01:00.010+01:00

I've just spent 15 minutes looking for a stupid mistake in some Scala code. The problem was that I had shadowed a variable.

In some situations in Scala, you are allowed to shadow variables. In other words, it is sometimes legal to give a new variable the same name as an existing one. This can lead to mistakes. The following legal code illustrates how you can shadow a method input variable:

def theShadow(list :Array[String]) : Seq[String] = {
  // Mistake! Inadvertently
  // shadowing the input parameter:
  val list = List("Asa", "nisi", "masa")
  list
 }

(The above is a very obvious example. When you make this mistake in real code, it will probably be in a less obvious context.)

Scala: XML serializer adds closing elements to empty elements

2008-12-09T12:48:00.017+01:00

When printing Scala XML nodes/elements, closing tags for empty elements are added, even if there weren't any in the input.

For example, if you input <childless/>, the XML processor will add a closing tag like this:

scala> val elem = <childless/>
elem: scala.xml.Elem = <childless></childless>

(The two versions of the XML element are equivalent, but sometimes it is practical to be able to do a simple string comparison of the input and output XML files. The added closing tags may make this harder.)

See this thread.

Scala: Problems using the XML API

2008-12-05T11:49:00.009+01:00

I've encountered some problems using the Scala XML API. The first one had to do with scala.xml.XML.loadFile throwing away comment nodes of the input XML file.

A helpful person on the scala-user list suggested instead using scala.xml.parsing.ConstructingParser.fromFile. This worked nicely, keeping the comment elements of the input file intact. However, when processing larger XML files, this approach did not work well, resulting in out of memory exceptions.

Finally, I got yet a helpful answer on the scala-user list, this time in the form of some code, translating Java XML nodes into the Scala equivalents.

If you get into the same trouble as I did, you may want to take a look at this code snippet posted on the scala-user list by David Pollak. (You might have to change the code a bit to suit your needs, though.)

Yet a problem I've encountered: you might be hit by a performance problem when extracting child nodes of a large Elem using the \\ or \ operators. (The fix seems to be to loop over the child nodes instead.)

Summary: The current Scala XML API may not work flawlessly if you both want to process rather large documents and at the same time keep all the information of the original input XML file... but it works fine if you write your own XML file reader (see link above) and are careful with the use of \\ or \ on large Elems.

Here's an earlier post on Scala XML processing.

Scala: New Netbeans 6.5 plugin

2008-11-19T12:56:00.006+01:00

There is a new version of a Netbeans 6.5 plugin for Scala programming.

The Scala plugin already seems quite useful, and it's getting better and better for each new version.

Check it out here. This is a link to the blog of the author of the plugin.

By the way, Netbeans 6.5 was just released too.

Scala: The Map += method expects a Tuple: += ((k, v))

2008-11-18T12:10:00.015+01:00

In Scala, you use the += method to add a key-value pair to a Map. The key-value pair should be in the form of a Tuple, or a Pair. You can use different syntax for such pairs: ("year", 2008), "year" -> 2008, Tuple2("year", 2008) or Pair("year", 2008):

scala> ("year",2008) == "year" -> 2008
res0: Boolean = true

scala> "year" -> 2008 == Pair("year", 2008)
res1: Boolean = true

scala> Pair("year", 2008) == Tuple2("year", 2008)
res2: Boolean = true

Thus, a few different but equal ways of adding a key-value pair to a Map:


scala> val map = new scala.collection.mutable.HashMap[String,Int]
map: scala.collection.mutable.HashMap[String,Int] = Map()

scala> map += (("year",2008))  //Notice the parentheses
scala> map += ("year" -> 2008)
scala> map += Pair("year",2008)
scala> map += Tuple2("year", 2008)

However, this one fails, because of missing parentheses:

scala> map += ("year",2008)
:6: error: type mismatch;
found   : java.lang.String("year")
required: (String, Int)
map+=("year",2008)
^

You can check out, e.g., this and this thread on the Scala mailing list.

Scala: Converting Java collections into their Scala counterparts

2008-11-10T20:51:00.006+01:00

In the scala.collection.jcl library, you'll find Scala wrappers, adding Scala methods to Java collections. This means that a Java collection (e.g., an ArrayList) will be converted to work as a Scala collection, making it possible to call foreach on a ArrayList, etc:

import scala.collection.jcl.Conversions._

val a = new java.util.ArrayList[String]
a.add("Asa")
a.add("nisi")
a.add("masa")

// foreach now works on a Java List:
a.foreach(println)

Simlarily, you can now call .mkString on a Java list:

// Let's use mkString to print the
// ListArray contents as a Prolog spell/3 fact:

println(a.mkString("spell('", "', '", "')."))

// -> spell('Asa', 'nisi', 'masa').

See this Scala mailing list thread.

Scala: You cannot run a companion object as a stand-alone program

2008-11-10T20:37:00.014+01:00

Update: In Scala 2.8, the below is no longer true. A companion object can now work as the entry point of an application.

===============================

In the Scala programming language, a companion object is an object with the same name as a class in the same source file. (Scala's companion objects can be used similar to Java's static methods.)

An object definition on its own can function as the entry point for running a Scala program. Compiling and running this object works fine:

object heyYouTheRocksteadyCrew{
    def main(args :Array[String]) {
     println("Make a break!")
 }
}

However, if you try to run the same object when it is a companion object to a class with the same name, this will result in an exception:

class heyYouTheRocksteadyCrew{}

object heyYouTheRocksteadyCrew{
 def main(args :Array[String]) {
     println("Make a move!")
 }
}

java.lang.NoSuchMethodException:
heyYouTheRocksteadyCrew.main([Ljava.lang.String;)

The above is true of the current release, 2.7.2.final. (Until this is fixed, these guys will not be too happy about any stupid heyYouTheRocksteadyCrew-exception...!)

There is at least one thread about the above on the Scala mailing list.

New version of FlameRobin, Admin GUI for Firebirdsql

2008-10-27T14:30:00.019+01:00

FlameRobin 0.9.0 is released

There is a new version of FlameRobin, a GUI for creating and managing Firebird databases. This version introduces tabbed browsing. In earlier versions of FlameRobin, you would end up with an unmanageable large number of open windows after a short while. This problem is now largely gone. (However, not everyone found this to be a problem --- see one of the comments below.)

I've mostly used FlameRobin for inspecting existing databases, for some minor editing of (existing) stored procedures, and for querying the database. The overall judgment is that it is a fine and very useful piece of software.

Update: There is no pre-compiled version for 64-bit Linux, so if this is what you need, you're on your own... (we gave up on compiling from source after 15 minutes of library dependency hell). This was on a Ubuntu machine.

Update of the update: Apparently, there was already a pre-compiled 64-bit Ubuntu version available! To find the information about the Ubuntu repository, you should not go to the download page that the announcement of the new version points to, you should go here. Thanks mariuz, for pointing this out (see comment below).

Scala: String vs RichString oddities

2008-09-05T18:31:00.015+02:00

Update: In Scala 2.8, the below is no longer true. String.reverse now returns a String rather than a RichString:


scala> "a".reverse == "a"
res0: Boolean = true

=========================================

In the Scala programming language, there is a class called RichString, that adds features to the underlying Java String. In the current version of Scala (2.7.2.final), this leads to some odd behaviour:

"Im a string" == "Im a string".reverse.reverse

returns false, while

"Im a string" == "Im a string".reverse.reverse.toString

returns true!

Just to make your head spin, the following code does indeed work as expected:

val str :String = "Im a string".reverse.reverse
println(str == "Im a string") // prints "true"

while

val str = "Im a string".reverse.reverse
println(str == "Im a string") // prints "false"

does not.

The explanation is that String.reverse returns a RichString, and that == returns false when comparing a String and a RichString, even though it is the "same" string (as in the example above).

If I understand it correctly, this oddity will be fixed in future releases of Scala.

(And no, Scala's == is not the same as Java's ditto. It means "equal objects" rather than "refers to the same instance of an object".)

Scala mailing list item here.

Case insensitive pattern matching of Unicode strings in Java

2008-09-05T16:17:00.009+02:00

To make case insensitive pattern matching of Unicode strings in Java, you can call Pattern.compile with a second argument, like this:

Pattern p = 
Pattern.compile(patternString, Pattern.CASE_INSENSITIVE | Pattern.UNICODE_CASE);

(This is useful when dealing with non-ASCII/non-Latin1 text, such as Cyrillic. However, it may not work flawlessly for the Turkish Unicode characters.)

Update: I just learned that there is a nicer way of doing this: start the patternString above with "(?iu)":

Pattern p = 
Pattern.compile("(?iu)"+ patternString);

Scala and implicit conversion: Turning a string into pure Weirdness

2008-08-30T00:38:00.042+02:00

In the Scala programming language, you can turn water into wine, or vice versa, using implicit conversion.

Imagine that you have a class called Weird:


class Weird(s :String) {
  def imWeird :String = {
    "I'm "+ s +" and I'm weird!"
  }
}

It consists of merely a string, s, and a method, imWeird, that returns a jolly message containing the very same string. (Thus, the code

val freak = new Weird("a freak")
println(freak.imWeird)

outputs I'm a freak and I'm weird!.)

Now, Scala allows you to create an implicit conversion that adds the method(s) of Weird to any other class. Or rather, turns an object into a Weird whenever one calls Weird's methods (functions) on the given object.

For example, the following implicit conversion

  implicit def string2Weird(s: String) = new Weird(s)

makes it possible to call Weird's method(s) on a String. This code

val happy = "Happy"
println(happy.imWeird)

will now output

  I'm Happpy and I'm weird!

The name of the implicit conversion method, string2Weird, is arbitrary.

Turning the lights on (and off) from your computer

2008-07-18T17:16:00.019+02:00

I just bought a Telldus Technologies TellStick. It is a wireless USB-device that can be used for, e.g., turning on and off the light (if it is equipped with a suitable receiver).

The TellStick software is free, and it works under Linux. It even comes with a Java API, and a NetBeans project with some sample code. However, to get the Java binding to work, you have to install some strange libraries (rxtx).

You can controll any number of recievers using your TellStick.

Next time a program throws an exception, it will be able to turn on (or off) a lamp somewhere in the office...

Saldo 1.0: Large, freely available Swedish morphologic and semantic lexicon

2008-05-21T20:35:00.012+02:00

Språkbanken has published a large, freely available Swedish lexicon, Saldo, "a Swedish basic language resource". The release appears to include some 68,000 uninflected lemma forms as well as more than 740,000 expanded (full) word forms. There is morphologic and semantic information.

This resource should be valuable for part-of-speech tagging, lemmatizers, spell-checking, (semantic) analysis of Swedish text, etc, etc.

The release includes software for interfacing with the lexicon. (Those of you into functional programming might be interested in the fact that the lexicon software is written in Haskell.)

It is released under the LGPL license.

Scala one-liner for upcasing lines of text

2008-05-16T11:26:00.023+02:00

The following is a Scala script that up-cases each line of an UTF8 encoded input file (args(0)) and prints the result to standard output:

import scala.io.Source

Console.setOut(new java.io.PrintStream(Console.out,true,"UTF8"))

Source.fromFile(args(0), "UTF8").getLines.foreach(line => print(line.toUpperCase))

If you're trusting the default character encoding to work for you, you may reduce it to:


import scala.io.Source

Source.fromFile(args(0)).getLines.foreach(line => print(line.toUpperCase))

Another way to do it, is to read the lines into an iterator, using the iterator's .map method to upcase each line:


import scala.io.Source

val lines = Source.fromFile(args(0)).getLines.map(_.toUpperCase)

lines.foreach(print)

A Java programmer may be relieved (or horrified) to learn that Scala does not have any checked exceptions. There are only runtime exceptions, and you don't need to add any try/catch statements if you don't want to.

When you run a Scala script, you can instruct the Scala interpreter to compile the script, and use the compiled version (a jar file) if it's younger than the source-file. This gives better performance (shorter start-up, etc). You use the savecompiled command line argument.

The Scala programming language and XML

2008-05-10T13:29:00.040+02:00

The Scala programming language is a combined scripting and "proper" language, that sits on top of the Java VM. You can either run scripts similar to how you run a Ruby or Perl script, or compile your Scala classes to Java bytecode. You run a Scala application similar to how you run a Java application. You can also run a Scala application using the Java VM (but you have to add the Scala library jar file to your class path). You can mix Java and Scala programs, calling Scala objects from Java, and vice versa.

Scala has a feature that I have never seen in a language suitable for general programming: XML (processing) as a feature of the language. The people behind Scala has added XML to the syntax of the language itself. You do not have to load some library or use some special API for processing XML, since it's already part of the language.

It is not only that XML is valid in Scala code, but XML has its own built-in data types. For instance,

val xml = <vegetable>potato</vegetable>

is a valid Scala statement. In other words, XML-elements written in a Scala program are just not merely strings. The xml object can now be manipulated in various ways, much like a DOM object in Java (but with less hassle than in Java).

You can refer to variables in your XML:

val veg = "potato"
val col = "white"
val xml = <vegetable colour={col}>{veg}</vegetable>

// The value of xml now corresponds to
// <vegetable colour="white">potato</vegetable>

You can also embed function/method calls into XML elements. Imagine that you have a method that returns a sequence of n XML elements, like this (you'll need to import scala.xml.NodeSeq and scala.xml.NodeBuffer):

def genNumElems(n :Int) :NodeSeq = {
val result = new NodeBuffer
for(i <- 1 to n) {
    result &+ (<number value={i.toString}/>)
  }
  result
}

(The odd-looking &+ operator means "add".)

You can now embed a call to genNumElems in an XML element, e.g., like this:

val numList = <number_list>{genNumElems(4)}<number_list/>

Printing numList produces:

<number_list>
  <number value="1"></number>
  <number value="2"></number>
  <number value="3"></number>
  <number value="4"></number>
</number_list>

If you want nicer output, you can use a PrettyPrinter (that you import from scala.xml._):

val pp = new PrettyPrinter(100,2) // width and indentation
println(pp.format(xml))

Reading and writing XML data/files to and from Scala is easy. The following is a one-liner that reads an XML file given as a command line argument (args(0)) and returns a list of all elements named "tr" that are child elements to any elements called "table" of the XML file:

val trNodes = scala.xml.XML.loadFile(args(0)) \\ "table" \ "tr"

You may print the <tr> elements (with an empty line between each element) thus:

trNodes.foreach(tr => println(tr + "\n"))

The built-in XML support in Scala's syntax and basic libraries are not the most important or interesting features of Scala, but they sure seem to be very useful.

(Incidentally, the table and tr elements above are present in Oocalc's (OpenOffice.org) XML format for spreadsheets.)

Update: It appears that is not always good advice to use scala.xml.XML.loadFile to read an XML document. One reason is that comment elements are lost. For more advanced XML processing, one should turn to scala.xml.parsing.ConstructingParser.fromFile.

Update: You may run into trouble when processing larger XML documents using the second approach. See this comment.

Bye bye, Ubuntu, Hello Debian

2008-04-27T08:38:00.024+02:00

Sadly, the new version of Ubuntu, 8.04, didn't accept my laptop (an a few years old Acer TravelMate 290, without any strange hardware). I couldn't find any information on how to resolve the problems I ran into, so I had to ditch Ubuntu, and replace it with Debian 4.0.

This was a pity, since the new Ubuntu looked quite promising. The install is incredible easy and rather quick. Apart from the desktop background image, the new system looks and feels good. They appear to have made good choices when it comes to the pre-installed software. But this doesn't help when Ubuntu fails to shutdown the computer properly.

Installing Debian is not as straightforward, but still not very hard. It took a little longer, mostly because I used the net installer that grabs the software packages from the internet and not from the installation CD-ROM.

However, compared to Ubuntu, it takes some more fixing after the installation to get a system that your are comfortable with. For instance, the Debian people appear to think that you should prefer a web browser called Epiphany to Firefox... They don't even offer you the standard Firefox browser, but their own version, "Iceweasel". (There seems to be a totally silly reason as to why Firefox is not called Firefox.)

Worse, the default fonts did not look good on my laptop, so I had to install new fonts (by running apt-get install msttcorefonts, I think?).

A bit surprisingly, Debian supports playing mp3 files without installing additional libraries.

After a bit of tweaking, Debian feels nice. Still, I would prefer a working version of Ubuntu. It would be interesting to know what went wrong in the relationship between Ubuntu 8.04 and my laptop.

Ubuntu 8.04 revitalised my laptop, but I'm still not happy...

2008-04-25T18:27:00.022+02:00

Ah, I just did a clean install of Ubuntu 8.04 (aka "Gniffly Gnaffly") on my Acer laptop. The net update failed, so I had to burn an install CD. I didn't mind, however, since last time I did an upgrade of the laptop, something strange happened, and it became incredibly slow.

With Ubuntu 8.04 installed, the laptop is back on track. It starts fast, and everything (that I care about) seems to work.

The first things to do after install, are to change the desktop background image (the default depicts an oil-drenched dead bird?), turn off the system sounds including the beep and turn off all visual effects.

Update: No! Ubuntu still doesn't behave well. It turns out, that sometimes when I turn off the laptop, it isn't turned off correctly! Ubuntu goes down, the screen gets black, but still, the laptop is not properly turned off (both the indicator that the computer is on, and the indicator that the hard drive is working keep glowing...). Gah.

Furthermore, I've noticed some instances of ill-boding flickering of the screen.

Maybe it is time to go back to Debian.

Keeping empty fields when splitting tab separated lines in Java

2008-04-22T15:08:00.012+02:00

Frequently, I process text files containing tab separated data. Sometimes these have empty columns, i.e., two or more tabs without any data between them. More often than not, I want to keep the empty fields. However, Java's String.split defaults to removing empty fields.

This is what you do to keep the empty fields:

String[] fields = string.split("\t", -1)

In the following example, the test string tst will be split into zero parts (result1) and four parts (result2) respectively:

String tst = "\t\t\t";
String[] result1 = tst.split("\t");       //result1.length == 0
String[] result2 = tst.split("\t", -1);   //result2.length == 4

result2 will contain four instances of the empty string ("").

The same thing goes when you split a string using a pre-compiled regular expression:

Pattern pattern = Pattern.compile("\t");
String[] result3 = pattern.split(tst);     //result3.length == 0
String[] result4 = pattern.split(tst, -1); //result4.length == 4

By the way, I compared the performance of the two variants above (String's split and a pre-compiled pattern matching a tab). Luckily, the difference in performance was negligible, the compiled pattern winning with a small margin. When the split pattern is more complicated, I would expect bigger performance differences between compiled and uncompiled regular expressions. (Running Sun's java command with and without the server argument made a big difference, however. The default client was significantly slower.)

Book: Dreaming in Code by Scott Rosenberg

2008-04-08T14:38:00.031+02:00

Title: Dreaming in Code - Two dozen programmers, three years, 4,732 bugs, and one quest for transcendent software
Author: Scott Rosenberg
ISBN: 978-1-4000-8247-6
URL: www.dreamingincode.com

This book reports how a well-funded, ambitious software project failed. For several years, the author followed the work of a group of developers employed to produce a revolutionary piece of software. The non-profit organisation set up for the task did not have any customers, but was funded by an idealist with deep pockets. Many of the people involved appear to be quite experienced and knowledgeable. The project is called Chandler, and the open source organisation is called OSAF.

The theme of the book is that producing software is complicated and that it is hard to predict whether a project will succeed or fail, and that there is no obvious solution to this problem. The book is aimed at non-programmers. The first 80 or so pages give a background on software development intertwined with the story of the software project. It takes a while for the book to get going. If you are familiar with programming and different programming concepts, you might find some of these pages less interesting. However, the author manages to explain things such as object oriented programming, open source, scripting languages, the halting problem, etc, for a non-programmer and without simplifying too much.

The author appears to be interested in his subject, and understands it well. He is also a good writer. The author does a good job of explaining and exemplifying how hard software development can be. (Still, I cannot keep from thinking, that with a tenth of the funding of the project he describes, my company could do wonders...) But the text is too long, and the problems of the project he is describing are in an exaggerated way generalised into problems of all software development.

This may have been a hard book to write, since I suspect that at the outset, the author figured that he would be describing what should turn out to be a successful project. Instead he had to describe and explain a failure. This is probably why the book to a large extent discusses different software failures. The author has made quite a lot of research, and describes different methodologies for software development meant to reduce the risk and to ensure the quality of software development.

On the whole the book is interesting, but rather pessimistic concerning the state of software development. While perhaps not an inspiring text, you should be able to pick up a few things to stay clear of, though.

With the help of some more editing, maybe the book could have been a little shorter (and better).

A side note: The book discusses the problem of producing reusable code. This is perhaps not the most central theme of the book, but every time I hear about the failure of software producers to create reusable code, I cannot help but to reflect that almost every day, I reuse code. If you program in Java, for instance, there is a huge set of reusable libraries for almost everything: XML processing, GUI building, cryptography, email, sound, sorting, hash tables, databases... the list of components you do not have to implement yourself, but can use as building blocks for new applications, goes on and on. You can even find software for automatically producing code (some machine learning approaches, for instance). There are programming languages and environments that sit on top of other such software: programming languages reusing other programming languages...! (See for example the Scala or JRuby languages.)
From this perspective, there is a breathtaking amount of (freely) available, high quality software about.

Automatic Simple backups: SBackup

2008-04-05T13:29:00.008+02:00

A colleague told me about a simple backup utility called... Simple Backup (or SBackup). With the help of Simple Backup, you can very easily do hard-drive backups (and restore the backups if needed). If you're using Ubuntu, you will find it with the help of apt-get, Synaptic or under Applications>Add/remove....

At a small office, SBackup may be suitable for doing daily, automatic PC-backups. You configure it to do incremental backups with a frequency of your own choice. You can tell SBackup to put the backups on a remote server through ssh. Notice that the connection settings are in clear text, i.e., your password for ssh-ing will be readable for anyone with access to your computer! (Thus, you should be a bit careful with how you use SBackup.)

A nice feature is that the backups are in tar.gz format, i.e., you can use standard tools to read the backed up files. You can also tell SBackup what local directories to include or exclude from the backups. Once configured, the only thing you may need to care about is that you have enough disk space on the machine that holds the backups. If you do incremental backups of a number of computers, the backups may grow quite large. (However, they will not grow infinitely large, since SBackup can take care of removing old/redundant backups.)

More info here.

Update: Due to a reboot of the target server, SBackup silently stopped doing its backups. It had to do with obsolete ssh keys, most likely. On the Ubuntu clients, the problem can fixed by removing the known_hosts file from the root home directory

sudo rm /root/.ssh/known_hosts

or by removing the same file from the user home

rm ${HOME}/.ssh/known_hosts

(It appears to work differently on different versions of Ubuntu.)

After this, we started the SBackup configuration GUI and tested the destination (you need to answer a question before it works again).

The same goes for moving the backup destination to a different server. SBackup silently stops working. You have to delete the known_hosts file, as above.

No Web Start for 64-bit Sun Java

2008-03-28T11:16:00.011+01:00

Sun does not include Java Web Start in its 64-bit version of Java. It appears that Sun thinks that you are not supposed to run Web Start on 64-bit machines, since these mostly are servers (?), and... eh... sorry, I cannot follow their reasoning. Let's hope they change their minds.

I haven't tried it myself, but here is a description of how to run 32-bit Java Web Start on 64-bit Ubuntu.

Update: At the time of writing this, an AMD64 version of Java Web Start is at the top of Sun's Request for Enhancements list.

Update: There will be support for 64-bit Java Web Start in an upcoming release, 1.6.0_12 (I think). Ismael Juma points out that an early access release is available. See his comment below.

Frequency list bash function

2008-03-26T10:40:00.001+01:00

In addition to command aliases (see an earlier post), you can add your own functions to the bash shell. Here is a simple but useful command line sequence:

function freq() {
    sort $* | uniq -c | sort -rn;
}

Put it in ~/.bashrc and you will have a freq command for creating frequency lists:

freq <FILES>

will sort and count all identical lines of the input file(s), and present them in descending frequency. Useful in many situations, not the least for checking that files that are supposed to only contain unique lines actually do so.

(I'm not too sure about bash function syntax, but the function above seems to do its work.)

If you're not familiar with the different commands of the pipeline above, there is plenty to read (e.g., egrep for linguists).

Favourite bash command line aliases

2008-03-25T14:04:00.006+01:00

My favourite bash aliases currently are

alias hist='history|egrep'

and

alias ös='ls'

The second one for the reason that 'ö' sits next to 'l' on my Swedish keyboard, and when I intended to type 'ls' I type 'ös' more often than not. The one I use the most, however, is alias more='m' (I also have the classic more='mroe' and more='moer' to catch some frequent typos).

The first one, hist, makes it possible to use regular expressions to search the history of earlier shell commands. This is useful when you cannot remember some tricky command line sequence, or are too lazy to type some long command that you know you issued the other day.

For instance

hist 'java|ruby'

will print any previous command (in bash's history) containing any of the two strings.

(Well, I think you can accomplish the same thing using the original history command, but to paraphrase Morrissey, now my head is full, and my brain doesn't have room for more cryptic command line arguments.)

You can put your bash aliases in ~/.bashrc.

(Thanks to Chris for spotting a (now corrected) mistake in the first example. See the comment below.)

Update: Hey, checkout the comment by Anonymous below: Ctrl-r seems useful for searching the Bash history!

Beware of Sun's Java equalsIgnoreCase --- Turkish example

2008-03-20T15:27:00.007+01:00

There appears to be a mistake in the implementation of String.equalsIgnoreCase in Sun's Java.

Look what a colleague sent me (and see an earlier post on Turkish characters below):

import java.io.PrintStream;
import java.util.Locale;

public class TestTur
{
 public static final void main(final String[] args) throws Exception
 {
  Locale.setDefault(new Locale("tr"));
  System.setOut(new PrintStream(System.out,true,"UTF8"));

  String s1 = "I";
  String s2 = "ı";
  String s3 = "i";

  System.out.println(s1+"=="+s2+"? "+s1.equalsIgnoreCase(s2));
  System.out.println(s1+"=="+s2+"? "+s1.toLowerCase().equals(s2.toLowerCase()));
  System.out.println();

  System.out.println(s1+"=="+s3+"? "+s1.equalsIgnoreCase(s3));
  System.out.println(s1+"=="+s3+"? "+s1.toLowerCase().equals(s3.toLowerCase()));
 }
}

Now, what do you think the above code prints? You would expect that

string1.equalsIgnoreCase(string2)

is exactly the same as

string1.toLowerCase().equals(string2.toLowerCase())

wouldn't you...?

Surprise, surprise. This is what the above code prints:

I==ı? true
I==ı? true

I==i? true
I==i? false

I bet Mustafa Kemal Atatürk didn't see that one coming!

The above peculiarity did actually lead to some problems for us, so this is a practical problem rather than an academic one.

Part of the problem when dealing with Turkish text (except for the mistake in how Java's equalsIgnoreCase works), is that "Latin" 'i' and Turkish 'i' as well as "Latin" 'I' and Turkish 'I' share the same Unicode codepoints. Maybe they should have been different characters. A little late for that now.

Firebird vs Postgresql

2008-03-18T12:14:00.017+01:00

We have similar databases running on MySql, Postgresql and Firebird. One of the reasons for moving away from MySql was the fact that the UTF8 support didn't work properly. I cannot remember the details, but it had to do with non-Latin-1 data, such as text in Czech or Russian. In some situations MySql refused to correctly identify equal UTF8 strings. You put in some word that you cannot retrieve again, bleh!

Furthermore, we've never understood how the user permissions are supposed to work in MySql (we always end up frantically running all possible variants of the GRANT ALL command).

We moved to Postgresql, which worked a lot better. Now we've started using Firebird, that also seems like a very nice piece of software.

Here is list of a few things I've noticed when moving from Postgresql to Firebird:

* Firebird lacks built-in support for regular expressions. (We make heavy use of complex string searches of natural language data. If we hadn't got help from an expert, who helped us compile some user defined functions, UDF:s, for this purpose, this would have been a show-stopper.)

* Postgres' psql command line tool is better than Firebird's isql(-fb). (If you are a Windows user, see Carlos' comment below)

* Firebird database files grow and grow. This is true even if you delete data. You have to manually back-up and restore a database to reclaim disk space. Maybe this is not a great problem in normal usage, but I noticed that the databases I use for running test suits against keep growing, though the test database itself is quite small (and the data are cleared out between test runs). [Update: Please notice that long-time users of Firebird insist that this is not a problem. See Carlos', Sergio Marcelo's and also Michal's comments below.]

* I've never had any luck installing Firebird from a Debian package. I have had to do a manual install to get it to work

* Firebird has a useful GUI, FlameRobin, that let's you inspect and change your databases. FlameRobin comes with an editor useful for writing/editing stored procedures. The editor has code completion, that helps you with suggestions of table and column names and the like as you type.

* Firebird has a nice way to manage database files: all tables of a database end up in a single file, that you can name whatever you like, and put wherever you like.

* It appears to be easier to find useful documentation for Postgres than for Firebird (but Firebird does have a nice FAQ site)

Answer to Darius Damalakas comment below: I'm not the right person to comment on the performance of the different DBMSs. However, we haven't noticed any significant difference in performance between MySql, Postgresql and Firebird. Currently, the bottlenecks in our software are to be found outside of the databases, so the performance of the individual DBMSs has not been a big concern. They're all fast enough.

Firebird does seem to be a snappy system, and I would be surprised to find it to perform less good than Postgres.

So far, the only difference in features that has mattered to us, is the lack of built-in support for regular expressions in Firebird (see above). In all other respects (of importance to us), the functionality of Postgres and Firebird seems equivalent.

Update: Support for regular expressions is scheduled for the upcoming 2.5.0 release of Firebird.

Update: In response to an anonymous (and rather critical) comment, mariuz has added some useful links in a comment below.

Update: In a comment below, Michal has posted some information on DatabaseGrowthIncrement, taken from the release notes of Firebird 2.1.

Beware of Firebird 2.0 Debian package

2008-03-15T15:48:00.002+01:00

We are migrating an application to the Firebird 2.0 database manager (firebirdsql.org). Our server runs Debian (AMD64), and we used the Firebird 2.0 (superserver) Debian package as suggested in the Firebird site's FAQ section. However, when the package was installed, it appears to have silently overlooked a dependency, missing a library necessary for getting the "user defined functions", UDF:s, to work correctly. (Firebird didn't find the UDF:s, resulting in runtime errors when calls to the functions were issued from a Firebird database.)

We made sure that Firebird as well as the UDF:s were all compiled for AMD64.

When uninstalling the apt-get Firebird package, and manually installing Firebird 2.0.3 from the standard .tar.gz file, the missing dependency was spotted, and the database could be properly installed. Unfortunately, I didn't keep a record, but it might have been the correct version of libstdc++ that was missing.

As far as I can remember, this is the only time a Debian apt-get package has failed me. In addition to the fact the apt-get install of Firebird might be broken, you have to be careful not to apt-get install "Firebird 2", since this will give you Firebird 1.5! Peculiar. (But see the comment from mariuz below).

I had a similar experience the first time I tried to install Firebird from a Debian package. This was Firebird 1.5 (the Firebird 2 Debian package), before Firebird 2.0 was released. I never got that one to run either, but had to install the tar.gz version obtained from the official Firebird webserver. I can't remember exactly what went wrong at that time, but it was impossible to get the Debian package that we tried at that time to work. The manual install worked perfectly, just as it did this time.

Update: The Debian Firebird2 package (containing Firebird 1.5) appears to be discontinued.

Don't concatenate Java strings using +=

2008-03-15T15:04:00.003+01:00

The other day, I ran into a Java performance problem. It was an extremely simple Scanner loop, reading a file of some 20,000 lines of text, concatenating the lines into one single string:

Scanner sc = new Scanner(new File(fName), "UTF8");
String result = "";
while(sc.hasNextLine())
{
  result += sc.nextLine(); //Avoid this!
}

// Do something with result

The above loop took incredible long time to finish, and I had no clue of what could possibly be wrong. A colleague glanced at the code and said "StringBuilder". I had forgotten about the poor performance of string concatenation using += (or +). I must have thought that this was a problem of the past.

Removing the += part for a StringBuilder resulted in excellent performance:

Scanner sc = new Scanner(new File(fName), "UTF8");
StringBuilder result = new StringBuilder();
while (sc.hasNextLine()) 
{
  result.append(sc.nextLine());
}

// Do something with result.toString

Update: ttaveira points out that you may gain some additional speed by initializing the StringBuilder to a suitable capacity. See the comment below.

Reading/writing non-default character encoded data in Java

2008-02-15T11:42:00.001+01:00

When in an environment where the default (system) character encoding differs from the desired character encoding of the output data, you can use System.setOut and System.setErr. For reading data of a different character encoding than the default encoding, you can tell e.g. the Scanner class what character encoding to expect.

The following could be used for reading and writing UTF8 data on a system where the default character encoding may be different from UTF8:


System.setOut(new PrintStream(System.out,true,"UTF8"));
System.setErr(new PrintStream(System.err,true,"UTF8"));

Scanner scanner = new Scanner(new File(fileName), "UTF8");

while(scanner.hasNextLine())
   {
    // Read input lines,
    String line = scanner.nextLine();
    line = doSomething(line);
    // Write some output to STDOUT/STDERR
    System.out.println(line);
    ...
   }

The boolean flag of the second constructor argument of PrintStream activates autoflush, but one does not need to use this argument.

Correct case with Java's Locale

2007-11-25T21:07:00.003+01:00

In Turkish, the uppercase version of 'i' is 'İ' (not 'I'). The problem is that the Turkish and the "ordinary" Latin 'i' is the same character (the same Unicode code point). If you upcase the 'i' in a Turkish context using the default settings, you might get the wrong letter.

In Java, you can use the Locale class to get this right:


Locale tr = new Locale("tr"); //Turkish
String trI = "i".toUpperCase(tr);
System.out.println(trI);

The above code outputs

İ

(and not 'I').

Be aware that comparing Turkish strings may not work flawlessly. See also this post.

You should also notice that changing the Locale does other things as well. For instance you might end up getting error messages in Turkish...

Sun's javac and source file character encoding

2007-11-23T10:53:00.001+01:00

Sometimes you may need to tell the Java compiler about the character encoding of the source files. For example, if an ISO-8859-1 encoded source file is compiled in a UTF-8 environment, any funny (non-ASCII) characters may become a problem.

This is an example of how you can tell Sun's javac about the source file encoding:


javac -encoding iso8859-1 <FILE PATH(s)>

(Without the encoding switch, javac uses your system's default encoding.)

This way, you can tell Sun's JVM to expect UTF-8 IO:

java -Dfile.encoding=utf8

Identifying Unicode code blocks in Java

2007-11-22T10:39:00.000+01:00

With the help of Java's Character class, one can identify to what code block a unicode character belongs. This may be useful when, for example, validating a string in order to find peculiar mixtures of character code blocks (see an example in a previous post).

The following code

Character.UnicodeBlock ub = null;

ub = Character.UnicodeBlock.of('\u042F');
System.out.println(ub);

ub = Character.UnicodeBlock.of('۲');
System.out.println(ub);

outputs

CYRILLIC
ARABIC

This is a method returning all code blocks for the characters of a string:


Set<UnicodeBlock> getUnicodeCodeBlocks(final String s)
{
      Set<UnicodeBlock> result = new HashSet<UnicodeBlock>();
      for(char c : s.toCharArray())
      {
        result.add(Character.UnicodeBlock.of(c));
      }
      return result;
}

The ancient art of bashing the Danish language

2007-11-20T17:26:00.000+01:00

This is what a Swede (Dr Hemming Gadh) is said to have said about the Danish language in the year 1510 (translation below):

‘Der till medh: så wærdas de icke heller att talla som annat folck, utan tryckia ordhen fram lika som the willia hosta, och synas endeles medh flitt forwendhe ordhen i strupan, for æn de komma fram, sammaledes wanskapa the munnen, då the talla, wridhan och wrengan, så att the draga then offwra leppen till then wenstra sidon och den nedra till then högra sidon, menandes dett wara sig en besynnerlighe prydning och wellståndh.’

The Swedish of the section above is quite weird in itself. The following English translation is lifted from Syllabic and morphological structure: what can be learnt from their interaction in Danish? (Hans Basbøll, 2006):

‘Also this: nor do they [the Danes] stoop (‘worthy themselves’) to speak like other people, but press the words forward as if they will cough, and appear partly to deliberately turn the words around in the throat, before they come forward (i.e. out of the mouth), partly they misshape the mouth when they speak, twist it and sneer it, so that they pull the upper lip to the left side and the lower to the right side, thinking this to be a particular ornament and well-standing.’

The first section of the paper referred to above is "Can spoken Danish be understood?". The paper cites studies pointing to the fact "[...] that Danish is found (more or less) difficult by everybody" (‘everybody’ meaning other Scandinavians).

The old Danish-bashing quotation is also found in The Phonology of Danish, Hans Basbøll, Oxford University Press, 2005, p. 83.

The byte-order mark

2007-11-20T12:37:00.001+01:00

Things you didn't know that you had to know about #42:

The BOM, or the byte-order mark.

The BOM is a hateful creature, that sits at the beginning of some of your files. It's there only to mess things up.

You cannot see the BOM, but when you do, it hurts your eye: ï»¿.

The most rational action when running into the BOM in a UTF-8 file, is to blame the nearest Windows user, and then delete it (the BOM, not the user).

Don't ask me why the BOM made me remember this old tune:

The file and iconv commands

2007-11-07T10:50:00.001+01:00

file and iconv are two simple but useful commands handy when dealing with files of different character encondings.

file <file path>

The file command makes a guess of what kind of file a file is. If you are lucky it may, for instance, help you to find out that a text file is encoded using UTF16, or ISO-8859-1, etc.

iconv -f <current encoding> -t <target encoding> <file path>

The iconv command is useful for converting between different character encodings. For instance, if you have noticed (with the help of the file command above) that a Unicode file is in UTF16, but you want in to be UTF8, you may use the iconv command:

iconv -f utf16 -t utf8 <file path>

(There are other, similar, commands, but for some reason iconv appears to be the only one that I can remember the name of.)

The unicode command

2007-11-06T23:55:00.000+01:00

I was just told about the unicode command.

On a Debian based system, try sudo apt-get install unicode.

Running the unicode command on the different characters 'y' and 'у' (see earlier post) looks like this:


nikolaj@fon:~$ unicode yу
U+0079 LATIN SMALL LETTER Y
UTF-8: 79  UTF-16BE: 0079  Decimal: y
y (Y)
Uppercase: U+0059
Category: Ll (Letter, Lowercase)
Bidi: L (Left-to-Right)

U+0443 CYRILLIC SMALL LETTER U
UTF-8: d1 83  UTF-16BE: 0443  Decimal: у
у (У)
Uppercase: U+0423
Category: Ll (Letter, Lowercase)
Bidi: L (Left-to-Right)

Unicode mystery: Identical typeface, different characters

2007-11-06T22:36:00.006+01:00

Do the following two strings look similar to you?

уeЕoОxХaАM
yеEоOхXаAМ

Sorry, but they aren't. Not one of the corresponding characters of those strings are equal, but for the eye. For example, 'y' and 'у' have different Unicode codes (0079 and 0443).

The fact is that both strings contain a mix of Latin and Cyrillic characters, but not the same mix. There are a number of Cyrillic and Latin characters that look the same, but aren't. Comparing these strings using, for example, Java's String.equals method will return false. Sorting strings of mixed character encodings (but with identically looking type faces) will produce odd results. Latin P is quite different from the Russian Р, etc.

Giving it a bit of thought, it is not strange at all, but the first time you run into the problem, it can be quite tough to figure it out. It's like the first time you accidentally activated the "Insert" key on you keyboard, and thought that your computer was broken. (By the way: "Insert", "Scroll Lock", "Pause/Break"... What on earth are these keys doing on my keyboard...?!)

If you are dealing with (language) data of different character encodings, it's wise to validate your strings, to ensure that, e.g., a Russian string contains only Cyrillic characters, and that the Western European ones contain only Latinos (see this related post).

And don't miss the 'A' example in the comment below!

Ubuntu 7.10 sluggified my laptop

2007-11-05T22:04:00.000+01:00

Upgrading my Acer laptop from Ubuntu 7.04 (aka "Bibbly Bobbly") to 7.10 (aka "Sniggly Snoggly") significantly downgraded its GUI performance. It now takes twice as long to boot the laptop. It takes five times as long to bring up the desktop after log-in. Starting Firefox takes a millenium. I didn't even get the promised 3D Desktop jingle-jangle. All I got were some lousy desktop icons.

Furthermore, something happened to the networking, and it now takes quite a while for the poor thing to find the Internets. All together, this means that I have to get up no less than 2.5 minutes earlier each day, to meet my busy schedule.

Unsnappiness 2.0, and I want my money back!

However, when I nikoloogle for people with a similar experience, all I find is people going "Wooah, I got better performance with Ubuntu 7.10!". One wonders, I'm I alone in this?

Well, maybe not everyone praised its superior performance:

2007-11-05T17:37:00.000+01:00

Welcome to the least visited page of the Internets!