Today I needed to scrape some data from a website and tried to use the trusted old scrAPI to do the job. Grrrr, its not working. Throwing an error:

Scraper::Reader::HTMLParseError: Scraper::Reader::HTMLParseError: Unable to load /Library/Ruby/Gems/1.8/gems/scrapi-1.2.0/lib/scraper/../tidy/libtidy.dylib

After some time on google I didn’t find any fixes for the issue. So I decided to build from source…

I grabbed the assaf’s github repository.

• git clone git://github.com/assaf/scrapi.git

Then tried the tests by running

rake test

117 tests, 346 assertions, 0 failures, 44 errors

Nope errors all over the show… Looking at the original exception message, I checked if the libtidy.dylib exist on the lib/tidy directory. Nope, not there….

So where do I get this library file….

MacPorts to the rescue… Install tidy from MacPorts using the following command:

• sudo port install tidy

Now we need to find where MacPorts installed the files using the following port command:

• port contents tidy

The result:

Port tidy contains:
/opt/local/bin/tab2space
/opt/local/bin/tidy
/opt/local/include/buffio.h
/opt/local/include/fileio.h
/opt/local/include/platform.h
/opt/local/include/tidy.h
/opt/local/include/tidyenum.h
/opt/local/lib/libtidy-0.99.0.dylib
/opt/local/lib/libtidy.0.dylib
/opt/local/lib/libtidy.a
/opt/local/lib/libtidy.dylib
/opt/local/lib/libtidy.la

Now all we need to do is copy the library file to our scrAPI source directory:

• cp /opt/local/lib/libtidy.dylib [your source location]/lib/tidy/libtidy.dylib

Ok, before we speed ahead. Lets just run those test to check if all is fine:

117 tests, 474 assertions, 0 failures, 0 errors

Awesome, we are almost there. Next we need to build the gem using the rake:

• rake package

Make sure you get a ‘Successfully built RubyGem’ message. Now we are ready to install the newly build gem and test scrAPI again.

• sudo gem install pkg/scrapi-1.2.1.gem

And there you go, scrAPI working again.

Inspired by Ben Maybey and his bash history summary, here is mine:

  → 'history 1000 | awk '{a[$2]++}END{for(i in a){print a[i] " " i}}' | sort -rn | head
  242 cd
  117 clear
  96 rake
  70 sudo
  57 ss
  52 AUTOFEATURE=true
  37 sc
  32 sg
  32 mate
  23 tail

When you have a query that you need to run as raw sql against the database, like batch data processing or maybe some data cleanup, and you don`t want to create ActiveRecord models to handle the data. Because creating models is expensive and slow. So we want to execute directly against the database. But how do we execute directly on the database from within our rails application environment? The solution is to use ActiveRecord::Base.connection to do the work:

sql = "my complex sql statement"
	ActiveRecord::Base.connection.execute(sql)

This is really nice way to execute queries within rake tasks. In the following example I show how you would establish a database connection as well:

namespace :db do
	
	  desc "Cleanup the database by setting rows to deleted when older than xxx. Defaults to development database.  Set RAILS_ENV=[production, test, etc.] to override."
	  task :cleanup => :environment do
	    sql = <<-SQL
	      -- do some cleanup code
	    SQL
	    # used to connect active record to the database
	    ActiveRecord::Base.establish_connection
	    ActiveRecord::Base.connection.execute(sql)
	  end

	end
	

Here is a very funny tutorial on rake. That`s all for now folks…

Technorati Tags: Rails, Rake, SQL

Post created with TextMate.

Create strong passwords on the command line by typing

openssl rand -base64 6

and the output:

=> 050L+4Kx

Run the command again:

=> /sUIIy8s

Cool, now you should be able to create strong passwords quickly.

I’ve first posted this tip on the 14 of May 2005 and I still get good traffic to the post. So I decided to update the post and code samples to use Framework 2.0 (I’m still not using 3.5) and improve the code somewhat.

The problem:

How do I databind my combo box to my domain object that has a property of some enum type?

The solution:

Quite simple, firstly we will create an object (EnumComboBoxItem) that would encapsulate and represent each item in the combo list data source. EnumComboBoxItem is a pure fabrication class that has two properties we’ll use when databinding to the combo. We have a Value property which is used by the ValueMember property on the combo box and EscapedValue that is used by the DisplayMember property of the combo box. Note that when we construct a new EnumComboBoxItem instance that we call the AsciiEscape method in the constructor when populating the escapedValue field.

Here is the code:

The original value.

32 public EnumComboBoxItem(object originalValue)

33 {

34 value = originalValue;

35 escapedValue = EscapeAsciiValues(originalValue.ToString());

36 }

37

38 #endregion

39

40 #region Instance Properties

41

42 ///

43 /// Gets the escaped value.

44 ///

45 /// The escaped value.

46 public string EscapedValue

47 {

48 get { return escapedValue; }

49 }

50

51 ///

52 /// Gets the value.

53 ///

54 /// The value.

55 public object Value

56 {

57 get { return value; }

58 }

59

60 #endregion

61

62 #region Instance Methods

63

64 ///

65 /// Returns a that represents the current .

66 ///

67 ///

68 /// A that represents the current .

69 ///

70 public override string ToString()

71 {

72 return escapedValue;

73 }

74

75 #endregion

76

77 #region Class Methods

78

79 ///

80 /// Creates the data source for the enum type.

81 ///

82 ///

Type of the enum.

83 /// When the type passed is not an enum the exception is raised.

84 ///

85 /// Returns a generic list to be used as the combo datasource.

86 ///

87 public static List<object> CreateDataSourceFor(Type enumType)

88 {

89 if (!enumType.IsEnum)

90 throw new ApplicationException(“When using the Enum combo data source you are only allowed to use enum types.”);

91

92 Array enumValues = Enum.GetValues(enumType);

93

94 List<object> result = new List<object>();

95

96 for (int index = 0; index < enumValues.Length; index++)

97 {

98 result.Add(new EnumComboBoxItem(enumValues.GetValue(index)));

99 }

100

101 return result;

102 }

103

104 private static string ConvertAsciiValueToStringRepresentation(string currentRegexMatchValue)

105 {

106 string result = string.Empty;

107 const int BaseValue = 16;

108

109 if (currentRegexMatchValue.StartsWith(“0x”))

110 {

111 // grab the remainder string after the 0x

112 string asciiCodeValue = currentRegexMatchValue.Substring(2);

113 // convert the remainder string to a byte

114 byte byteValue = Convert.ToByte(asciiCodeValue, BaseValue);

115 // now we can convert to a character and find the string of the character

116 result = Convert.ToChar(byteValue).ToString();

117 }

118 return result;

119 }

120

121 private static string EscapeAsciiValues(string rawValue)

122 {

123 string result = string.Empty;

124

125 if (rawValue == null)

126 return result;

127

128 // Find 0x with any four alpha-numeric characters

129 Regex regex = new Regex(“0x[0-9A-Z]{4}”);

130

131 MatchCollection matchCollection = regex.Matches(rawValue);

132 Match matchPrevious = null;

133

134 for (int i = 0; i < matchCollection.Count; i++)

135 {

136 // if no match (ie. null) set to 0, else the current index + length

137 int startIndex = (matchPrevious == null ? 0 : matchPrevious.Index + matchPrevious.Length);

138

139 // add the part of the string that is not the ascii expression

140 result += rawValue.Substring(startIndex, matchCollection[i].Index – startIndex);

141

142 // convert the ascii expression to a normal string

143 result += ConvertAsciiValueToStringRepresentation(matchCollection[i].Value);

144

145 matchPrevious = matchCollection[i];

146 }

147

148 if (matchPrevious == null)

149 result += rawValue;

150 else

151 result += rawValue.Substring(matchPrevious.Index + matchPrevious.Length);

152

153 return result;

154 }

155

156 #endregion

157 }

158 }

The EscapeAsciiValues method is used to Replace the Ascii characters in our Enum with the character representation. We do the Ascii characters because we want our combo to display “Bi- Annual” instead of “BiAnnual”. Look at the following enum as an example:

Now, all that we need is the following code behind the form to bind our Combo box and we are away:

The EnumComboBoxItem.CreateDataSourceFor method creates an List<object> for us to use as a data source for the combo box.

And there you have it, a Combobox bound to an Enum!

I’m contracting at a client running a Novell network and wanted to create some small little ruby scripts to automate some tasks for me. But I ran into problems using gem install behind their firewall. I’m getting the following error:

ERROR: While executing gem … (Gem::RemoteSourceException)
HTTP Response 407

Here is the steps I followed to get gem install working on my windows xp machine in painstaking detail:

Firstly we need to set the HTTP_PROXY environment variable. I’ve played around with different options, using uid and pwd and other suggestions, but all I needed was the following:

1. On the desktop right click the ‘My Computer’ icon and select properties.
2. Click on the advanced tab and then the ‘Environment Variables’ button.
   
3. Now you can either add the HTTP_PROXY variable to the ‘User’ or the ‘System’ variables by clicking the ‘New’ button.
   
4. Within the new system variable dialog, specify ‘HTTP_PROXY’ as the variable name and in the value area a url in the following format – http://[proxy_ip]:[proxy_port] – i.e. http://127.168.1.3:5865
   

Next we install the Ruby Win32 SSPI which gives ruby the ability to authenticate with ISA natively. We need this library because as far as I understand, it patches the Open-uri library because of some incompatibilities with windows. Anyways, follow the following steps to get it working:

1. Download rubysspi from the Ruby Win32 SSPI project page(also available as gem install rubysspi but that doesn’t help much, does it?)
2. Install the gem locally using the following command: gem install [local_path_to_gem]\rubysspi-1.2.3.gem i.e. C:\gems\rubysspi-1.2.3.gem
3. Now copy the spa.rb file from the gem install directory and paste it in the site-ruby directory. i.e. If ruby is installed in C:\ruby, then the paths should be:
   • origin path – C:\ruby\lib\ruby\gems\1.8\gems\rubysspi-1.2.3\spa.rb
   • destination path – C:\ruby\lib\ruby\site_ruby\spa.rb
4. Find gem.bat in your ruby bin directory (C:\ruby\bin) and replace the last line (i.e. “%~d0%~p0ruby” -x “%~f0″ %*) with the following line – @”ruby” -rspa “c:\ruby\bin\gem” %1 %2 %3 %4 %5 %6 %7 %8 %9
5. Test you can access the remote ruby gems by executing the following command: gem list rails -r
   
6. You should see similar output as shown above.

Now we can work again without any issues when trying to install or update gems.

Enjoy!!!

Technorati Tags: Ruby, Rails, XP

Edit: If you can’t wait until the end, here is the download link.

Who hasn’t used the Annotate Models plugin written by Dave Thomas, of Rails Pragmatic Studio fame? If you haven’t don’t fret… The plugin basically adds a comment block to the top of all of your model classes documenting the current database schema for the given model. I like to have the info in my model files to make it a little easier to work with models.

The rails bundle within TextMate gives you a ‘Show DB Schema for current class’ command you can press to show a tool tip with the database schema (Control + Shift + Command + S). This works great, but I find the command sequence to long and the slight wait to see the schema info breaks my rhythm.

To solve the problem I made a copy of the source files used by the command and modified it slightly to output the info as comments at the top of the file. Now I have nicely annotated model files. This is a real time saver.

The bundle also contain the the Beautify command. This command adds automatic code formatting capability for your ruby code. I found the beautify command from Tim Burks. Cheers Tim…

You can download the bundle here.

Technorati Tags: Ruby, Rails, TextMate, JavaScript, Mac OSX

UPDATE: Tim Haines notified me that you can install all the necessary tools with one command:

apt-get install build-essential

Thanks Tim, I’ve tried it on a clean VPS and it works a charm…

If you are busy setting up a new Ubuntu Linux server and you get the following error:

configure: error: no acceptable C compiler found in $PATH

when trying to run ./configure, then you need to install the gcc compiler (Yeah, I know… this is like a no brainer for you *nix guru’s).

Here is the command to install the gcc compiler on Ubuntu:

sudo apt-get install gcc

But now you get the following error:

error: checking for C compiler default output file name

Most results on on google search point you glibc-devel as the solution. But, alas… apt-get will report the following message when you try to install glibc-devel:

Couldn’t find package glibc-devel

So what to do? The correct Ubuntu package that is similar to the glibc-devel is libc6-dev. Below is the command to install the Ubuntu package:

sudo apt-get install libc6-dev

Happy compiling and installing…

Technorati Tags: Ubuntu gutsy glibc-devel libc6-dev

I’ve updated the source code to include some conditional compilation directives to support different Visual Studio IDE versions. So if you are still stuck in VS2005 land, you can now build the source and be able to take advantage of the new features that Owen Evans added.

These include the new attributes (Context, BeforeAll, BeforeEach, AfterAll, AfterEach) and new functionality like Collection.Contains#WithProperty. If you lucky to be using VS2008 you’ll have new functionality that allows you to pass lambda’s. For more information look at http://bgeek.net/2008/02/14/nspecify-rspec-well-closer-anyway/

Owen also added a new library called NSpecify.Framework.Extensions. This gives you some extension methods that you can use to specify your expectations directly on the object under test.

i.e. newDeveloper.Surname.Must().Equal(“Evans”);

Nice hey!

Ruby

• Object.alias_class_method – adds a singleton/eigen/virtual/meta-class to the given instance which you can use to add methods.
• assert{ 2.0 } – An interesting new gem that make your assertion definitions feel more natural.
• Have you killed a design pattern today?
• Timeout code execution – to ensure a snippet of Ruby code doesn’t run for too long you can use the timeout function.
• Loading classes from strings – Using method like Module#const_get, Kernel#qualified_const_get and eval.

Rails

• Using DRb to preserve a Selenium Driver – Speed up your selenium tests considerably
• Integrating Google Maps in your Rails 2.0 application

TextMate

• HTML Bundle Tips – Great tips on using the HTML bundle in Textmate

JavaScript

• Language Integrated Query for JSON – This is really interesting. Shows you have versatile JavaScript is.

Mac OSX

• Install ruby memcached on Mac OSX

Subversion

• Distributed Version Control – A comparison between Darcs, Subversion, Git and Mercurial

Technorati Tags: Ruby, Rails, TextMate, JavaScript, Mac OSX, Subversion