Archive for the ‘Development’ Category

Choosing a data storage format

Published by Rob on May 6th, 2009 - in Agile, Development, Java

In case you haven’t noticed, XML is not a silver bullet. (google xml+silver+bullet). It is not, and should not be an automatic choice when thinking of a data storage format. The ubiquitous libraries for working with XML are often hard to use, and are often overkill for a simple storage format. In today’s world, I’d suggest that the following options should be considered (at least briefly).

  1. Native Object Serialisation
  2. Custom format
  3. XML – Extensible Markup Language
  4. YAML – YAML Ain’t a Markup Language (obviously created by geeks with the recursive name)
  5. JSON – JavaScript Object Notation

Join me in having a look at these formats, and I’ll let you know some of the issues to consider. The main problem I’m solving is for data that belongs to your own application. I’m not considering databases or interoperability.

Native Object Serialisation

Consider this briefly before running away. I’m particularly familiar with the idea of Java Object serialisation. I’ve used Prevayler in the past storing java objects, and xml (So while I’m having a dig at Java Object serialisation in general, I’m not specifically having a go at prevayler).

While the use of native object serialisation is often easy, it has costs, making the content unreadable by humans, coupling the data storage to your implementation language, and can create object migration issues. These costs will typically outweigh the benefits. Having human readable data to aid debugging would provide reason for not using native object serialisation if there was nothing else.

Custom Format

The use of a custom simple text format should not be discarded out of hand. The lack of any third party dependancies is a useful feature, and should be considered. That said, if you have a library that does the parsing for you, that should not be sneezed at.

XML

As wikipedia says, “XML is a general-purpose specification for creating custom mark-up languages” (Wikipedia on XML). Parsers and tools exist for many platforms and environments, which makes it a useful tool when you want to share information between different environments. While a good tool, the syntax is verbose, and can be hard for humans to read.

XML has influenced the birth of two of two more recent notations which are useful for data storage: YAML, and JSON

YAML

YAML purports to be “a human friendly data serialization standard for all programming languages” (Yaml.org). It has a well defined specification (YAML Spec), and makes for an easy to understand data storage format. Implementations of YAML exist for a wide range of languages, including Java, C++, Ruby and Javascript. It’s been around for a while, and has a decent amount of uptake. If it wasn’t for JSON, it would probably be a good default choice.

JSON

At first glance JSON seems much less suitable than YAML for languages other than JavaScript. The kicker against it is that it has “JavaScript” in the name, which has always made people feel icky. That said, it does make for a good cross platform format, it is human readable, and is implemented on a wide range of platforms (Json.org).

JSON has also has the advantages of having mindshare, and is slightly more familiar to developers than YAML. Every developer who has had anything to do with the web has done stuff with JavaScript, so the basic format will be familiar to them. Also in JSON’s favour is the fact that JSON and YAML are syntactically very close (see Redhanded). JSON appears to be very close to a subset of YAML(Ajaxian). In addition, the general applicability of JSON is higher, particularly for people who are going to be doing Javascript development. Also, if you have any possibility of playing with JavaScript, JSON is a very good option because of the native support in JavaScript.

These factors combine to make JSON an excellent choice.

Summary

Tim Bray makes a good case for this being an automatic choice based on your circumstances(http://www.tbray.org/ongoing/When/200x/2006/12/21/JSON ). You’ll still need to think about the pros and cons of the different technologies for your situation (see http://webignition.net/articles/xml-vs-yaml-vs-json-a-study-to-find-answers/), but you’ll often find that JSON is a good format to use for data storage.

Tags: , ,
1 Comment »

Bridged Network Connection + VMWare Fusion 2 + SMEServer + WiFi = Fail

Published by Rob on April 25th, 2009 - in Development

When recently doing some testing of a SME Server VM using my wireless network I was getting a variety of different connection failures. 

Accessing the server manager in Safari was giving messages containing text like: Safari can’t open the page…. Safari can’t establish a secure connection to the server.  Occasionaly connections would work but 9 times out of ten the connection error was occuring. 

In addition to the https related issues, ssh connections were being reset. A ssh connection attempt with full debug (-vvv) can be seen below.

ssh -vvv *****
OpenSSH_5.1p1, OpenSSL 0.9.7l 28 Sep 2006
debug1: Reading configuration data ****
debug2: ssh_connect: needpriv 0
debug1: Connecting to ****
debug1: Connection established.
debug1: identity file ****
ssh_exchange_identification: read: Connection reset by peer

Strangely google wasn't much help with my issue.  It was only after reinstalling a new VM using NAT that I saw things working.  From what I can see it must be the combination of the services running on SME Server, WiFi and Bridged networking.  

So if you have a Apple laptop, wireless, and SMEServer running in a VMWare virtual machine, use NAT for your test server and things will work much much better for you.

 

No Comments »

Getting php and MySQL up and running on OS X 10.5

Published by Rob on January 14th, 2009 - in Development, OS X Tools, Tips and Tricks

Much of the free and open source web technologies in the world runs on Apache, MySQL and PHP. Getting this up and running on your Mac is important to ensure you can do development on the Mac. Once you know the right steps to follow in setting this up, it isn’t too hard, but finding the right information can be a bit tricky.

As at January 15 2009, here is the process that works. If you have problems, let me know and I’ll update this page accordingly.

I’m assuming that you have basic knowledge of how to use a Terminal.

1) Enable php in Apache

Ensure that the php module is loaded. This is done in the httpd.conf, and simply involves uncommenting the LoadModule line. sudo vim /etc/apache2/httpd.conf remove the comment (#) char at the front of the line containing LoadModule php5_module

1
LoadModule php5_module libexec/apache2/libphp5.so

After doing this you will need to restart apache

1
sudo apachectl restart

2) Install MySQL

Download MySQL from http://dev.mysql.com/downloads/, and follow the instructions (if you go with the dmg file it’s really easy).

3) Download and Install the MySQL System prefPane

System Preferences is the place to start and stop things in OSX. Many versions exist. The one that actually works with 10.5 is available for download from the MySQL site: ftp://ftp.mysql.com/pub/mysql/download/gui-tools/MySQL.prefPane-leopardfix.zip

4) php.ini

You’ll need to tell php where the MySQL socket lives. To do this take the default php.ini provided, copy it to php.ini, and ensure that the mysql.default_socket and mysqli.default_socket settings both reference /tmp/mysql.sock (that’s for the MySQL and MySQL improved extensions).

1
sudo cp /etc/php.ini.default /etc/php.ini

1
sudo vim /etc/php.ini

1
mysql.default_socket = /tmp/mysql.sock<br />       mysqli.default_socket = /tmp/mysql.sock

After making these changes you’ll need to restart apache.

1
sudo apachectl restart

5) Install phpMyAdmin

phpMyAdmin is the standard tool for administering MySQL databases. It’s a php client that just works. While you might be able to get away with using a rich client on Mac OSX you should probably use phpMyAdmin because it’s the standard, and running on servers everywhere. Grab a copy from http://www.phpmyadmin.net, read the documentation (there is a quick install guide), and you are off and running.

1 Comment »

Fixing”No backend servers available” running Vignette on Weblogic

Published by Rob on November 26th, 2008 - in Development, Java, Vignette, Weblogic

After spending a considerable amount of time looking at WebLogic and Vignette groups and sides on the net, I finally worked out the cause of a recent “No backend servers available” error message which was occuring when trying to access the vcm 7.5 AppConsole on a recently installed version of Vignette.  In the end google didn’t know the answer to my question.  It’s time to teach google a lesson.

So after getting quite frustrated with what was happening, being unable to find documentation, and finding that a reinstall wasn’t helping much, I pulled out a developers true friend, and started poking around with the runtime services console.

After playing around with the runtime services, I was able to work out that the default install of Vignette  was setting up a cluster, containing the single server.  The combination of suspending and restarting my vmware virtual machine was leading to the Weblogic server to not be started correctly, resulting in the “No backend servers available” error when trying to visit the AppConsole. 

This problem was resolved by following two simple steps:

  • being patient after a restart, allowing the system to do all its checks, (waiting about 5 minutes worked for me) and then

  • simply starting the server using the runtime services console. clusters->[cluster name]-> control -> start/stop -> startup.

So if you end up in a situation where you are seeing “No backend servers available” with Vignette and WebLogic, don’t despair, it can be resolved by simply starting the servers in your cluster (even if it’s a cluster of 1) after patiently waiting (about 5 minutes) for the server to get into a stable state.

Thankfully google knows this too now.

2 Comments »

Configuring Windows 2003 as a Development Server

Published by Rob on October 16th, 2008 - in Development

Like many developers I spend a decent amount of time stopping and starting VMWare servers. Many of these are running Windows Server 2003. Win 2K3 has been designed as a server, and therefore has things like automatic tracking of reasons why the server was shutdown, which requires you to enter reasons into the dialog. As a server it also has pretty tight security around the Internet Explorer,making it even less usable than usual.

Fortunately (like always) the internet has the answers to these problems. The Microsoft Software Forum Network has a good series of articles titled “How to convert your Windows Server 2003… to a Workstation!“.  These talk about a whole heap of different options on what to do, the most useful for making W2k3 run as a development server are:

  1. Disabling Internet Explorer Enhanced Security, and
  2. Disabling the Shutdown Event Tracker.

Following the simple steps in these articles will help save time when working with a Development Win 2k3 server.

No Comments »

Reducing the Startup time for IBM WebSphere Portal

Published by Rob on September 10th, 2008 - in Development, Java, WebSphere Portal

After spending too much time today waiting for WebSphere Portal to restart (due to a nasty bug causing Portal to crash when I was working with it) I've been revisiting how to improve the startup time.

The key step in doing this is to uninstall portlets that aren't needed. (see http://www-01.ibm.com/support/docview.wss?rs=899&uid=swg21213982&ca=lsall for a list). For the lazy developers (read me) there are a series of scripts that have been developed that do this. The article Installing and configuring WebSphere Portal V6.0 Servers for development with Rational Application Developer V7.0 and Rational Software Architect V7.0 includes these scripts in a zip file (search for StartupPerformanceConfig.zip in the article). The article also has some good suggestions for memory and jvm settings for your server.

I have known about the existence of this process for a little while, but it took me a little to find them just now (when tuning a development server). Google was have troubles pointing me to the right article, so I thought for my sake I'd try and help google find this in the future. (Even if google did find the right spots, the title of the developerworks article wouldn't have made me think that it had the nugget of information about IBM WebSphere Portal Startup performance).

3 Comments »

SSH Tunnelling is your Friend

Published by Rob on September 9th, 2008 - in Development, Java

I've been using ssh tunnels for a while now. I first encountered the technique when setting up my e-mail client when doing research at the ISI. The ISI are understandably quite security conscious, so a ssh tunnel was required to access my mail on my mac. I'll admit that I set this up initially withot really having a clue, simply following the directions of the guys at the ISI that knew what they were doing. I had a couple of ssh commands sitting in an e-mail (and more often than not in my history), which I would simply refer to and reuse.

More recently at Ephox I've been doing a fair bit of ssh tunneling. I've been doing this to access the UK demo server for E2, exposing the Websphere admin and wcm portal urls via the tunnels. This process has worked relatively well for a couple of weeks for me, but in my current stage I'm going through some pain as the transfer of my ear file from the local build to the remote server is taking between 15 and 20 minutes currently. To get around this for some rapid bug fixing work, I've entered the next stage of my ssh tunneling jujitsu

  • ssh to remote server
  • ssh back to local Brisbane office, setting up a tunnel into subversion server
  • then use subversion to check out the codebase.

After having performed this setup work, I can easily do my changes on my local box, update the remote server via svn, then build and deploy from the remote server. The build and deploy are currently too manual, and could use some automation, but I'm currently happy to wear the pain, and drop my deploy process from 20 minutes to under five.  

My current next step is to automate the rest of the process.  Assuming my recent foray into sysadmin land didn't completely kill the server, this should be done very soon.

1 Comment »

Web Character Encoding

Published by Rob on July 11th, 2008 - in Development, Java

Making character encoding work well on the web can be a bit of a tricky task. There are many different layers to get right, and many different places in which things can go wrong.

At Ephox we have seen many of our clients experience problems with character encoding, and different fonts. We've actually created a variety of different articles to help people make there content editing environment work better:

http://liveworks.ephox.com/2007/05/15/why-international-characters-display-as-boxes/
http://liveworks.ephox.com/2007/05/08/solving-character-set-issues-with-legacy-systems/
http://liveworks.ephox.com/2007/09/03/character-encoding-and-special-characters/
http://liveworks.ephox.com/2008/03/11/url-encoding-and-character-sets/
http://liveworks.ephox.com/2008/02/05/controlling-entity-encoding

More recently I have been seeing some issues with encoding without EditLive! included. Sun proved to have a decent article talking about Character Conversions: http://java.sun.com/developer/technicalArticles/Intl/HTTPCharset/.  This articles includes good details on how to setup your JSP/Servlet layers to support encodings well.

This proved to be the answer to my problems, and well worth viewing if you are having encoding related problems in a Java Web application.

No Comments »

Can the Government be Agile? – discussion with a PMP PM

Published by Rob on June 22nd, 2008 - in Agile, Development

I’ve enjoyed my relationship with my brother-in-law for a long time. We’ve often talked about business and career, sharing where we are at, and the directions we are taking.

Over the past few years Paul has been doing a great job as a project manager in Canberra, in particular his involvement with the Department of Defence in the successful and ambitious e-voting project (Remote Electronic Voting Federal Election 2007). It has been great hearing about his work, and some of the stories around it. He has also been sharing his growing understanding of project management, and some of the courses and information he has been collecting, and applying. I have been listening with interest, and a degree of skepticism, thinking that much of what he has been learning would not apply directly to me, as it seems to be focused on heavyweight/rigorous methods, and I very much prefer working in Agile environments, where good smart people are encouraged to do great work.

So it was with great interest that I was recently listening to him discussing and presenting some of his recent thinking and work around scrum. Paul has been looking at how to apply scrum to government projects, and has some great ideas about this. I was particularly interested to hear what he saw as the strengths that scrum would bring, and the advantages he saw in outcome based results (delivery of working software that meets outcomes) versus delivery based results (traditional waterfall delivery of documentation is part of the results).

I look forward to seeing Paul’s work going forward, and to seeing how he continues to apply agile techniques to his work. If you are in Canberra, and are looking for a great contract or consultant project management, it is well worth talking to Paul and the team at Barton Institute of Management.

3 Comments »

Howto Learn an API

Published by Rob on June 20th, 2008 - in Development, Java

I recently have been learning a number of different APIs. After reflecting on my process for learning I have ended up with the following strategy for working with APIs.

  1. Find someone who has used the API in the past and get a brief introduction.
  2. Read documentation about the product (starting at the central documentation site for the product, include any tutorials and getting started guides). The goal from this first step is to get a general background understanding of what the product is about, the core concepts and jargon and principles of the product.
  3. Use the product without the APIs. If there is a gui to access to product that is available, ensure that you know this, and the jargon around the product. Try and ensure that the information you have from step 1 is grounded in reality.
  4. Read more documentation around the APIs. Make the top goal for this round to get enough of an understanding of what you can do.
  5. Explore the APIs to see what they expose of the feature. The set of features exposed by APIs often do not match up with what is exposed through the GUI. The most important spot to aim for is the subset of features that are supported by both the API and the GUI.
  6. Google (as usual it’s your friend). Sometimes there are some useful results in google that are missed elsewhere. Start with searching for the product name, try adding in API/howto/tutorial/getting started keywords to filter down content if you get too many results.
  7. Write some exploratory code to see how the APIs work, and what they do. This should enable you to know enough to actually write the code that you need.

All steps are required, as there are often features which are not easily discovered by just taking one of the above paths. One of the most important steps is ensure that you do understand the subset of what is supported by the API and the front-end. This helps to ensure that you are actually hitting the right part of the API.

From time to time there are options which are not well documented at all, and are only alluded to in technical documents. It can be worth contacting authors of documents if the feature is not clear. The coding and exploration of the system will need to continue after this. It should be expected to continue learning the APIs. At every stage there is value in talking to whatever experts/more experienced people you have available. Being able to quickly pickup a technology and move from no knowledge to confidence in using the API is a valuable tool for developers. Learning new technologies has been an important part of my job as a software engineer, and I expect that it will continue to be for many years to come.

No Comments »
© Rob@Rojotek