codecraft blog

Search as Data Store

2011-11-29T08:47:00.000-08:00

A while back I came across Solandra, and began wondering about using it as a general purpose store for a typical content-heavy webapp. The idea was based on a pretty vague notion that combining search with NoSQL would be powerful. What Solandra does is bolts the SOLR search engine onto Cassandra instances, resulting in a distributed search with all the data persisted in a NoSQL store.

In poking around, I've found that it is not unheard of to use SOLR as a web backend. Perhaps the most prominent example would be The Guardian's use of SOLR for their "Content API." There was also supposedly a twitter-like example that "uses the Lucandra store exclusively and does not use any sort of relational or other type of database", though the link did not work as I wrote this (Lucandra is the predecessor to Solandra).

Why not just stuff everything into SOLR? It is effectively a document store with powerful and fast queries. Sure it does not support transactions and the data is not normalized in a relational sense. But do you need those things for content? What I mean by content is dynamic stuff that might be edited by humans or updated by feeds: articles, comments, tags, product descriptions, locations; data that is viewed a lot but edited infrequently, not super structured or requiring crazy integrity. With the Cassandra backing, you should at least get durability and eventual consistency on the data that you shove into SOLR (Solandra).

I guess now I will have to give it a try, see where the trade-offs are in practice between search, NoSQL, and Relational stores on a real content project.

Objects in practice: class basics in Java and Ruby

2011-11-15T15:57:00.000-08:00

This post looks at basic class and inheritance mechanisms in both Java and Ruby. I might to decide to expand this into a series, illustrating basic programming building blocks using Java, Ruby, and Scala; we'll see how it goes.

For this post, we are modeling employees of a software company, something like:

(via YUML:
edit)

NOTE: This is NOT a good design--doesn't allow employees to fulfill multiple roles, doesn't account for changes over time--but I'll show better designs in later Howtos.

So, the general idea is that we want to reuse some behavior and also provide some specialized behavior. Probably the most simplistic way to do this in OO languages is to set up concrete inheritance and leverage polymorphism. I'll show how this is done in Java and Ruby, and the complete code can be found at github

JAVA

A simple parent class, provides startDate member and monthsOnJob to subclasses:


public class Employee {
  private Date startDate;

  public Integer monthsOnJob() {
    Integer months = 0;
    // Do the math
    return months;
  }

  public String jobDescription() {
    return "New employee"; // Default description, will be overriden by subclasses
  }
}

A simple subclass, adding behavior to the super and overriding the description


public class Developer extends Employee {

  @Override
  public String jobDescription() {
    return "Developer: " + listTechnicalSkills(); 
  }

  private String listTechnicalSkills() {
    String skillStr = "";
    // TODO get skills from getTechnicalSkills and build string
    return skillStr;
  }

  private Collection technicalSkills;

  public Collection getTechnicalSkills() {
    return technicalSkills;
  }

  public void setTechnicalSkills(Collection technicalSkills) {
    this.technicalSkills = technicalSkills;
  }
}

Another specialization, Manager has Employees as reports


public class Manager extends Employee {
  Collection reports;

  @Override
  public String jobDescription() {
    return "Suit";
  }

  public Collection<Employee> getReports() {
    return reports;
  }
}

Ruby
Now the parent class in Ruby


class Employee
  attr_accessor :start_date # method that creates accessors

  def initialize(start)
    @start_date = start
  end

  def months_on_job
    # Math is easy
    (Date.today.year*12 + Date.today.month) - (@start_date.year*12 + @start_date.month)
  end

  def job_description
    "New employee"
  end

end

A Ruby Developer class


class Developer < Employee

  attr_accessor :technical_skills

  def initialize(start)
    super(start)
    @technical_skills = []
  end

  def job_description
    ret = "Developer: "
    @technical_skills.each {|skill| ret << " ${skill"}
  end

end

The Manager


class Manager < Employee

  attr_accessor :reports # accessor
  def initialize(start)
    super(start)
    @reports = []
  end

  def job_description
    "Suit"
  end

end

John's Independent Forray v3.0

2011-10-24T19:59:00.000-07:00

After 3.5 years with Rearden, I decided to go back to freelancing, for the 3rd time.

This time around I am going by codecraft solutions. I am presently subing with Marty Haught and intend to continue for some time. Eventually though I intend to run my own projects too.

That's the plan, stay tuned.

Rearden Commerce hiring initiative

2011-02-23T12:49:00.000-08:00

http://goo.gl/2UePJ

Job: Senior Software Engineer: JVM Webapps and Content - Denver area

2011-01-27T15:08:00.000-08:00

Work from home in Colorado, for Rearden Commerce (Foster City CA)

Senior Software Engineer: JVM Webapps and Content

Responsibilities:

Your role will be to develop and maintain web applications and middleware related to aquiring, managing, and accessing mission-critical content. The role calls specifically for the development of user interfaces, but also requires "full-stack" knowledge and development. As a senior contributor, you will own the architecture, design, and implementation of these applications and will be responsible for delivering new capabilities with high quality. You will have autonomy and responsibility for the following:

* Partnering with product management, user experience developers, QA, and services delivery to understand business requirements and translate them into high quality designs
* Developing user interfaces and APIs using HTML, CSS, JavaScript, and JVM languages and frameworks (Java, Groovy, Maven, Spring, Grails, GWT, Wicket)
* Developing high-volume batch processing solutions using map-reduce, batch frameworks, NoSQL, and message oriented middleware
* Taking ownership of the quality and usability of the application
* Contributing to a culture of innovation and quality across the engineering organization

Qualifications:

* Minimum 5 years of commercial web development experience in Java, preferably on a Linux/UNIX environment
* Strong background with user interface design and development featuring XHTML, JS, CSS and AJAX
* 3+ years HTML/XHTML/CSS/JavaScript/DOM cross-browser development experience
* 3+ years experience developing middleware and back-end code: SQL, JMS, Java ORM, etc.
* Experience with multiple programming languages including Groovy, Python, Ruby, or Scala.
* Strong object-oriented analysis and design skills
* Experience with design patterns (e.g. MVC, Composite, Factory Method, etc.) and refactoring
* Excellent written and oral communication skills

* Experience and competence developing in *nix environments and shells
* Able to collaborate and thrive in a fast-paced high-performance environment
* BA/BS in Computer Science or equivalent experience

Desired Qualifications:
* Experience design tools
* Experience working with Agile Methodologies
* Experience developing UI’s for mobile applications on Android or iPhone devices is a plus

Respond to jtroxel at yahoo.

The Holy Grail(s) UPDATED

2007-03-09T08:49:00.000-08:00

In Which I Claim that Java is NOT Dead, and Groovy will be Bigger than Ruby

UPDATE 12/11/2007: I have recently started using Ruby and Rails on a Java project. Does this invalidate my assertions here? I don't think so, it merely underscores the key application for Groovy: tight integration with Java source and libraries. I had the need to do some schema management, for which Rails migrations are the simplest approach I know, and some batch scripting; neither of which intersected with Java at all. A final factor is that I just know Rails far better at this point.

As I have talked about in previous posts, Java (EE) has been optimized for Enterprise requirements: Big applications with heavy-duty scalability, performance, and security, often weaving together many disparate databases and integrating with other applications. On the other hand, Java has most decidedly NOT been optimized for productivity, maintainability, or developer self-actualization. Although some frameworks and tools have attempted to address the latter set of requirements, nothing comes close Ruby and Rails.

For green-field development, small-medium sized projects (and medium can mean easily thousands of users nowadays), I would really, really lean toward Ruby on Rails. However, what about for projects in organizations that have standardized on Java, or for developers who are already Java ninjas? Enter Groovy, and Grails.

The promise of Groovy, and Grails (though I am sure there will be other frameworks), is to bring productivity, maintainability, and, well, fun to the Java world. Because these are Java-based technologies, projects can leverage these capabilities to get to market quickly, and Scale Later, falling back on all those nice big E Enterprise features of the Java EE stack. That, I think, is the Holy Grail of Java development: Rapid development early, beefing up later.

That is why, IMHO, Groovy and Grails might be as big as, if not bigger than, Ruby and Rails. I expect I will be writing a lot more about this in the future. Also, look forward to a new blog called GroovyCode, where I intend to capture bits of Groovy coding goodness.

For your further reading pleasure, I include here an email thread I had with Scott Davis, a Groovy pioneer and operator of aboutGroovy (as well as all around coding Rock Star) [from Aug. '06]:

> Thanks for the info, very helpful. As for the client, it's an
> outside shot right now but I will see where it goes. I originally
> suggested ROR, but he wasn't "comfortable" with it. He couldn't
> really state the requirements that he thought were lacking, but I
> think he just feels it is not industrial-strength enough. In order
> to get his business, then, I proposed a Java solution.
>
Ahh, "industrial-strength". Now he's showing his cards. Grails is
based on "industrial-strength", well-proven technologies like Spring,
Hibernate, et al. I firmly believe that RoR is "industrial-strength"
as well, but it doesn't have the track record that Java does at this
time. +1 Grails.

>
> Now here comes GOG, seemingly on the cusp of critical mass. Which
> got me thinking that I could offer ROR productivity, with the
> option to (readily) leverage all the enterprise capabilities of
> Java down the road. And I can talk about reusing some of the code
> I have, though it might not actually make sense.
>
You've hit on the sweet spot of Grails. I honestly haven't gone into
a Grails app and seen what exactly is salvageable in terms of pulling
out the Spring, Hibernate files and using them in a pure Java
environment, but I can't imagine that it'd be too hard.

> Well, there. Probably more than you wanted to know. Before I let
> you go... What do you think the productivity (functionality /
> time) advantage of GOG is over Java Webapp development? Any reason
> it could not scale to, say, 100s of 1000s of users?
RoR folks like to claim 5x-10x increase in time to production over
J2EE. I'm not *even* going to venture into that rats nest for Grails,
especially this early in the game.

The bootstrapping aspect of Grails is pretty compelling -- download a
single tarball and *poof*, instant app. It, however, is running Jetty
and HSQLDB. I haven't played with hooking it up to an external db, or
running it in another servlet container, but that certainly
diminishes the "out of the box" / time-to-production. Of course, this
is pretty much a one-time cost.

Creating initial views is pretty easy (grails generate-all), but hand
editing them takes some time and coordination. If you don't like the
default ordering of the fields, for example, you need to hand edit
list.gsp, edit.gsp, view.gsp, new.gsp, etc. If you add a single new
field to the table, there is no clean/automated way to migrate your
views as well. This is an ongoing cost.

I think the biggest win is letting Grails handle all of the
"plumbing" issues for you -- Spring, Hibernate once again.
Infrastructure like that getting autogenerated is huge.

> How about ROR by comparison?
Much more mature, and I think at this stage of the game it has a
better coordination/migration story for iteratively adding fields and
keeping your model and view in sync. For green-field development with
no ties to Java, it's a pretty compelling story.

W2I

2006-10-03T09:33:00.001-07:00

I have recently made my independent business more official, I have a card and everything:

I help clients as an independent: either as a highly productive hands-on contributor, or by acting as their Solution Architect and "General Contractor" for software Design and Development. Java, Ruby, Groovy... Architecture, design, project management... I can help get it done.

With over 13 years as a Developer and Manager, I know how to get projects done, and who to put on the team. I can provide the quality skills that full-service shops offer, at independent contractor rates. With my network of talented professionals, I can get it done on sites and applications from $500 to $50,000.

For more information on me and what I do, you can browse to Rapid and Affordable Development or take a look at my Resume.

- John
jtroxel@yahoo.com

What's So Great About Rails?

2006-09-20T21:05:00.000-07:00

OK, so now I guess I am becoming a "Rails Guy." At least, I have been doing some Ruby on Rails programming lately and am generally very impressed.

It is still Web programming, there are no major paradigm shifts here. However, Ruby on Rails is an example of how a lot of little things done right can really add up. Ruby is a fine scripting language, improving upon popular languages like Perl and Python. And Rails is an example of getting the little details of Web programming right. Rails is optimized (as I see it) for (1) Productivity and (2) Maintainability. The following are what I really like:

The clean, simple ActiveRecord ORM layer.
A full-fledged yet simple MVC that is easy to comprehend and get around in.
Sensible defaults and very little configur-bation; "convention over configuration."
A good job of making the most common tasks simple; the 80/20 rule.
"rake migrate" is the coolest way to manage schemas that I have seen.

So, is Rails the answer to everything? Of course not, you won't find sweeping generalizations here. As always, it depends on *specific* requirements. Ruby and Rails are geared toward fast and simple development and maintenance. Java and JEE technologies generally have not prioritize these factors as highly as Enterprise requirements, e.g. massive scalability, security, etc. And I am not saying that one or the other stack is incapable of solving a kind of problem, rather that it is a matter or priorities, focus, and fit.

So I do like Rails a lot, I think it is probably the best solution for many small-medium sized webapps. But it is not necessarily a replacement for Java, especially in the enterprise. The trick is for decision-makers to weight their needs for big E Enterprise requirements against speed and simplicity.

Productive Editing and Eclipse

2006-09-19T20:59:00.000-07:00

I went to a great presentation by Neal Ford last week at the DJUG, entitled the Productive Programmer. Neal is a sharp guy, a good speaker, and a solid thinker on improving productivity. Go ahead and CRTL-Enter nealford in your address bar to check out his site (you'll see what I mean).

The talk was a bit of a wake-up for me. His premise is that the rise of the IDE has made programmers, well, lazy (my words, I don't think he went so far). I realized that I have let some old but valuable skills languish in the past few years. So Neal's talk has inspired me to sharpen the saw back up. For example, I have made the decision that Ruby will be my scripting power tool moving forward (used to be Perl), and I will not shy away from opportunities to explore automation (even if it is initially just exercise).

Another point that resonated with me was the idea of the "perfect" editor. I do think that I am now as productive over all in Eclipse as I used to be in Emacs (with a side trip through NetBeans). But Neal hit on two things that are missing: Multiple cut/paste buffers (kill-loop in Emacs) and editor scripting. Though I am intrigued by the lightweight JEdit, I am determined to stay in Eclipse and make the best of it. I think that surely these two deficiencies can be addressed.

On the multiple buffers front, I quickly came up with an answer that works for me: CLCL. This tool is actually a utility that caches your Windows clipboard. So CRTL-C still works like normal, but you can also reach back in time and paste something from 30 cuts ago. Sounds trivial, but it can change the way you edit.

As for the scripting integration, I haven't looked very far. Prashant Rane's blog led me to Eclipse Monkey, which looks interesting--it's Javascript, which I can defnitely code. I might keep looking, though, for something that provides editor hooks for Ruby, Groovy, or BeanShell. I'll post again if I find the silver bullet.

Shameless Promotion 4: Rapid, Affordable Application Development

2006-04-27T11:36:00.000-07:00

I am available to help Clients develop applications cheaply and quickly, applications that help them to reach more customers and reduce costs. I can help clients use the Web to make breakthroughs in marketing, sales, and operations.

I have built dozens of successful applications for the Web and other environments. I can provide hands-on expertise across the software development lifecycle: from strategy and requirements to architecture and design to development and testing. I also provide project management and "general contractor" services, so I can bring the right talent to the table for most any web development project.

The following is a quick summary of the skills and capabilities I provide to help you achieve robust solutions quickly and at affordable rates:

Experience in online communication, eCommerce, and improving efficiencies.
Technical lead, architect, and developer experience on dozens of projects, including complex applications for Sheraton Hotels, Travelocity, Budget Rent-A-Car, and the Veteran's Administration.
Programming experience in Java (5+ years), C (3), VB/ASP (3), Perl (2), and lately Ruby.
Effective, hands-on skills as a software engineer, architect, and business/systems analyst.
MBA and a Bachelor's in Computer Science.
Extensive network of software and Design professionals for general contracting services.
Very competitive rates, for myself and subcontractors.

My Resume describes my skills in much more detail. I'd love to make your development project a huge success, drop me a line if you are interested.

- John

jtroxel@yahoo.com

Software Success Factor #1: Manage for Agility UPDATED

2006-04-26T14:00:00.000-07:00

The ability of an organization to Manage for Agility is one of the most important success factors for modern software development. This factor encompasses the critical role that Management--or the project Client or Sponsor--plays in Iterative Development. In this post, I will attempt to address what this factor entails and how it is accomplished. I will also briefly recap why this management style is superior. I have yet to complete the Overview of the Reference Model, but Agile Management is so important that I wanted to flesh out this success factor first.

What
Iterative development is perhaps the key attribute of modern software success, but without management support it is impossible. To manage for agility, leaders must go against many traditional management styles and ingrained practices. It is a natural for successful business people to try to avoid future change through detailed planning and up-front analysis, but changes are inevitable. Trusting the team and the process to produce the best solution, allowing the scope, the design, and even the process to emerge, is what it means to Manage for Agility.

How
To manage agile projects, sponsors must resist the desire to plan and design the entire project before starting any development. To be agile, management must instead embrace a flexible model that responds to changing requirements and an evolving understanding of the solution. Management must trust the process and the team to produce top quality software, without a relying on a rigid scope.

Often it is the "commercial Model," contracts, SOWS, etc., that most influences the interaction between management and the project team. To manage for agility, the commercial model must provide for evolving scope and design. Put another way, attempting to fix the scope in agreements up-front is simply at odds with an agile style. When using consultants or contractors, a Time-And-Materials approach allows for the most agility, or the agreement can employ separate SOWs created at the start of each iteration. Another approach offered by Kent Beck is the use of "Optional Scope Contracts."
It is also important to understand that agile does not mean that no planning or requirements are done up front. The project still needs a first cut at the inventory of requirements--e.g. a Scrum backlog or a Use-case survey--that is the source from which iteration requirements are selected, refined, and built. An agile project can also develop a detailed, "pro-forma" project plan, but managers should not feel compelled to stick to the original as the team adjusts to accumulating change.

Why
Why should managers buy into this approach, when traditional project management techniques have worked on all kinds of projects, from construction to aerospace engineering? Because software development is fundamentally different from construction or engineering, and treating it like these disciplines magnifies the risk of failure, raises costs, and produces worse software.
Change is unavoidable in Software Development, more so than most any field. Alan Shalloway made the claim, at a recent Agile Denver meeting, that software development is not like building a bridge; it is more like building the first bridge... or the design phase of a bridge project. An analogy is also often drawn between construction and software development. However, many experts claim that software development is essentially all design work, and the only step that is really like construction is the compile/build cycle.
The highly successful Scrum process is based on proven new-product development processes used by Japanese companies. Indeed, the Scrum proponents and many others have claimed, accurately I believe, that software development is very like new product development. If we look at software development as roughly analogous to the design and development of a new kind of car, or a new consumer electronics gadget, it becomes clear that the nature of the process is unpredictable and should be managed accordingly. Agile management uses an empirical process management approach to produce a better product. One need look no further for evidence than the difference between American cars and Japanese, at least until very recently.
Alistair Cockburn expands on the idea of "responding to change over following a plan:"

...traditional process management—by continuous measurement, error identification, and process refinements—strove to drive variations out of processes. This approach assumes that variations are the result of errors. Today, while process problems certainly cause some errors, external environmental changes cause critical variations. Because we cannot eliminate these changes, driving down the cost of responding to them is the only viable strategy. Rather than eliminating rework, the new strategy is to reduce its cost.
[NOTE: I think he means code rework--Big Up-Front planning and design will result in more rework of other work products, assuming the effort is not wasted entirely as these products are abandoned.]

Business needs, priorities, understanding of user needs, expertise with the
technical platform, market forces, and available technologies... All of these
can and usually do change over the life of a project. Agile management rolls
with these changes and capitalizes on them to make the best software possible.
Smart project sponsors can use agile management to build better software for
less money.

The Software Success Reference Model - Overivew

2006-04-14T09:45:00.000-07:00

Here, again, is my Reference Model: a 30,000 ft. view of software success. This post will provide an overview of the categories from the model--sort of a taxonomy of success. See my previous post for more information on how I derived this model.

Groupings
Organization and Project Management
Perhaps this should be entitled simply Organization and Management. This grouping deals with factors that are beyond the scope of any project team--organization and management at the highest levels.

Project Team
These factors deal more specifically with the project team involved directly with producing the software.

Iterative Development
This category contains factors that relate to producing software in an iterative, incremental, adaptive manner.

Productive Culture
Factors dealing with the kind of people the Organization employs and how the Organization fosters a culture that promotes productivity.

Productive Practices
Specific practices that help the Project Team develop software efficiently.

Software Engineering (for Large Projects)
Factors necessary for projects with a large number of developers and/or critical consequences of failure.

Factors
Manage for Agility
This factor encompasses the critical role that Management--or the Client--plays in Iterative Development. Leadership must be willing to launch a project without detailed plans or specs, and trust the team and the process. Further, project clients must realize that the agile approach is a Good Thing, allowing the team to respond nimbly to a changing world, and producing more relevant software in less time.

Deliver Value Quickly & Often

Another aspect of Iterative Development, this factor focuses on the ultimate goal of software development: working software. Teams must focus on producing working software from the very first iteration, and keeping implementation as relevant to the needs as possible by avoiding unnecessary interim work products and long time lags.

Continuously Learn & Optimize

The final component of Iterative Development is the concept of continuous improvement. Iterations afford the team with the opportunity to exercise all lifecycle activities in short bursts. This means they can reflect, learn, and optimize their process and practices with each cycle.

Empower the Project Team

Also solely in the Management arena, the client of the project must provide the project team with the tools, resources, and support necessary to succeed. As much as possible, the team must be allowed to organize and direct itself however necessary to deliver the requirements.

Enable Open Communication

Software development is inherently a team design activity. Communicating ideas and building a shared mental model of the solution are critical. Organizations must strive to achieve "osmotic" communication, provide access to experts, and involve users throughout the process.

Demand Quality

Teams must develop a high standard of quality as a norm, and internalize quality validations within the team, rather than "throwing" code "over the wall."

Use More Realistic Specification(s)

Just as a picture is worth a thousand words, the richer, higher-fidelity one can achieve in software specifications the better. Teams should use mock-ups, prototypes, and visual models as much as possible, while still constraining specification to whatever is "minimally sufficient" for the current increment.

Use Automation and Tracking Tools

The value of SCM, task & issue tracking, automated tests, etc. is well established. Just do it.

TODO: Software Engineering factors

Software Development Success Reference Model - Background

2006-03-02T11:45:00.000-08:00

What does it take to run a successful software development project? Success does not come easy, but it can be done. The industry has many success stories, and software development thought leaders have identified the key principles that lead to success. But what are the critical success factors, and what principles find common ground among the experts?

About 6 months ago, I started trying to create a set of success factors that were synthesized from the experts. I have since completely revamped my thoughts. What I ultimately want, for my own purposes, is a "Reference Model" (e.g. the OSI 7-Layer Model) that provides some structure with which I can I can think and talk about the various factors that lead to success--independent of any specific methodologies. I hope to be able to say things like "Reflection Workshops in Crystal are one technique to Continuously Learn and Optimize," or "XP provides a system for Iterative Development and Proven Practices, focused on smaller Project Teams."

I have now spent some more time thinking, and included some more popular, respected sources. The following slide is my Reference Model, a 30,000 ft. view of software success.

In future posts, I will provide more details on the individual success factors and how they relate to the work of industry luminaries. So, how did I arrive at this model? I started by taking the most important, highest-level principles and factors from the following excellent resources:

The Standish Group's CHAOS study [CH1..9]
The Unified Process or RUP [RUP-1..6]
The SCRUM meta-process (web, book) [SC-1..4]
Alistair Cockburn and Crystal methods (web, book, book) [AC-1..7]
eXtreme Programming (web, web, book) [XP-1..5]
My own experience on dozens of successful projects [JT-1..6]

There were several steps between extracting the critical success factors from these sources and defining my "nutshell" view, but the following slide captures the start of my reasoning. This is an "affinity diagram" that attempts to consolidate all the factors into logical groupings.

Comments are most welcome.

New Frontiers of Enterprise Java Development

2006-01-11T10:24:00.000-08:00

I've been thinking lately about trends in enterprise software development, especially in the Java world. Overall, what I see is a change in priority for enterprise software projects: from struggling not to fail completely, to actually excelling and competing. The latest technology and tools, along with modern "Agile" development methods, enable companies to really compete on quality and functionality, efficiency and productivity. Smart development organizations can not only avoid failure, but deliver better software more efficiently.

Background:
In the mid-90s, the ubiquitous distribution and instant maintenance capabilities of Web-based applications pushed them to the forefront. When applications moved beyond brochure-ware, a significant problem arose: Teams developing Web-apps could not readily achieve the enterprise requirements, or qualities, necessary for success: performance, scalability, security, etc. So, most of the effort and money spent by development teams went to achieving the necessary enterprise qualities.

Teams also struggled with process in the 90s, and to a large extent still do. The fast and loose style of early Web development did not work for mission critical apps, and traditional Methodologies did not work for Web teams. Furthermore, I think the frenetic rate of business and technical change finally killed the notion of viability of traditional, waterfall processes, at least for shops that "got it. "

By the late 90s, the major Web application platforms, like J2EE and compliant app. Servers, made it practical to address Enterprise requirements--although not easy or efficient. In addition, the effects of Moore's Law have made it much cheaper to throw more hardware at a solution, reducing the effort involved to handle the scale of enterprise apps. At the same time, Agile project management approaches began to gain visibility, blending the engineering-like discipline with the creative nature of software development.

In 2006, I contend, the development tools and frameworks for Web development have largely caught up to the platforms. Creating a robust, scalable, enterprise app no longer necessitates an army of J2EE coders rolling EJBs by hand. Looking forward, ISVs and IT shops just don't have to focus as much on addressing enterprise requirements*. Nor do teams need to struggle with heavy Methodologies--there are new mountains to climb. It is time for Web application development to grow up and act like "real" product development.

Competing with Agile Project Management:
Today, I think any manager who doesn't at least vaguely acknowledge the benefits of an Iterative or Agile project management approach has been hiding under a rock for several years. Some managers even claim to use these approaches, although few really do it right. Just because a team breaks up the coding for a project into increments, doesn't mean that they are Agile.

To improve productivity with process, management must get away from the compulsion to scope, define, and plan the entire project up front. Rather, they must accept and embrace two essential facts:

Needs and requirements will change constantly during the project; and
User needs, technological pitfalls, and team capabilities will never be adequately or accurately understood until the team has gone through the complete lifecycle with one or more slices of the working system.

Yes, it's a catch-22 situation: Managers have to accept that reality. Managers must embrace change, accept the empirical nature of the software development process, and stop trying to do all the planning and specification work up-front. Companies still trying to use linear, silo'd, paper-heavy methods for Web-based apps are dead, whether they realize if or not. To increase productivity, and reduce costs and time-to-market, managers must truly understand and utilize Agile management. I personally think that Alistair Cockburn's work on Agile development is the most comprehensible. Scrum is perhaps the best overarching management view, and XP provides a system of proven team practices.

Another project management "move" that can greatly reduce development cycle times is rapid prototyping. Simply put, exploring and defining requirements with documents is a relatively poor way of getting the system right. More productive projects user "richer" media for definition and validation, and the richest vehicle is a working prototype. For example, a team can develop a Ruby on RAILs prototype to quickly and cheaply explore and validate the functional requirements. Another powerful option for rapid prototyping and validation is Gorilla Logic.

The working prototype provides much more comprehensible, complete, and unambiguous information than stacks of documentation, which saves a ton of wasted work downstream. If one accepts that an iteration or increment can be thrown away, then prototyping fits very nicely with Agile methods.

Though not strictly a project management move, I also think that automation has tremendous potential to make teams more competitive. That will be the subject for another post. For now, I will simply say that I think that XDoclet is definitely a tool that teams should look at using and extending... and MDA tools like AndroMDA show some potential as well.

Competing with Richer User Interfaces
Along with the powerful distribution and update benefits of the Web came with a catch: The capabilities of a browser-based interface, though often visually attractive, are weak. Things like tree browsers, master-detail views, and fine control over interface widgets have generally been painful to achieve. Teams did not have time to work around these issues, and the user experience suffered.

Now that the technologies are available that help to take care of enterprise issues with much less effort, teams can make there applications better with richer user interfaces. AJAX, a technique for making asynchronous HTTP requests in the javascript for a page, seems to be a very hot technology of late. For applications that require a highly graphical look and feel, along with better usability, Macromedia Flex is strong player. But the next great client platform, I believe, is Eclipse RCP.

*NOTE: I also think that tools Ruby on RAILS, PHP, ColdFusion, etc. might now present a viable challenge to J2EE and .NET in the Enterprise space, with the right hardware.

**NOTE: I really don't have a feel for what is going on in the Microsoft/.NET world. However, to their credit, I think Microsoft has always focused more on productivity than on Enterprise requirements (e.g. VB). How successful they have been on this front is a matter of debate.

Implementing the Value List Page Iterator Pattern

2005-12-01T08:28:00.000-08:00

UPDATE: Brad Long's Article is now found here.

I have come across a situation that warrants the use of the "Value List Page Iterator" pattern. First off, the official name is the "Value List Handler," previously known as the "Page-by-Page Iterator." The pattern is part of the Core J2EE Patterns from Sun. I have chosen to use the name "Value List Page Iterator" because the other two names miss the main points: iterating over a value list grouped into pages. The context is situations where a query can return more results than can be presented on one screen, the solution is to allow the user to page through results, like with your typical Web search engine.

The code part of the pattern is simple enough, and the data access is nothing fancy--as long as the query is pretty quick, and/or it is acceptable for the app to keep all the results in memory. In my case, the query takes minutes and can return potentially millions of results. I need something that "slices" the data from the db into manageable chunks. Brad Long of Oracle has written up a pretty comprehensive treatment of database slicing strategies and offers his own variant of the pattern. Brad nails down the best way to do a slicing query, which I have tweaked:

SELECT *
FROM (SELECT ROWNUM RNUM, INLINE_VIEW.* FROM
(
your-select
your-where
and key-or-order-by-field > lower-boundary
[order by order-by-field]
) INLINE_VIEW
)
WHERE RNUM < max-range

My current problem is that the row-limited (sliced) query takes just as long as the full query. So unless I can figure out some way to effectively cache the query in the database, I am still faced with either sticking all of the results in memory, or taking the performance hit of executing the query multiple times. I am also thinking about a hybrid solution, where I guess at the most likely initial pages and get those at once, then re-execute the query if the user navigates outside the initial results.

More to come...

More on Composition versus Inheritance

2005-10-20T16:54:00.000-07:00

Is there any more basic OO design decision than this? I started off, like many newer OO programmers, writing code with a lot of concrete inheritance. After some painful lessons and more reading, I now lean heavily toward using composition. So OK. How do we nail down exactly when to use which approach?

Bill Venners writes about this in JavaWorld. The key criteria to consider is whether the relationship is really an “is-a” relationship, and just how strong the is-a is.

Make sure inheritance models the is-a relationship [...]
An important question to ask yourself when you think you have an is-a relationship is whether that is-a relationship will be constant throughout the lifetime of the application and, with luck, the lifecycle of the code. For example, you might think that an Employee is-a Person, when really Employee represents a role that a Person plays part of the time. What if the person becomes unemployed? What if the person is both an Employee and a Supervisor? Such impermanent is-a relationships should usually be modeled with composition.

He continues to point out that both code-reuse and polymorphism can be achieved with composition, and should not be a reason to use inheritance without a strong is-a relationship.

java.net hosts a discussion on the topic. A common approach is to code using interfaces, and use concrete inheritance in the implementations--in other words, use composition (with interfaces) in your design, use inheritance (for re-use or polymorphism) in the implementation. The is-a rule would still apply to the implementations, of course.

OO Design Principles: a Quick Rundown

2005-10-14T12:38:00.000-07:00

I wanted to collect some good resources on basic OO design principles, with brief summaries of each, so that I have everything in one place. I hope to ultimately distill these into a one-page "crib sheet.'

Some of the best wisdom in OO design is captured by the Gang of Four. Artima has an interview with Erich Gamma where he discusses the core principles from the intro of Design Patterns:

Program to an Interface, not an Implementation

Once you depend on interfaces only, you're decoupled from the implementation. That means the implementation can vary, and that's a healthy dependency relationship.

Rod Johnson expands on this idea in Chapter 4 of the pre-Spring book, and provides some very well-phrased guidelines:

Program to interfaces, not classes. This decouples interfaces from their implementations. Using loose coupling between objects promotes flexibility.
...
A few of the many advantages of an interface-based approach include:
The ability to change the implementing class of any application object without affecting calling code. This enables us to parameterize any part of an application without breaking other components.
Total freedom in implementing interfaces. There's no need to commit to an inheritance hierarchy. However, it's still possible to achieve code reuse by using concrete inheritance in interface implementations.
The ability to provide simple test implementations and stub implementations...

Turning back to the Gamma interview...

Favor object composition over class inheritance

Composition has a nicer property. The coupling is reduced by just having some smaller things you plug into something bigger... In a subclass you can make assumptions about the internal state of the superclass when the method you override is getting called. When you just plug in some behavior, then it's simpler. That's why you should favor composition.
Mr. Johnson expands:
To clarify the distinction, let's consider what we want to achieve by inheritance. Abstract inheritance enables polymorphism: the substitutability of objects with the same interface at run time. This delivers much of the value of object-oriented design... Concrete inheritance enables both polymorphism and more convenient implementation.
...Interface inheritance (that is, the implementation of interfaces, rather than inheritance of functionality from concrete classes) is much more flexible than concrete inheritance.

Robert Martin is another thought leader on core principles, has defined several core principles including the following:

[NOTE: Still under development]

The Single Responsibility Principle (SRP)

There should never be more than a single reason to change a class... Changes to one responsibility may impact other responsibilities.

Open Closed Principle (OCP)

A module should be open for extension but closed for modification
Should be able to add behavior without modifying common behavior

Liskov Substitution Principle (LSV)

Subclasses should be substitutable for their base classes

The Interface Segregation Principle (ISP)

Interfaces shouldn't have stuff that their clients don't need... Factor interfaces so that clients can use just relevant members

The Law of Demeter

Only talk to your friends who share your concerns... Never call a method on an object that you got from another call or on a global object.

http://c2.com/cgi/wiki?LawOfDemeter

http://www.ccs.neu.edu/home/lieber/LoD.html

http://c2.com/cgi/wiki?LawOfDemeterRevisited

The Dependency Inversion Principle

Modules should depend on abstractions, not concrete modules... Details [controllers?] should depend on abstractions, not the other way around.

"Inversion" in contrast to structured, procedural thinking.

Garbage Collection is still important

2005-10-10T08:58:00.000-07:00

I think in J2EE, it is easy to get away from the basics of solid Java programming, assuming that the container will take care of the hard stuff for us. Understanding how GC works, however, can be crucial for J2EE applications, expecially high volume or long running runs. I don't know about you, but I might need to brush up on this stuff.

Simon Roberts delivered a great primer on the topic to the DJUG Architecture SIG last week, we'll try to get slides up on the archive soon.

The Main Things for Software Projects

2005-09-23T08:03:00.000-07:00

The preceding is a summary slide from a deck I am using to clarify my own approach to methodology. It is a synthesis of recommendations from the CHAOS report, Alistair Cockburn, and my own experience.
Essentially, I mapped out the lists of "main things," or critical success factors, provided by CHAOS and Mr. Cockburn, and also made my own list. Then I went through an exercise of correlating and consolidating. This is where I ended up.
I think this checklist provides a concise, strategic guide for improving a project's chance of success. More useful still would be individual guides that provide tactical guidance for each of these, probably mapped against key project variables (like Alistair's team size and safety). I'll get right on this in my free time.
I also think it would be interesting to check this list against other similar lists from other modern methodologies like RUP, XP, etc. Rhythm, presented to the DJUG by Brian Boelsterli, looks like an interesting framework that might also have similar factors to consider.

Rapid Prototyping and Development--Options

2005-09-09T08:58:00.000-07:00

In this post, I continue my analysis of rapid prototyping for Java Web apps. I see a few essential ingredients necessary for rapid prototyping:

Fast and simple development of pages and page flow: must be able to easily create pages, get parameters from a submission, do business stuff, go to the next page, and present dynamic data--without manually touching 12 different files.
Automation of basic CRUD activities: Even in complex apps, so much code fits a basic CRUD template. It is extremely helpful to be able to automate the first cut at this, by specifying model information once in the database, the code, or a neutral spec.
Quick develop/run cycles: If it takes 45 seconds to see a change to the width of a text field, rapid prototyping is in trouble.
Extensibility: Whatever cool automatic stuff is done for us, we need to be able deviate from the happy path and customize stuff.
Anything else I am missing?

Are there Java technologies that allow developers to skip the enterprise stuff initially, and add it in later with a little refactoring?

JSPs with code in the pages (or Velocity templates w/out the MVC, etc.): Good for 1 & 3, but ignores 2.
IDE / Graphical tools: E.g. Java Studio Creator (is there a comparable Eclipse plugin?).
Rife framework:
Trails framework:
Model Driven tools: I looked at AndroMDA a while back, and I don't remember it striking me as a tool for rapid prototyping. But it is probably worth another look. Since I started this post, I have come across Gorilla Logic, which looks very intriguing.

Are non-Java tools, e.g. Ruby/RAILs, that allow for development that is so inexpensive that we are willing to throw the prototype away?

RAILS claims a 10x productivity advantage over J2EE, and judging by their demo it is at least that. So even if Ruby CGI is insufficient for Enterprise requirements, you can create a throw-away that is worth the effort.

To Be Continued...

Rapid Prototyping and Development--Questions

2005-09-09T08:29:00.000-07:00

I've been thinking again about rapid prototyping and RAD tools for Web development. Perhaps it is because I have been working with Enterprise apps for a few years, with their focus on hard-core non-functional requirements (security, scalability), but I don't see much going on anymore. It seems to me that J2EE, and the various Java frameworks, are so focused on robustness and maintainability that rapid prototyping is nearly impossible.

In the wild and whacky early days of Web development, it seemed that we got the functional requirements right, but then endured hell when our applications didn't perform or missed the mark on non-functional requirements. Why? Because we did very minimal requirements and immediately dived into scripting pages, essentially rapid prototyping. Now, as I see it, the Web world has inverted this situation: we have to do more requirements because it is more expensive to get working code in front of the user, but the technologies are more robust. I contend that, in general, it has become more difficult to get the functionality right, and that we now have applications that are often over-engineered.

I'd like to figure out how to re-introduce rapid prototyping into the realm of Java Web development. Here is what I think I need to figure out:

I see a few essential ingredients necessary for rapid prototyping: Fast and simple development of presentation code, automation of basic CRUD activities, and quick develop/run cycles. What else?
Are there Java technologies that allow developers to skip the enterprise stuff initially, and add it in later with a little refactoring?
Are non-Java tools, e.g. Ruby/RAILs, that allow for development that is so inexpensive that we are willing to throw it away?

Java Exception Prescription

2005-08-15T13:26:00.000-07:00

What are truly the best practices for using Exceptions in Java? Should checked exceptions be the norm? How should the Exception classes be structured? In this posting I attempt to achieve a personal approach to using Exceptions.

In Handling Exceptions from the Sun Java Tutorial, stresses the use of checked exceptions:

If a client can reasonably be expected to recover from an exception, make it a
checked exception. If a client cannot do anything to recover from the exception,
make it an unchecked exception

Sun's view is probably the excepted wisdom. But some leading Java guys are rethinking this, like Alan Griffiths, Bruce Eckle, and Rod Johnson.

Code written using checked exceptions tends to exhibit a couple of problems: catch clauses are often stubbed out for expediency, leading to "swallowed" exceptions. Also, all calling code ends up bloating with handling for errors that the caller may not want/need to handle.

Why not let callers catch what they want, otherwise let the error propogate up the chain? Rod defines the following rule of thumb from when to use a checked exception:

Should all callers handle this problem? Is the exception essentially a second return value for the method?

For Example,

Spending limit exceeded in a processInvoice() method

I seems to me right now that this is the way to go. Basically, throw checked if the caller can and should do something about it, otherwise throw unchecked.

Looking at Spring

2005-08-10T12:11:00.000-07:00

I have found that I can no longer resist the Spring Framework. Having been beguiled by its popularity and intrigued by its principals, I have decided to spend some time checking it out. The following chronicles my findings.

First, I am really interested in some of the underlying design goals, for which I turn to Chapter 4 of Rob's book, at the ServerSide. Spring aside, this is just a great essay on OO design.

So far I am impressed. Dependency Injection is just too sensible, and makes TDD much more practical.

To Be Continued...

Shameless Promotion 3: John in a Nutshell

2005-08-08T10:39:00.000-07:00

I am a software development expert who can contribute directly to successful application development, and also make broader contributions to an organization's success:

I know how to develop software with maximum value for any timeframe and budget.
I am a versatile professional who can fulfill many roles... roles often requiring several resources.
I am a highly productive contributor, I can get solutions done quickly.

I have top notch software development skills--coding expertise in Java and experience in C, Perl, VB, and Ruby; architecture, analysis, and design skills; and competence with a host of acronyms (UML, J2EE, SOAP, SQL, TDD, Ajax etc.)

Hands-on development is my foundation, but I have also worked hard to hone strong business skills: I have an MBA, several years of line and technical project management experience, and polished communication skills. I have led technical teams of 10 people, departments of around 20, and have countless hours interacting with Clients at all levels.

Encompassing both my technical and business skill sets is an overarching expertise in software development. This expertise is what enables me to deliver maximum value for any timeframe and budget. My unique combination of skills allows me to add strategic thinking and decision-making on top of a concrete, in-depth understanding of technology. This combination is also what enables me to fulfill so many roles; from developer to architect to resource manager and more.

My Resume and generic Cover Letter describe my skills in much more detail.

In terms of full-time positions, my mix of skills probably best matches a Principal Engineer, Architect, or Lead role with a small software company or consultancy. I am also very open to freelance work, where I can deliver on projects of all sizes. On top of software design and development, I can fulfill many of the crucial needs of a successful project:

Planning, staffing, defining team structure and roles, scoping, defining development process and tools, prototyping, and architecture.

As a consultant, I must always keep an eye out for the next gig. I'd love to discuss potential opportunities, whether in the planning stage or thereafter, Lead, freelancer, or other key role. If you are interested in someone like me, drop me a line.

- John Troxel

Denver, Colorado

jtroxel@yahoo.com

Shameless Promotion 2: John's Resume

2005-08-08T10:15:00.000-07:00

John D. Troxel
Wheat Ridge, CO | 303.355.1121 | jtroxel@yahoo.com

Objectives

Opportunities to help companies outperform their competition with pragmatic software solutions.
To deliver effective business solutions using my unique software engineering expertise.
To provide the leadership in analysis, architecture, design, and engineering necessary to successfully get software to market.

Skills Summary

BS Computer Science, MBA / MIS with over 15 years of software development experience, specializing in Web-based applications
Programming: Java, Javascript, Groovy, Ruby, C, Perl, JSP, Visual Basic, Rails, Grails, AJAX, STRUTS, Spring, Maven, MySql, Oracle, MS SQL Server, JBoss, Tomcat
Software engineering methodologies, processes, and tools: UML, Requirements Analysis and Management, Agile methods, Unified Process (RUP). Object design (OOAD), system and software architecture, Open Source, SCEA, SCJP, Rational Certified Consultant RUP/OOAD
Technical line and project management, organization design, team mentoring

Professional Experience

Rearden Commerce, Foster City, California

Principal Software Engineer -- March 2011 – Present

Key architect and developer for content integration tools using Spring Batch, Camel, and custom tools.
Lead developer for a custom content curation system in Grails.

Senior Software Engineer -- January 2008 – March 2011

Senior contributor to common Ajax and UI components.
Develop Endeca faceted search applications and data-processing tools.
Enhance in-house MVC framework.
Contribute to design and development of new UI framework using Spring MVC.

Freelance Software Developer & Architect -- June 2006 – December 2008

Web2 Impact, LLC, Denver, Colorado

Lead all development efforts for BuyBak.com--an eCommerce startup providing reverse logistics capabilities--using Java, Struts, Spring, iBatis, Grails, as well as Ruby and MySQL.
Developed the system architecture and design (UML domain model, site map, etc.) for start-up healthcare service.
Developed Ruby on Rails, Javascript, and Java code to create a digital video system inside a Web-based, Ajax-style email system (Zimbra).

Sr. Software Engineer -- March 2005 – June 2006

McKesson (System Engineering Services), Denver, Colorado

Java software design and development for mission critical applications.

Development key internal application using Java, Tomcat, XSLT, and AJAX
Development of all Web Service interfaces to the core system using JbossWS
Development of core components using JMX on JBoss

Sr. Java Developer; Sr. Systems Analyst -- March 2004 – March 2005

Veteran’s Administration (Express Management Solutions), Denver, Colorado

Major contributor to critical systems development for the Health care Administration Center.

Leading establishment of System Analysis discipline and spearheading analysis of major initiative
Key team contributor for software architecture and programming for Oracle 9i and STRUTS

Developer; Product Manager -- June 2002 – March 2004

Grizella Corporation, Lakewood, Colorado

Developer of Web-based products for a Transportation software company.

Architecture and programming for service-oriented system using Tomcat/Java Servlets/JSP
Developed an e-commerce ordering application with a credit card gateway interface
Created Web application to effectively utilize resources for a major Trucking association
Created a Web UI framework (MVC) for rapid development of Web-based services

Technical Consultant -- February 2001 – May 2002
Experio Solutions, Denver, Colorado

Manager of technical project teams.

Project Manager for a critical process re-engineering of the IT department of a major utility
Developed components and workflows for a business service model using SeeBeyond
Contributed to J2EE and Enterprise Application Integration interest groups
Attended TIBCO training and maintained a Web-based InConcert demo

Operations Director; Process Improvement Lead -- June 1998 – January 2001
iXL, Denver, Colorado

Various Management roles for Denver office of iXL.

Managed departments of up to 18 reports: Technical Services, Quality Services, Requirements Management
Successfully implemented improved Requirements Management and Change Request Management processes as Chair of Denver office best practices taskforce
Defined the corporate methodology framework utilizing RUP principles as key member of Corporate Methodology Team
Provided guidance to teams on program and project best practices

Technical Director; Technical Lead -- October 1998 – June 2000
iXL, Denver, Colorado

Leader of technical development group for a top Internet-based business solutions company.

Managed technical teams of up to 10 developers, from requirements through deployment, for Budget Rent-a-Car, National Processing Corporation, and American Water Works Association
Lead Architect and Developer for ServletExec/NES/Oracle and IIS/SQL Server/ASP projects
Created and managed Technical Requirements and Architecture documents using Use-case specifications and realizations, sequence diagrams, and Deployment diagrams
Technical Project Management and primary technical interface with high-level Client management and vendor partners

Director, Engineering Services -- August 1996 – June 1998
iXL (Green Room Productions), Sausalito, California

Leader of Web application development for a successful Web development company.

Established and managed the Engineering Services team, that grew to 17 employees
Internet application projects for travel clients including Sheraton Hotels, Walt Disney World, Travelocity, Mountain Travel Sobek, and Abercrombie and Kent
Developed interactive Web applications using Perl, Server-Side JavaScript (SSJS), Oracle, ASP, and MS SQL Server
Led development of Web-based Workflow system, including project, assignment, and hours tracking

Systems Engineer (Part Time) -- May 1995 – March 1996
Visual Movement Corporation, Denver, Colorado

Internship with a software development team specializing in the transportation industry.

Technology selection and market research, implementation of installation utility in Visual Basic

Associate Engineer -- June 1991 – August 1994
Cooper & Chyan Technology, Inc., Cupertino, California

Integral member of software engineering team for a start-up CAD software company.

Directed the design and content of a successful new GUI for an advanced CAD system
Maintained and enhanced a run-time graphics library using C, Motif, and X-Windows. This library was used to build the GUI listed above

Education
MBA / MIS. University of Denver (Daniels College of Business) June 1996
· GPA: 3.51 / 4.00
· Member of Information Technology Association and charter member of Toastmasters

B.S. Computer Science. Colorado State University May 1991
· Active member of Sigma Alpha Epsilon Fraternity. Vice President, Alumni Chairman
Certifications
· Sun Certified Enterprise Architect for J2EE, November 2003
· Sun Certified Java Programmer, December 2001
· Rational Certified Consultant in RUP and OOAD, March 2002

Shameless Promotion 1: John's "Cover Letter"

2005-08-08T09:54:00.000-07:00

I am a professional software engineer and manager, and my expertise in software development can improve the success of software projects. I have over 11 years experience in software development as a leader and a key contributor in several roles.

I am a leader who can code, and a developer who can manage people and projects. I believe I have a truly unique breadth of skills that make me an indispensable asset to software development projects:

Technical lead, architect, and developer on dozens of projects, including large engagements for clients like Sheraton Hotels, Travelocity, Budget Rent-A-Car, and the Veteran's Administration
Sun Certified Architect for J2EE
Programming experience in Java (4 years), C (3), VB/ASP (3), and Perl (2)
Sun Certified Java Programmer
Effective, hands-on skills as a software engineer, architect, and business/systems analyst
Several years of experience as an engineering line manager and as a technical project manager
Rational Certified Consultant in RUP and OOAD
MBA and a Bachelor's in Computer Science

Please let me know if you would like a current resume, or to discuss how I can help you to succeed. Sincerely,

John Troxel

303.667.3280

jtroxel@yahoo.com

Unit Testing Thoughts

2005-07-28T08:49:00.000-07:00

In no particular order and with absolutely no structure, some thoughts on unit testing:

In order to properly unit test, your design must properly seperate concerns. This means that true business logic should be decoupled from specific APIs, conventions, and protocols... and these from eachother. For example, you shouldn't have the following mixed into business logic: SQL statements, JDBC calls, J2EE API objects, etc.
There is a spectrum from "pure" unit testing to complete integration testing, something like:

(Pure) Unit Tests: Have no dependencies and no side effects: test constructors, accessors, comparisons, algorithms. Generally this consists of a single client, the test class, and a single class under test. Developed by developers for developers.
Partial Integration / Integration Boundary Tests? Tests integration between layers, components, or technologies. Where possible, tests should focus on a single boundary, e.g. data access code <-> Database, SOAP Client code <-> SOAP Service, etc. Code that has dependencies on other APIs should use mock objects or some sort of debug-dummy mode. For the debug mode strategy, the property could be "injected" by the runtime invironment with a settor or a special constructor. These tests are also by developers for developers.
Full Integration / System Tests? These test a path through the system that provides verification of some subset of requirements. These tests can be employed by developers, using tools like HTTPUnit. They can also be maintained by testers or properly trained business experts, using functional test automation tools.

Many automated tests, including some unit tests, have side effects on the data that can mess up other tests. For example, tests may rely on a pre-condition state of the database to succeed, and manual testing can be maddening if the data is changing underneath you. Either all tests must mutate data completely independent of any other test or user, or the database must be restored to a consistent state before testing (which still might mess up other testers). Some strategies for managing this situation include:

Complete database restore, in an isolated database or schema. E.g. SQL DDL scripts, dbUnit scripts, or programmatically. If the developer's database is a different technology or version from production, this introduces the challenge of maintaining a restore capability for two technologies, as well as potential bugs caused by differences in the production database. To test for problems caused by a different technology in production, the team could use a shared database instance specifically for periodic execution of all unit tests. You could also have multiple levels of restore, e.g. a bare bones one for unit testing, one that adds reference data, and one that puts some example data in for manual testing. I like the idea of using Hibernate (or something similar) to abstract the db technology, using HSQL (or similar) locally, and providing utility methods to add in the required data.
Transactions / rollback? This implies that data access code must have mechanisms for using injected transactions, managed by the test code, and/or skipping commits in test mode. Add setAutoCommit, commit, rollback to DAO API?
Have each test clean up it's data. This requires that 1) all data created can be identified or tracked by the test, and 2) that the database allows the proper permissions to delete data.

To Be Continued...

Qualifying the "Simplest Thing" rule

2005-07-26T08:39:00.000-07:00

In XP, and Agile methods, decisions are often made with the criteria of "the simplest thing that could possibly work." I think this applies not only to code construction and detailed design decisions, but to Architecture as well. Strict use of this rule will almost always result in the most prudent approach, but I do think there are a couple of caveats:

The decision should still be reversible, so that you don't paint yourself into a corner for future changes in the requirements.
The simplicity should be weighed carefully against the flexibility and the cost of changing later. If the simplest solution costs 20% less now, but will likely cost 30% more over time, then it may not be the best choice. You have to decide how likely likely is.

I would stress that even with these caveats, one should still be very careful breaking the rule. If you don't have a strong case for a very real risk of ir-reversibility problems or future costs, you should just stick with the simplest solution.

JBoss CMP versus Hibernate

2005-07-08T13:26:00.000-07:00

UPDATE 07/26/05
I stuck with Hibernate. Very simply put, they both have many of the same benefits--declarative programming, transactions, etc. But EJB introduces a whole layer of overhead realted to remote access. Unless you are doing a true N-tier, highly distributed application, I don't think that overhead is warranted.
=======

I am starting to look at a persistence technologies for an application I am working on in JBoss 4.0.2. I am currently looking at the (built in) Hibernate in JBoss versus CMP. SO far I am just kicking the tires, gathering info. If you come across this, please feel free to comment.

Harshad Oak has a posting on the topic with comments.

Right now, I can't see any reason why not to use Hibernate: I can use pojos, but still get transactions. I don't see the benefit of creating EJBs for every entity we need to persist.

UML: Labels versus Stereotypes for Associations and Dependencies

2005-05-05T15:18:00.000-07:00

I think I am clear on what to use where for Associations: The label is very specific, identifying the association or relationship between the two specific nodes. A stereotype doesn't normally make sense for associations, since it identifies a kind of association. Examples I have seem is <<include>> for use cases, and identification of the protocol for component associations.

For Dependencies: A label often does not make sense, since the relationship is always "depends on." A stereotype further defines the nature of the dependency.

Good UML Links

2005-05-05T14:50:00.000-07:00

I really like Scott Ambler's stuff at: http://www.agilemodeling.com/style/

This looks like a solid overview: http://www.vico.org/UML_FAQ_jGuru.pdf

Looks interesting: http://www-inf.int-evry.fr/COURS/UML/semantics/

And of course: http://www.uml.org/#UML2.0

Hmmm: http://www.holub.com/goodies/uml/

Thoughts on Requirements Terminology

2005-05-04T13:49:00.000-07:00

It occurred to me a while back that there are two distinct categories of requirements: Those that are deemed necessary and can be categorically verified, and those that are more lofty and more difficult to test.

I suggest that the first set are called Imperative Requirements, or simply requirements. These are binary in nature, either they make it into the system or they don't--a test case can be written that will pass or fail. Use cases and functional requirements fall into this category, as well as performance, reliability, security, and perhaps other non-functional areas as well.

The second category, which I'll call Non-Imperative Requirements (or maybe just Qualities), are different. They can not be tested or verified in any straight forward way, but rather are qualities to shoot for. Scalability, maintainability, useability, and ease-of-development, for example, fall into this category. These Qualities are better expressed as goals with relative importance. Those less verifiable, these need not be completely useless. Qualities can be expressed with relative importance weights, and still traced with a correlation matrix.

In future posts, I plan on fleshing out my idea of a requirements model, especially regarding how requirements should be systematically addressed in architecture.

2005-04-21T14:16:00.000-07:00

If you ever get a java.lang.NoClassDefFoundError at runtime, and you are certain the class identified is in your classpath, take a look at the inheritence hierarchy. The java.lang.ClassLoader will identify the child, even if it is a parent who is missing. At least that is what happened to me.

JBoss EJB Security

2005-03-25T13:59:00.000-08:00

In addition to the well-documented steps for securing EJBs, configuring in JBoss has some potential issues. Setting up an authentication policy requires editing some JBoss-specific files:

conf/login-config.xml:
<application-policy name="policy-name">
<authentication>
<login-module code="org.jboss.security.auth.spi.UsersRolesLoginModule" flag="optional">
<module-option name="unauthenticatedIdentity">anonymous</module-option>
</login-module>
</authentication>
</application-policy>

META-INF/jboss.xml:
<jboss>
<security-domain>java:/jaas/policy-name</security-domain>
...

If you set up a policy like this, you apparently must establish a method-permission in ejb-jar.xml for every method (or *) of every bean. Misnaming a method, or trying to do a partial wildcard (get*, can't do that), gives an error:

Insufficient method permissions, principal=user, ejbName=Bean, method=method, interface=SERVICE_ENDPOINT, requiredRoles=[], principalRoles=[user-role1, user-role2]