Programming user interfaces has many challenges. Fetching the data from remote service, populating the UI controls, tracking user input against the predefined set of validation rules, persisting the data back to the server, handling transitions between different application screens, supporting offline mode, handling error cases in the communication layer. All of these are just part of our daily programming tasks. While all of these relate directly to what the end user sees on the screen, there is a small subset that greatly affects the overall user experience.

The main two items in this subset are pixel perfection and application responsiveness. It is important to remember that the end user does not care about all of the wonderful underlying layers that take up a larger part of our day. As bad as analogies usually are, i would compare it to the vast majority of the drivers. I personally do not care how many people have worked on the specific model, how intricate are the internals of the engine and how many technical challenges had to be overcome during the development and testing. All i care about is that when i turn the key, the engine starts, when i turn the wheel / press the brake / turn on the AC, it responds correctly and immediately, and that it feels good to be sitting behind the wheel or in the passenger seat.

It is thus rather unfortunate that a lot of user interfaces are not implemented with these two main goals in mind. It would seem that programmers would rather enjoy letting the implementation complexity leaking into the UI layer – if it was hard to develop, it should at least look like that on the screen. There are multiple factors contributing to this. The farther you are from the target audience, the less you are able to judge the usability of the application, especially when you are immersed in its internals for long periods of time. When you operate at the level of data objects (with perhaps direct mapping to the optimized backend data storage), the users don’t. They see your application as a means to accomplish the specific flow that they have in mind. If you fail to crystallize the flows during the design stage, your users will see your application as unintuitive, time wasting and counter productive.

And even if you get the flows right – at least for your target audience – there is the issue of responsiveness. Imagine the following scenario. You’re in the kitchen, and want to heat that last slice of pizza from yesterday’s party. You put it in the microwave, press the button – and the entire kitchen freezes for one whole minute. You can look at the fridge, but you cannot open it. You remember that you pressed the microwave button, but the tray is not spinning and it does not make any noises. You step in your living room, and are not able to get back into the kitchen.

This is what happens when your code does I/O, database or any network operation on the UI thread. It does not really matter how small the operation is or how fast your machine is. If it locks the UI for even a fraction of a second, people will notice. You can say – well, this operation here that I’m doing is persisting the current screen state to the backend, so i cannot really let the user interact with the UI while i’m doing that. Or maybe it’s just not that important and the user can wait. Or maybe it’s just going to add to the development complexity of the UI layer.

Doing this right is hard. First, you need to understand what is right – and that varies depending on the specific scenario. Do you prevent the user from doing anything with the application? Or maybe let him browse some subparts of the screen while you’re running this long operation? Or maybe even let him change information elsewhere in the application? Do you show the progress of that operation? Do you allow canceling the operation? Do you allow creating dependencies between executing / scheduled operations?

And after you know what you want to do, the next step is actually implementing and testing it. At this point the word “multi-threading” will be your friend and nemesis. I cannot say that we are at a stage where doing multi-threading in UI is easy (although i certainly am not an expert in all the modern UI toolkits and libraries). It is certainly easier than it was a few years ago, but it’s still a mess. The words of Graham Hamilton from five years ago are still true today:

I believe you can program successfully with multi-threaded GUI toolkits if the toolkit is very carefully designed; if the toolkit exposes its locking methodology in gory detail; if you are very smart, very careful, and have a global understanding of the whole structure of the toolkit. If you get one of these things slightly wrong, things will mostly work, but you will get occasional hangs (due to deadlocks) or glitches (due to races). This multithreaded approach works best for people who have been intimately involved in the design of the toolkit.

Unfortunately I don’t think this set of characteristics scale to widespread commercial use. What you tend to end up with is normal smart programmers building apps that don’t quite work reliably for reasons that are not at all obvious. So the authors get very disgruntled and frustrated and use bad words on the poor innocent toolkit. (Like me when I first started using AWT. Sorry!)

I believe that any help we can get in writing correct multi-threaded code that deals with UI is welcome. This is why i continue enforcing the Swing threading rules in Substance. It is by far the biggest source of complaints ever since it was introduced about a year and a half ago in version 5.0. The original blog entry on the subject implied – rather unfortunately – that i wanted to make my job easier and not handle bugs that originate from UI threading violations. Allow me to clarify my position on the subject and repost my reply from one of Substance forum postings:

I do not intend to provide such API (disabling the threading checks) in the core Substance library. Once such an API exists, people will have little to no incentive to make their code more robust and compliant with the toolkit threading rules.

If the code you control violates the threading rules – and you *know* it – you should fix it. Does not matter if you’re using Substance or not.

If the code you do not control violates the threading rules – either put pressure on the respective developers to change it or stop using it.

It may be painful in the short term. I may lose potential users because of this. It may cause internal forks of the code base. I am aware of these issues. In my personal view, all of them are dwarfed by what is right in the long term interests of both Substance itself and Swing applications in general.

The followup posting by Adam Armistead provides a deeper look into why this matters – and i thank Adam for allowing me to repost it here:

I would just like to say I strongly support Kirill in this and I am very glad to see he is sticking to his guns on this. I feel that it is too easy to violate threading rules in UI code and that I run across entirely too much code that does. I feel that if someone doesn’t strictly enforce these threading rules then there is not enough pressure on developers to fix the problem. If there was enough pressure I wouldn’t see so damn much of this.

As for it causing problems due to 3rd party dependencies having bad UI code, there are tons of solutions for this. I have personally put pressure on developers to fix problems, submitted patches to open source projects to fix code myself, extended classes, forked codebases, used bytecode manipulation, proxy classes, and Spring and AspectJ method injection, method replacement, as well as adding before/after/around advice to methods. I have yet to encounter a situation where a little work or ingenuity on my part has not been able to overcome someone else’s crappy code. In the worst cases I have written my own libraries that don’t suck, but in most cases I didn’t have to go to this extreme.

I sincerely believe that having substance crash my applications has made MY applications as well as those I interact with better. I have seen people in comment sections in blog posts and on forums advise using Substance in order to assist developers in finding UI threading violations. I have fixed open source code and seen others do the same because Substance throws this exception. I can also say, I know may coders that if they had the choice to just log it, they would do so and just say, “aww, I’m busy, I’ll fix it later.” and likely never get around to it…

They’re always busy, so when one thing gets done there’s always half a dozen more that pop up. If it’s not crashing their code its not critical to fix. Besides, its too easy to violate these rules. I’m a Swing developer and I know the rules, and sometimes when I’m hammering out features I slip up and do things incorrectly. Personally, I am glad Substance catches it so I can fix it now.

What are the chances that Swing will start enforcing threading rules in the core? Zero. Between focusing all the client-side resources on JavaFX and sticking to binary compatibility (even when any reasonably sized Swing application needs code changes when it is ported between different JDKs), there is no chance. However, as SWT and Android show, a toolkit / library that aggressively enforces its own threading rules from the very beginning is not such a bad thing. And who knows, your users may even thank you for it.

The core Substance look-and-feel library is in the feature freeze state, and it’s time to tie the loose ends. Over the next couple of weeks i will be updating documentation, fixing bugs and perhaps adding a few new skins. Along with that, i will polish the visuals of third-party components covered by the Substance plugins that i maintain, and today it’s time to talk about the date picker component from SwingX.

The ever-changing plans for Swing in JDK 7 (which went from grandiose to acceptable to almost non-existent) at one time included a new date picker component available in the core Swing library. While the fate of such a component has not been addressed in the five-hour press conference today, there are a number of open-source and commercial date picker components available for use in Swing applications. One of them comes from SwingX – a library that is supported by the matching Substance SwingX plugin.

The latest 6.0dev drop of Substance SwingX plugin aligns the visuals of the JXDatePicker component with those of editable comboboxes. The inset look, double border and flat button that responds to the mouse rollover – all of these add up to a consistent and polished appearance of your UI running under Substance 6.0. Let’s see a few screenshots.

Here is the date picker component under the Business skin in the latest 6.0dev drop:

Here, you can see four date pickers in different areas of the application – toolbar, sidebar, status bar and the general area. While under the Business sin there is not much visual difference, it becomes more evident under the Business Blue Steel skin:

Here, the toolbar has a darker shade of blue used on its controls, and sidebar / status bar use a lighter shade of blue. Notice how the component automatically picks up the colors for the background and the border based on the decoration area that hosts it.

Next up the Gemini skin:

Here, note how the date picker in the toolbar has picked up not only the lighter color for the border, but also the correct light color for the foreground and the arrow icon. The same can be seen under the Magellan skin:

with date pickers in different decoration areas picking the matching foreground / background colors. Finally, the date pickers under the Dust skin:

You are more than welcome to take the latest 6.0dev drops of Substance and its plugins for a spin and report bugs / visual artifacts to the project mailing lists or forums:

• The latest 6.0dev drop of the core library (code-named Sonoma)
• The latest 6.0dev drop of Substance SwingX plugin aligned with the SwingX daily drop #2283 from January 25th.

Today i’m going to talk about the last big change that went into the next release of Substance look-and-feel – enhanced support for component states.

What is a component state? Let’s take a look at buttons – the most basic building blocks on any UI toolkit. Buttons usually have icons and texts so that the users know what will happen when they click them. In addition, modern UI toolkits provide rich texturing capabilities that allow skinning different parts of the button visuals – such as background, border and focus ring. These visuals usually depend on the current state of the button.

If a button does not respond to UI events (such as mouse click, for instance), it is said to be disabled – and usually has a lighter or partially translucent appearance to match its disabled state. A toggle button can be in a selected state, conveying that certain application parameter is toggled on. A selected button is usually painted with different hue to facilitate quick scanning of the application state. There are more button states – rollover when the mouse is moved over it, pressed when the user pressed but has not yet released the mouse and default button which is invoked when the user presses the Enter key.

Different controls have different states. For example, a progress bar can be determinate or indeterminate – depending on whether the application code can reliably assess the overall length of the work to be performed. A text component can be editable or uneditable. Input controls can be marked as required – to prevent the user from leaving the screen until he makes a selection on that control.

Component states in Substance are manager by the org.pushingpixels.substance.api.ComponentState class. Instances of this class correspond to states of Swing core and custom controls. This class provides a number of predefined static instances to cover most action-based controls such as buttons, check boxes and menu items. In addition, application code can define custom component states that create fine grained mapping between arbitrary states of controls and specific color scheme bundles in custom skins.

Each component state is defined by two arrays of component state facets (available in ComponentStateFacet class). The first array specifies the facets that are on, and the second array specifies the facets that are off. For example, when a selected toggle button is pressed, it transitions to PRESSED_SELECTED state. This state has ComponentStateFacet.ENABLE, ComponentStateFacet.SELECTION and ComponentStateFacet.PRESS as its on facets. If a selected toggle button is disabled, it has ComponentStateFacet.SELECTION in its on facets and ComponentStateFacet.ENABLE in its off facets.

The ComponentStateFacet class defines a number of core facets. The ComponentStateFacet.ENABLE facet is universal – it is relevant for all Swing controls. Other facets apply to a wider range of controls. For example, ComponentStateFacet.ROLLOVER facet applies to all controls that can show rollover effects – including buttons, menu items, comboboxes, sliders, scrollbars and many more. Some facets apply to a very narrow range of controls. For exaple, ComponentStateFacet.EDITABLE is only relevant for editable controls, such as text components, editable comboboxes or spinners.

The static instances of ComponentState defined in this class do not aim to cover all possible combinations of on and off facets. In addition to making this class to unwieldy, it is not possible to do since application code can define its own facets. Instead, Substance provides three ways to fine-tune the mapping between the component states and the color schemes used to paint the components.

1. When the skin is queried for the color scheme that matches the specific component state – let’s say ComponentState.PRESSED_SELECTED – the skinning layer first looks for the exact state (as passed to SubstanceColorSchemeBundle.registerColorScheme(SubstanceColorScheme, ColorSchemeAssociationKind, ComponentState) or similar APIs). If the exact match is found, it is used. If there is no exact match, the skinning layer will look at all color schemes registered for the specific color scheme association kind in the matching color scheme bundle. The decision is made based on how “close” the registered component state is to the component state of the currently painted component. For example, ComponentState.PRESSED_SELECTED is a better match for ComponentState.PRESSED_UNSELECTED than ComponentState.ROLLOVER_SELECTED – since the ComponentStateFacet.PRESS has more weight than the ComponentStateFacet.ROLLOVER in the decision process. The skinning layer will choose the “closest” registered component state that is sufficiently close. For example, ComponentState.DISABLED_SELECTED will never be chosen for ComponentState.SELECTED, even if there are no other registered component states. This way the application code can register a few color schemes in the specific bundle, and have all other states “fall back” to the smaller subset of states.
2. Facets such as ComponentStateFacet.DETERMINATE or ComponentStateFacet.EDITABLE are relevant only for a small subset of controls. In order to simplify the API signature of ComponentState, these facets are not part of any of the predefined static states in this class. Instead, they are used internally in the matching UI delegates (such as for progress bar or text components) to find the best match among all the registered states of the current skin. The specific skin can define its own ComponentState instances that use these facets. For example, NebulaSkin defines a number of component states that use the ComponentStateFacet.DETERMINATE facet, and maps the matching color schemes. At runtime, the procedure described in the previous item will match the state of the specific progress bar to the states defined in this skin, and use the matching color schemes.
3. Custom application components may have facets that do not directly map to the core facets defined in the ComponentStateFacet class. In this case, the application code can create its own facet instances, and its own component states that use those facets in the on and off lists. Part of the custom code will be in the UI delegates that compute the current state of the custom component using the new facets. Other part of the custom code will be in the skin definition that maps the component states defined with the new facets to the specific color schemes.

Note that you do not have to create explicit dependency between custom component states used in the skin definition and custom component states used in the painting routines (in the UI delegates). In fact, the custom component states defined in the Substance UI delegate for progress bar are not accessible to the application code. The recommended way to separate the skin definition from the model lookups in the painting is:

The skin definition defines a sufficiently broad set of custom component states that use the new facets. Note that you do not have to create a custom state for every possible permutation of new facets (along with the relevant core facets). A well defined set of component states will provide a good fallback state for every relevant permutation of facets, keeping the skin definition small and manageable.
The UI delegate that queries the component model will use accurate component states that account for all the relevant on and off facets – including the core facets defined in the ComponentStateFacet class. When this (perhaps elaborate) state is passed to SubstanceColorSchemeBundle.getColorScheme(ColorSchemeAssociationKind, ComponentState) API, the the procedure described above will match the this state to one of the “base” states defined in your skin, and use the matching color scheme.
Note that the matching algorithm only looks at the facets in the on and off lists, and ignores the component state name. This allows you to create a broad component state in your skin, and a number of narrow component states during the painting – and have the Substance skinning layer find the best match.

When the matching algorithm cannot find a sufficiently close match, the skinning layer will fall back on one of the three base color schemes passed to the SubstanceColorSchemeBundle.SubstanceColorSchemeBundle(SubstanceColorScheme, SubstanceColorScheme, SubstanceColorScheme) constructor. States with ComponentStateFacet.ENABLE in their off list will fall back to the disabled color scheme. The ComponentState.ENABLED will fall back to the enabled color scheme. The rest of the states will fall back to the active color scheme. To change the fallback behavior pass a non-null fallback color scheme to the ComponentState.ComponentState(String, ComponentState, ComponentStateFacet[], ComponentStateFacet[]) constructor as the second parameter.

Let’s see two examples. As mentioned above, the Nebula skin defines custom color schemes for progress bars using the ComponentStateFacet.DETERMINATE:

ComponentState determinateState = new ComponentState("determinate",
	new ComponentStateFacet[] { ComponentStateFacet.ENABLE,
			ComponentStateFacet.DETERMINATE }, null);
ComponentState indeterminateState = new ComponentState("indeterminate",
	new ComponentStateFacet[] { ComponentStateFacet.ENABLE },
	new ComponentStateFacet[] { ComponentStateFacet.DETERMINATE });
SubstanceColorScheme determinateScheme = schemes
	.get("Nebula Determinate");
SubstanceColorScheme determinateBorderScheme = schemes
	.get("Nebula Determinate Border");
defaultSchemeBundle.registerColorScheme(determinateScheme,
	determinateState, indeterminateState);
defaultSchemeBundle.registerColorScheme(determinateBorderScheme,
	ColorSchemeAssociationKind.BORDER, determinateState,
	indeterminateState);

ComponentState determinateDisabledState = new ComponentState(
	"determinate disabled",
	new ComponentStateFacet[] { ComponentStateFacet.DETERMINATE },
	new ComponentStateFacet[] { ComponentStateFacet.ENABLE });
ComponentState indeterminateDisabledState = new ComponentState(
	"indeterminate disabled", null, new ComponentStateFacet[] {
			ComponentStateFacet.ENABLE,
			ComponentStateFacet.DETERMINATE });
SubstanceColorScheme determinateDisabledScheme = schemes
	.get("Nebula Determinate Disabled");
SubstanceColorScheme determinateDisabledBorderScheme = schemes
	.get("Nebula Determinate Disabled Border");
defaultSchemeBundle.registerColorScheme(determinateDisabledScheme,
	determinateDisabledState, indeterminateDisabledState);
defaultSchemeBundle.registerColorScheme(
	determinateDisabledBorderScheme,
	ColorSchemeAssociationKind.BORDER, determinateDisabledState,
	indeterminateDisabledState);

And the resulting visuals – note that the progress bars use brown color scheme, while all the other controls use gray colors:

The second example is from the Magellan skin that registers a specific color scheme to be used for painting uneditable text components:

ComponentState uneditable = new ComponentState("uneditable",
	new ComponentStateFacet[] { ComponentStateFacet.ENABLE },
	new ComponentStateFacet[] { ComponentStateFacet.EDITABLE });
SubstanceColorScheme uneditableControls = colorSchemes
	.get("Magellan Uneditable Controls");
defaultColorSchemeBundle.registerColorScheme(uneditableControls,
	ColorSchemeAssociationKind.FILL, uneditable);

Note that if your custom components use the DETERMINATE or EDITABLE facets in the computation of their states, they will get the matching visuals from the corresponding core Substance skins.

The animation series that was published on this blog last week has been largely the product of reworking the animation layer in Substance look-and-feel and replacing it with the Trident animation library. This work has some implications for the users of both library, and today i’m going to talk about those.

If you’re using Substance look-and-feel library in your applications, you will need to add the matching Trident jar to your classpath – starting from release 6.0 of Substance. The matching Trident version is 1.2 and it will be officially released at the same time with Substance 6.0. While this is not a major Trident release, it does remove deprecated APIs and as such will break applications that are using those APIs. All the removed APIs have direct replacements, and the final release notes will provide additional information (if you cannot find it in the code). The final Trident 1.2 / Substance 6.0 releases are scheduled for March-April 2010 timeframe.

Substance 6.0 breaks API signatures of most published painter interfaces. The painter interfaces that received two color schemes and the cycle position now only receive a single color scheme. If you are just using Substance as the application look-and-feel, you should not need to worry. If you are using Substance painter APIs to create consistent visuals for your custom / 3rd party components, you will need to change the code. If you have passed the same color scheme to a painter API call, changing your code is simple. If you passed two different color schemes, you will need to call the Substance API twice, changing the graphics composite to match the value of the cycle position.

Breaking the painter APIs is a necessary step to enable multi-state color transitions discussed at length in this series. The old APIs assumed that animating a control always involves two states – previous and current. This assumption is not correct. Suppose your buttons are painted with light blue color. When the mouse is over a button, the button is painted with light yellow color, and when the button is pressed, it is painted with saturated orange. Substance animates the button colors based on the state transitions. Suppose it takes 500ms to complete a single animation. The user moves the mouse over a button, and Substance starts animating the color from light blue to light yellow. Halfway through the animation (250ms), the user presses the button. Now, there are three states participating in the animation: default with light blue, rollover with light yellow and pressed with saturated orange. All the states contribute to the overall appearance of the button as long as the combined animation is in progress.

Tracking state transitions is done internally in Substance – in a layer built on top of Trident base timeline APIs. The tracking layer is not going to be part of Trident 1.2, since it is closely tied not only to Swing classes, but to Substance skinning model. Applications interested in adopting multi-state transitions should not use the internal state tracker layer of Substance. It is subject to change at any point in time. The base rules for multi-state transitions are derived from the examples in this series that have shown different movement paths of a physical object between three points.

The current implementation of the multi-state transitions in Substance 6.0dev does not model most of the physical laws discussed in this series. Handling momentum / inertia, direction change involving smooth turns, and the matching velocity models are not implemented. The work on this will continue throughout 2010, and some of it might find its way to the Trident itself. I am not aware of any other animation library (Java based or otherwise) that provides out-of-the-box support even for simple animations based on the physical rules discussed in this series.

Substance animations deal exclusively with colors. The current straight-line / straight-turn / no-momentum movement paths in the RGB color space result in visually consistent and smooth animations, for any reasonable durations. However, the long term goal for both Trident and Substance is to create mathematically correct animation model which is based on the rules of the physical world – where applicable. At the present moment the mathematics behind the graphs in this series is left as an exercise to the readers.