Functional testing of UIs is an important and often neglected way of ensuring quality and prevent regression. The Flex world of functional tests seems at the very beginning. We evaluated some of the tools available and used the following criterias:
- OS independence
the tool and the created test scripts should run on at least every platform the Flex SDK and the Flash platform are available - Tool changes
how much we need to change or adapt the tool to suit our needs - Code pollution
how much the actual code needs to be polluted to support this testing tool - Capture/Replay
the tool needs at least an option to capture and replay test scripts - Additional License Costs
if we need to pay additional (besides the tool) license costs for things like the FlexBuilder Pro
| OS ind. | Tool changes | Code pollution | Capture / Replay | Add. costs | |
| Automation based tools | + | – | 0 | + | – |
| SeleniumFlex | + | 0 | + | 0 | + |
| FunFx | – | 0 | + | – | 0 |
| Fluorida | + | – | + | – | + |
Automation based tools (like FlexMonkey, QTP and RIATest) use the Flex automation API and have additional costs for FlexBuilderPro (700$ per license). For custom components you have to add automation code to them (pollution) and introduce them and their events in FlexMonkeys event map (tool changes).
SeleniumFlex uses the JavascriptBridge (ExternalInterface) of the FlashPlayer and needs you to add the custom components and events to this external interface which resides in the tool/test code (therefore a 0 at tool changes). You can use the Selenium plugin for spy (the ids)/replay but the capture option isn’t working yet (0 for capture/replay).
FunFx also uses the ExternalInterface and is written in Ruby but runs only on Windows (- for OS independence) because it connects to the Flex application via Win32OLE. I found no capture/replay (-) and the website says you need FlexBuilder (I don’t know why therefore a 0 for license costs, we use IntelliJ IDEA for Flex development)
Fluorida seems to be at the beginning and there is very little documentation so it looks like to need an investment (- for tool changes). It has no capture/replay (-).
Conclusion
So our tool of choice is SeleniumFlex and we hope to get capture/replay working in the near future.
What experience have you made with functional testing in Flex? Which one do you use?
Schneide Blog 
SilkTest 2009 also has Flex support. Why haven’t you checked that one, especially since SilkTest has been one of the more prominent testing tools in the last decade?
Didn’t know SilkTest2009 has Flex support and if they use the automation API and therefore need a FlexBuilderPro license to support custom components.
I could not find any decent info on the web pages. Can you provide any insight on this?
Hi, this is a brief yet interesting comparison of Flex functional testing tools. Is a ‘+’ a good thing or does it depend on the category? I assume the latter, although I think the ratings should be normalised or recategorised to make drawing a conclusion easier i.e. the more +`s the better.
I’ve used FunFx and have glossed over literature on FlexMonkey. Your summary of FunFx is either misleading or I have misunderstood its inner working:
FunFx uses the Flex Automation API extensively and it uses the ExternalInterface mechanism to do so. I do not believe you need FlexBuilder, despite what the website says; test scripts are written in Ruby so there is no obvious need for FB. (- Add. costs)
The Win32OLE aspect is also news to me, but perhaps that is a detail in testing with IE? Testing in Firefox leverages its JavaScript Shell (JSSh plugin) server i.e. it communicates over the wire, not through COM. I can’t imagine why this would hinder testing on Linux, for example. (+ OS ind.)
To test custom components you have to implement a corresponding automation delegate. In order to record new or existing events (that are ignored by default), you need to implement/instrument the automation delegate. Bare in mind these changes must be made to automate custom components; standard components should be fine out-of-the-box. Some extra accessibility can be achieved by simply modifying XML configuration, which I think is a bit harsh to categorise as a tool change. I also think it’s a bit unfair to consider the implementation of automation delegates as code pollution; this component+delegate model achieves a nice separation – yes you _may_ have to write code, but that code does not feature in your production build. These last two points equally apply to at least QTP also. We can’t get something for nothing (“,) (- Code pollution) (- Tool changes).
I’ve been keen to get FunFx working with Cucumber for a bit of BDD action. I had successfully configured the two sometime ago and it worked great. However, I failed on what I was trying to test (some drag and drop stuff), not realising then that the events I needed to record/replay/inject are ignored by default. When I get some time I might try this again.
Cheers, Darren
Darren, if I understood the automation API correctly you need FlexBuilderPro if you want to support custom components or events to compile your component with automation support, hence the FlexBuilderPro license costs.
Sorry, I must have not checked the ‘Notify…’ box.
The ‘standard’ IDE is not free and what you get for your money for going ‘pro’ goes way beyond automation.
Speaking subjectively, automation won’t be the motivating factor in a Flex Pro spending the extra cash – so I don’t think it is fair to say it is a cost solely attributable to automation/testing.
However, to the extent that you can’t use the automation api for free, yes you are right.
Thanks for sharing details.