[Appium]Image comparison with Appium and other libraries

In general, we compare two images when we’d like to detect differences between them in visual.

I believe OpenCV and ImageMagick are famous tools to achieve the purpose lately. But sometimes we faced building errors, for example. So, I’ve used a simple enough comparison tool. That is Kobold. The library uses blink-diff as a comparison engine. The library provides some features to configure how to compare each other.

I’ve published Ruby wrapper to use them.

And you can also see some articles for it on this site. (Most of them are Japanese, and you need to translate though…) Link

By the way, I also researched some similar and straightforward libraries. I believe perceptualdiff also a famous library. I compared ImageMagick and perceptualdiff a bit before.

A few years ago, I started to investigate OpenCV, opencv_sample is one example, to make such comparison more flexible. The reason why I investigated it is many guys have been implemented Machine Learning / Deep Learning features in it. So I’ve believed it helps us.

And lately, Appium is going to support OpenCV based comparison features as one of the fundamental features. https://github.com/appium/appium-support/pull/65 is the brilliant PR. We can use some OpenCV features via them. We are also able to get diff images to ensure the results in visual.

The feature will help many users who would like to get image comparison easy.

Actually, to make test suites stable, separating steps are important like:

  • Conduct test cases and assert the results without image comparison
  • Compare images
  • Collect and save the results

Otherwise, the new Appium feature will encourage automation more, I believe.

Advertisements

[Flutter]Flutter app with Appium

I touch [Flutter]some thoughts for the Flutter (v0.1.0) before. And this time, I tried to run the app with Appium.

We can run UI test with Appium as well as native apps. But Flutter apps have no accessibility identifier and customised resource id. So, it’s quite difficult to implement robust test automation scenario so far.

https://github.com/KazuCocoa/flutter-appium

Probably, Appium is easy to kind of automation than Espresso/XCUITest so far.

[NPM][Appium] Restrict installed module size

I found https://github.com/appium/appium/issues/9912 and I learned the npm‘s --production flag to reduce the module size.

Install without devDependencies

$ npm install --production

https://docs.npmjs.com/cli/install#description

With the –production flag (or when the NODE_ENV environment variable is set to production), npm will not install modules listed in devDependencies.

Remove devDependencies resources

https://docs.npmjs.com/cli/prune

$ npm install
$ cd node_modules/appium
$ npm prune --production

[Appium][Ruby]Run tests in parallel

When we run Appium in parallel on one Machine and one Appium server with Ruby, we have two ways to implement it. (For Android, we don’t need to configure this kind of environment. We must cover this for iOS because of the platform’s limitations.)

One is thread programming and another is process-fork programming. I’ll put examples to run tests with each method in the below.

  • rake run_parallel_t
    • Run tests with thread programming. In this case, each thread shares their data on the memory, classes. It means once our scenario depends on class method, we can’t keep clean test environment.
  • rake run_parallel_p
    • Run tests with process-fork programming. In this case, each process is isolated each other. So, we can keep the hermetic environment in each process.

I think process fork model is good for this time since we don’t need to depend on thread programming when we’d like to run tests in parallel.

[Appium][Android]Scrolling to an element

Just for my memo

When we scroll views using Appium, then we have two ways to achieve it.

  1. Use UiScrollable

The below method is implemented in Ruby client.

  1. Use TouchAction or related actions.

The below is methods I used to implement as helper methods in my framework.

In my experience, 2 is almost stable than 1 since 1‘s behaviour depends on OS side and it also depends on devices,(and OSs).

[Appium]See the screenshots and source on offline after tests failing

Almost software engineers take a bunch of time to investigate the cause when automated tests fail.
And they try to make sure whether the error is test code itself or product code. In the case, helpful error messages and additional information help us so much and save time for the investigation.

In this article, I explain some tips to make such investigation comfortable and show you a helpful app for it.

The repository is: https://github.com/KazuCocoa/appium-source-viewer

What kind of data do you get when tests fail?

Almost case, guys, use GUI testing tools like Appium, get screenshots when tests fail. Also. Some guys get videos as an additional information as well. They’ve become common in the mobile testing world recently.

By the way, can you find the way to fix the fail only from the screenshots or videos?

If your automated test code uses displayed words to find elements or some actions, you probably fix the test code from them. But, using resource-id for Android and accessibility identifier for iOS are common recently, and they can’t find from screenshots or videos because they are embedded in code level. Not for humans. In this case, you must find elements in code level and you also must run again to make sure the test code fixed.

The time is sometimes dull and wast. Also, we must repeat launching the test app and go to the target view to see view hierarchy by uiautomatorviewer, for example. The activity takes a bunch of time.

Thus, if we can get the hierarchy and view at the same time and we can see them easily in the investigation, the time will be short.

Get the source

To achieve it, I strongly recommend you to get page_source as well as screenshots when tests fail.

The page_source looks like:

<AppiumAUT>
  <XCUIElementTypeApplication type="XCUIElementTypeApplication" name="UICatalog" label="UICatalog" enabled="true" visible="true" x="0" y="0" width="375" height="667">
    <XCUIElementTypeOther type="XCUIElementTypeOther" enabled="true" visible="false" x="0" y="0" width="375" height="667">
    ...
    ...
    </XCUIElementTypeWindow>
  </XCUIElementTypeApplication>
</AppiumAUT>

Espresso or EarlGrey get such results and show us as their results. So, we can understand why the tests failed and which elements should we get or something… (XCUITest can’t get such information, I think…)

Screenshot x Source

Only getting source is a bit difficult to image and understand the target view from it since it is just XML.

To reduce such difficulty, we can use visualisation tools like Appium-Desktop.

Offline x Screenshot x Source

Appium Desktop is so helpful to see elements mapping to the view. But the tool requires network connection between Appium server and test targets.

So, I developed a tool which can map a screenshot and its source, and visualise us like Appium Desktop. We can see some highlights in the screenshot and the source equivalent each other.

sample

The repository again: https://github.com/KazuCocoa/appium-source-viewer

The tool will help us to find the cause and how we can fix it only for failed test data, the screenshot and its source information.

I’ve put the usage in the README, but you can use the app following:

  1. Download the binary from release
  2. Unzip and copy the app in Application
  3. Set a screenshot path and a source path
  4. Reload from reload icon

I’ve checked only on macOS, but it’s Electron-based application. So, I guess it can work on windows, too.

Conclusion

I put some tips to make fix failed test easy and the app to help it.

Have a good testing!

Appiumのテスト失敗時の修正をちょっとわかりやすくする

こんばんは。Selenium/Appium Advent Calendar 2017の15日目の記事です。

多くのエンジニアの方々は、テスト失敗時の問題修正に時間を使っていることかと思います。それはテストコードの問題なのか、製品コードの問題なのか、はたまた別な問題なのか。

そのため、例えばテスト失敗時のエラー情報から、再度テストを実行することなく、どのように修正すれば治るのか、どうなおせば良いのかを教えてくれるとその時間を減らすことができますね。(これに似た話は色々と目にしますね)

今回は、Appiumを使った時の、そんな修正を便利にするちょっとしたアプリを共有します。

リポジトリはこちら => https://github.com/KazuCocoa/appium-source-viewer

テスト失敗時に取得する情報は何を取得する?

多くの場合、Appiumのようなツールを使う人はテスト失敗時にテスト対象のスクリーンショットを取得するかと思います。また、人によっては動画を取得する場合もあるでしょう。Appiumに限らず、GUIが関係する場合は特に、そのような情報を失敗時の情報として取得するでしょう。

では、テストを修正する人はその情報を見ただけでどのように修正すればテストがOKになるか判断することができますか?

表示されている文字だけを見ているのであれば修正できる可能性はあるでしょう。ただ、例えばAndroidだと resource-id 、iOSだと accessibility identifier を使うことが普通になっている場合は、スクリーンショットを取得したとしても該当しそうな箇所をその表示のみから追うのはキツいのではないでしょうか。ソースコードを確認し、再度繰り返し実行し、調整していくことも多いのではないでしょうか。

とても時間が勿体無いですね。 どこが失敗 し、 どう修正すべきか を教えてくれるテスト結果であれば、この時間は短縮でき、楽に問題を修正することができるでしょう。

Sourceも取得する

どこが失敗 し、 どう修正すべきか を知る手段として、テスト失敗時の情報として page_source の情報は1つの役立ち情報でしょう。例えば以下な感じのXMLです。

<AppiumAUT>
  <XCUIElementTypeApplication type="XCUIElementTypeApplication" name="UICatalog" label="UICatalog" enabled="true" visible="true" x="0" y="0" width="375" height="667">
    <XCUIElementTypeOther type="XCUIElementTypeOther" enabled="true" visible="false" x="0" y="0" width="375" height="667">
    ...
    ...
    </XCUIElementTypeWindow>
  </XCUIElementTypeApplication>
</AppiumAUT>

GoogleのEspressoやEarlGreyはテスト失敗時にそのように画面のヒエラルキー情報を出力してくれますね。そのおかげで、修正者は要素がそもそもないのか、ある場合はどのような情報を付与するとそれを特定できるのか、はたまた別な要素である必要があるのかを判断することができます。(XCUITest自体は残念ながら取得できないです(できなかったですよね…?))

スクリーンショット x source

sourceだけの場合、そのXML情報を想像できなければ結局は表示内容を確認することになるでしょう。

そこで役立つのがスクリーンショット。ただ、sourceとスクリーンショットを自分の目で照らし合わせることはそれはそれで慣れが必要です。

そんな時、例えばAppium-Desktopのようなツールを利用できたらどうでしょう? 視覚的にスクリーンショットから該当するsourceの要素を確認できる ようになり、だいぶ問題を見つけることが楽になるのではないでしょうか。

オフライン x スクシーンショット x source

便利だろうということで作って見ました。Appium Desktopを参考に、オフラインで使える形にしてみました。

sourceのXMLを読み込み、任意の要素をクリックすると、該当する要素がハイライトされます。逆に、スクショの任意の要素をクリックすると、該当するsourceのXML要素がハイライトされます。

sample

再度、リポジトリはこちら => https://github.com/KazuCocoa/appium-source-viewer

これを用いると、

例えばAndroidだと resource-id 、iOSだと accessibility identifier を使うことが普通になっている場合は、スクリーンショットを取得したとしても該当しそうな箇所をその表示のみから追うのはキツいのではないでしょうか。ソースコードを確認し、再度繰り返し実行し、調整していくことも多いのではないでしょうか。

に対して、失敗した情報からどのように修正すれば良いのかという内容を、より楽に見つけることができるでしょう。要素を特定する為にAppiumが必要とする情報がだいたい見られる為です。XPathのコピペは確か実装していないですが、XPathは不安定なこともあるので個人的には問題になってはいません。

使い方はREADMEに書いているのですが、 releaseからバイナリを取得して、macOSの Application ディレクトリに .app を移動し、アプリを実行してください。そこに、スクリーンショットへのパスと、sourceへのパスを指定、画面をreloadすると使うことができます。

著者の環境がMacなのでmacOSしか試していません。。。が、一応はElectron製なのでWindowsやLinux系列でも動くものをビルドすることはできるでしょう。(どなたか…)

最後に

というわけで、テストの失敗は、どう修正すればそれが治るのかということも知ることができるととても嬉しいです。その為に、ちょっとした便利アプリを必要なところだけ抜き出して作って見ました。

Tips for UI/Scenario Tests: Recording screens and getting view hierarchy

When we run UI related tests, debugging failing test is one of the most difficult and need a bunch of time task. To make it easier, we usually take screenshot, capture videos and make error report helpful.

Taking screenshot is very famous, and I skip it.

Recording

Android

Android provide screenrecord command to record a video for Android 4.4+.
I’ve implemented a bit helpful library, named droid_adbs

In the library, you can record video like the following. record.start call the screenrecord command in forked process. So, you can also start recording in setup and finish it in teardown.

require 'test_helper'

class DroidAdbsCommonsRecordTest &lt; Minitest::Test
  def test_record
    record = ::DroidAdbs::ScreenRecord::Recording.new
    record.start

    sleep 10

    record.stop

    # Should wait to finish exporting recorded file
    sleep 1

    record.pull
    assert File.exist? &quot;./droid_adbs_video.mp4&quot;

    record.delete
    File.delete &quot;./droid_adbs_video.mp4&quot;
  end
end

iOS

For Simulator, we can capture video easily with the following method.

xcrun simctl io booted recordVideo capture.mov

Tips

More or less, recording video increases devices/machines load. So, I recommend recording screens only you need. For example, once run your test suite and retry failed tests with enable recording the screen then.

Capture view hierarchy when test failed

With ruby_lib and Appium x RSpec

RSpec.configure do |c|
  c.after(:example, type: :feature) do |example|
    if example.exception
      # capture screen
      view_hierarchy = source # source is getting view hierarchy command
      # Save view_hierarchy in particular files.
    end
  end
end

Espresso

  • We can get hint to finding elements.

EarlGrey

  • We can see view hierarchy by default.

XCUITest

  • I’m not sure we can get such information from error result.

Conclusion

Helpful error results decrease a bunch of time to debug and detect the cause.

[Appium]Get invisible elements

uiautomator2 start supporting findInvisibleElement.
https://github.com/appium/appium-uiautomator2-server/issues/61

After the following method, you can get invisible elements from uiautomator2.

update_settings allowInvisibleElements: true

https://github.com/appium/ruby_lib/blob/cc913edb4bc24c18a58c2c5490dfb7b5a705d303/lib/appium_lib/common/command.rb#L46