Easy automation via SikuliX: Starters

What is SikuliX?

SikuliX is an Java based automation software that identifies and operates GUI fields by image recognition. It automates anything that is visible on your desktop screen by utilizing “OpenCV” to identify the GUI components. So, in a way it is series of queued up images waiting to get recognized on the screen/GUI (mouse and/or keyboard) and on the basis of recognition particular actions will be performed.

Automation lets you perform the task much faster with no human intervention

It supports scripting via SikuliX IDE with Ruby, Python, JavaScript and as SikuliX is implemented in Java, so you might want to use SikuliX API in your Java projects. I have seen lot of projects on Github, where developers are using Sikuli along with other frameworks, especially Robot Framework.

SikuliX script consists of scripted workflow statements and reference to image files that will be used to identify particular fields on screen.

Basic programming knowledge is required to understand the concepts of variable, conditions and loops in SikuliX, however you should still be able to automate simple tasks without any hassle.

The SikuliX might look like primitive way of automating things but it can help you automate lot of daily tasks and most of the developers do use it with Selenium for Windows related tasks or test automations.

I will soon come with other important automation frameworks and python libraries that can help you automate things.

Let’s try SikuliX GUI for automation: Please follow few easy steps as listed below.

1. Download SikuliX and place it under your project folder.

2. Install JAVA runtime as per your OS.

3. Window user can go to ‘Win + R’, Open CMD as Administrator.

4. Now toggle to Sikuli’s JAR file and open it via CLI > java -jar sikuliX.jar

5. Now the SikuliX GUI should load up as below.

SikuliX Screen: (ScreenShot)

Sikuli Main Screen

SikuliX mainly works on matching the images with the screen pattern and then performs the required action.

So, you mainly utilize Pattern, String, Match, Region Objects and Location (Screen Pointer) options to successfully automate the tasks via SikuliX.

To demo the functionality of “SiKuliX”, I have used basic example of searching and opening ‘CMD” in Windows 10.

The images in Sikuli’s condition can be either “manually” captured image or you may use “Take Screenshot” from the SikuliX GUI. But before running/testing the functionality it is necessary to save your SikuliX project in a particular directly that will contain your scripts and images used to identify the patterns.

~Commands:

click(Photo of input box)
wait(Till this screen appear)
type(String input with action defined as Key.ENTER)

Sikuli GUI with Code and image selection

Second Example:

Conditional Automation that can be used for user interactions.

Code:

name = input(“Enter your name”)
popup(“Hope you are doing well! “+name, “Namaste!”)
months = (‘1’,’2',’3',’4',’5',’6',’7',’8',’9',’10')
whatdoyouthink = select(“How many months you been in lockdown?”, options = months)
real = popAsk(“Like really! “+whatdoyouthink+” months ?”)
howareyou = inputText(“How are feeling about COVID19 situation?”)
popup(“Yeap, “+howareyou, “New Normal”)”

Refer to my video for explanation: Above scenario explained.

SekuliX Conditional Automation by AshishSecDev

Some of the other commands that you may want to try:

~find, findall, wait, click, doubleClick, rightClick, input, inputText, select, hover, dragDrop, paste, popup, popError, popAsk and type.

There are zillions of combinations/commands that you can try and if you are finding GUI annoying then you are not alone. But you can always use Sekuli along with Robot Automation Framework (selenium) for advanced level tasks.

List of commands and API options: Commands and SikulX API

Thanks, I hope you learnt something new today :-)

~AshishSecDev