Running Selenium Webdriver on Bash for Windows

Bash for Windows has been working pretty great for me until I needed to run Selenium Webdriver on it. I quickly learned that it wouldn’t work just work right out of the box, and the set up for it is quite convoluted.

You will need

Bash setup

Install Firefox:

sudo apt-get install firefox

Export DISPLAY variable in ~/.bashrc. Just add this to the ~/.bashrc:

export DISPLAY=:0

Xming setup

Just download it from the link above and run it.

Download geckodriver

Download geckodriver from the link above and put in the /usr/bin/ folder in Bash for Windows.

Running Selenium

Here is a piece of sample Python code I used to setup the web browser. It will setup the browser, open Google, sit there for 10 second and then quit.

import time

from selenium import webdriver
from selenium.webdriver import DesiredCapabilities

def execute_with_retry(method, max_attempts):
    e = None
    for i in range(0, max_attempts):
        try:
            return method()
        except Exception as e:
            print(e)
            time.sleep(1)
    if e is not None:
        raise e

capabilities = DesiredCapabilities.FIREFOX
capabilities["marionette"] = True
firefox_bin = "/usr/bin/firefox"
browser = execute_with_retry(lambda: webdriver.Firefox(
    firefox_binary=firefox_bin, capabilities=capabilities), 10)

browser.get("https://www.google.com")

time.sleep(10)

browser.close()

Coca Cola Museum (Go there with a Pepsi shirt)

We had taken a super early flight out from Guadalajara that morning and had a short layover in Atlanta. Because of the early flight, I was still wearing the Chivas jersey from the day before, which was cool because randomly in the airport I’d see other people wearing the same jersey and we’d acknowledge each other in our mutual support for the Guadalajara team.

During the layover, we went to the Coca Cola Museum. At the ticket counter, one of the staff members randomly approaches me and asks me, “do you want a free T-shirt?”. When anybody offers anybody free anything, the first response is always skeptism and confusion. “What do I have to do?”, I replied. She replies, “WELL, you’ve got a Pepsi logo on your shirt”. The entire time, I never realized the Chivas jersey I was wearing even had a Pepsi logo. Kudos to her for picking that out (or maybe it was the red white and blue colours of the jersey that happen to be the Pepsi colours). At this point, I see the irony, and I take the free T-shirt and put it over my Chivas jersey. Pro tip to everyone going to the Coca Cola Museum: wear something with a Pepsi logo for a free T-shirt.

Inside the museum, they show off a ton of cool advertising/merchandise that Coke has made over the years in different countries. The museum was pretty busy and so it was pretty crowded in a lot of places.

Imagine decorating your living room like this!

This was probably the best part: drinking Coke branded drinks from different continents around the world.

For people into touristy things, then this museum would be a pretty cool place to go!

Photo album here

Spontaneous Trip to Guadalajara and the Liga MX Finals

My Mexican friend, who is an almost fanatical fan of the Guadalajara Chivas football team in Mexico and who also grew up in Guadalajara, asked me, “there’s a good chance that the Chivas will make it into the finals this year. If they do, I’m going to go back to Guadalajara and go to the game. Wanna come?” I replied back, “OK let’s do it”. Three weeks go by and my friend says, “so the Chivas made it into the final!” We bought our plane tickets and flew out two days later. We didn’t even have tickets to the game yet. An American friend of ours even bought his tickets to fly to Guadalajara even before the guy who is actually from there.

Alright, now let’s get the tickets for the game

“Don’t worry about the tickets for the game”, my friend said. Next thing we know, the tickets were all sold out to season ticket holders. Our other friend found some tickets on viagogo, BUT, we didn’t know if they were fake. We needed a new plan. “It’s OK, we can buy tickets off scalpers on the black market”, my friend says. The black market sounded all well and good until the news reported armed robberies occurring where scalpers were robbed of their legitimate tickets which were now worth a couple hundred US dollars, a value that is a large chunk of a typical local’s monthly salary. We could now only rely on the viagogo tickets being legitimate. Then there is me, the clueless foreigner in his country and doesn’t speak any Spanish, who can only helplessly watch events unfold without knowing at all how to help.

In our already not-so-good situation, a representative from viagogo calls us and said that the tickets can only be picked up at 5pm close to downtown. The problem was that the stadium is on the edge of the city and the game starts at 6pm. There was no way to go to the pickup location for the tickets AND get to the game on time because traffic on game day would be terrible. My friend calls the viagogo representative and tells him the situation and guilts him into letting us pick up the tickets earlier by saying how we trusted the company with these tickets and whatnot and the rep agrees to let us pick up the tickets earlier. Whether or not the rep was sincere in that offer we didn’t know. We needed a plan B in case we really had to pick up the tickets at 5pm. This is where our plan got SUPER elaborate. We spent a good hour brainstorming a good plan and this is what we came up with:

The guy who bought the tickets (the American), would have to ride a motorcycle to the ticket pickup location, pick up the tickets, and go to the stadium, while the rest of us would get to the stadium early with a van so that when the bike arrived with the tickets, we could hide the bike inside the van. When his dad heard the plan, he began to scold him saying how crazy it was, and in his scolding, I heard a word come up over and over again. When I looked it up, I found out it was profanity. Awesome! My Mexican friend called all of his connections that he had to see how we’d get our hands on a van and a motorcycle. According to him, “he knows a lot of people, people owe him favours”. The problem with that plan was that the guy who had the van got into an accident (whether or not that was true, we’ll never know), which suddenly broke our whole plan.

Okay, so what if he rode a bicycle to pick up the tickets? He’d have to ride 12km from the pick up location to the stadium in terrible traffic. Not to mention the person riding the bike isn’t familiar with traffic in Mexico (traffic that does not respect bikers), and that most streets don’t have stop signs or lights. Throw in some lack of sleep – the guy would be flying in that morning in an overnight flight from LA. He was up to the challenge. My Mexican friend summarized the whole plan into a PowerPoint, calling it “the story to be told about this amazing adventure”. We wouldn’t be able to execute this plan until the day of the game. In the meantime, my Mexican friend showed me around.

Lake Chapala

Lake Chapala is Mexico’s largest freshwater lake, and for whatever reason, there is a substantial population of Canadian and American retirees here.

Tequila

Got to see some Agave fields (and eat some Agave too!) on a tour at the Jose Cuervo Distillery in a town called Tequila (where the name of the alcohol comes from). I made the tour guide speak English because I was the only guy on the tour who didn’t know Spanish.

Driving in Mexico?!

At one point, my friend’s brother asked me if I wanted to drive. I thought to myself, “okay, so there are no stop signs anywhere, I have no idea where to stop, and based on the driving I’ve seen, nobody really follows the rules on the road”. I drove anyway!

Knowing that I was a complete newb on the road in Mexico, my friend and his brother were both very watchful to make sure I was stopping at the right times. They had practically memorized all the places where cars should and shouldn’t stop. I had no clue whatsoever. I stopped and went when they told me to.

Later on, we saw police cars stopped on the side of the street in front of us. My friends just told me to play it cool and just drive normally. I also realized I didn’t have ID on me, so if I had gotten pulled over or something, I would have had to pay $100 USD just to get out. But luckily, nothing happened.

Picking up the tickets

Now it was time to execute the plan. My friend called the Viagogo rep in the morning and he says that we can pick up the tickets at noon. Okay, so no problem. We went and picked up the tickets while in the back of our minds we were still wondering if the tickets are fake. If the tickets were fake, then our whole plan would be meaningless.

We meet the rep and he gives us the tickets. We closely inspect them, and everything looks good. We were in a bit of a disbelief. Everything seemed too easy. But we wouldn’t know if the tickets were real for sure until we actually go to the ticket counter. At this point, we wouldn’t have to execute our crazy bike plan, and could only wait and see when we got to the stadium. With the couple of hours until the game, we spent some time walking around the city center of Guadalajara.

Touring the city center

Lots of people out and about on a hot day. We went in May, statistically the hottest time of year in Guadalajara.

The local boy/girl scouts were running a recycling campaign or something.

Game time

After lunch, my friend’s dad drove us to the game. As expected, the traffic was real bad. On the way there, as my friend’s dad drove, he says to us that he’s going to break a bunch of traffic rules and not learn from him. He then proceeds to cut a bunch of lanes, crossing the hatched areas on the road, and cuts in in front of people. He even made use of an handicapped sign on his car to get into the parking lot quicker even though he wasn’t handicapped at all (seniors just get those because they’re above a certain age). With his epic driving, we got to the stadium right on time.

Now, the moment of truth. The guard checking the tickets would be the one to tell us if the tickets were real or not. As we got closer, we got more and more nervous. The guard scanned our tickets and waved us through. What a relief. We didn’t need any of our contingency plans after all. Logically, everything went smoother than we could have possibly imagined. Now it was time to enjoy the game!

The entire crowd pretty much supported the home team, and there was a small section of Tigres fans, probably about 50-100 people only. Tons of people waved Chivas flags, and some even waved the French flag because it had the same colours as the Chivas jersey. We all bought jerseys for the game too!

As the game went on, the crowd cheered and chanted, “dale dale dale rebaño!”, which roughly means “let’s go herd [of goats]” (chivas means goats). At one point, when it looked like the Tigres were making a comeback, the crowd cheered even harder. With an amazing 2-1 finish (4-3 aggregate), the Chivas had beaten the Tigres to win their 12th championship!

Celebration

That night, large crowds gathered at the Minerva statue in the city to celebrate the team’s big win. My friend’s dad said he’s never seen the crowd this big in a long time. To celebrate, the dad made us all tequila cocktails (there was probably 3-4 shots worth of tequila in that) before we headed out and had a great time. As the only Asians in the city (my American friend is of Japanese descent), we stood out a lot and people occasionally said random phrases to us in different Asian languages. I guess that’s what happens when two Asians show up with Chivas jerseys!

Last, but not least, the food

My first meal when I arrived in Mexico: Menudo. My friend thought my stomach wouldn’t be able to handle it, because the meat in the soup was tripe. But, nope! Chinese people eat tripe all the time!

Not exactly a meal that I had, but there is a running joke that a guy can take a girl out with roses and a can of Jack Daniels. Once that joke caught on, the store intentionally started putting roses next to the Jack Daniels in the store.

Tacos with carnitas and guacamole? Sign me up!

Lonches. Basically a sandwich with meat inside served with a dipping sauce on the side (can also just be poured onto the plate to let the bread soak). The bread used is a kind local to Guadalajara (birote/bolillo).

Then there are tortas ahogadas. Very similar to lonches. It has a the same bread, but this one is way meatier and the sauce is a tiny bit of spice.

Ceviche! A dish with raw fish cured in lemon/lime with added spices. Can’t say no to seafood!

My friend’s mom also cooked food for us and I totally wish I had taken some pictures of that food. She cooked a pork stew and I wish I knew what was in the sauce! Both parents encouraged me to eat so much and they showed so much hospitality! They kept offering me more and more food until I was super stuffed. Good thing my friend’s brother taught me “estoy bien”! This was definitely one of the best trips I ever had – full of excitement and not just doing touristy things. Having a local show you around really helps!

Link to photo album: here

Why are the days of the week in Japanese named after classical elements?

A random thought stumbled into my mind the other day when I realized that Sunday matched the Japanese word for Sunday is “日曜日”. The “日” part means “sun”. At first I thought it was just a coincidence, then I realized that Monday represented the moon, and in Japanese that’s “月曜日” and the “月” part means moon. Something is clearly going on here. Since the idea of seven days in a week doesn’t originate from Asia, the Japanese days of the week probably just followed western convention, which makes sense. Now when it comes to the English days of the week, the names were originally named by the Romans with the names of the sun, the moon, and the five known planets at the time [1].

Day of the week Latin word (English word)
Sunday Solis (sun)
Monday Lunae (moon)
Tuesday Martis (Mars)
Wednesday Mercurii (Mercury)
Thursday Jovis (Jupiter)
Friday Veneris (Venus)
Saturday Saturni (Saturn)

Now on the other side of the world, the Chinese and the Japanese gave names to these planets using their classical elements (the Chinese and Japanese names are the same). They gave the celestial bodies these names:

Celestial body Chinese/Japanese name Corresponding element
The sun
The moon
Mars 火星 Fire
Mercury 水星 Water
Jupiter 木星 Wood
Venus 金星 Gold
Saturn 土星 Earth/soil

So now, when the Japanese went to adopt the western calendar, they took the base meanings of the days of the weeks by their planets, and used their planet names in their place. So Sunday (from the sun) became “日曜日”, Monday (from the moon) became “月曜日” and so forth. Like this:

Celestial body English day of the week Japanese day of the week
Sun Sunday 日曜日
Moon Monday 月曜日
Mars Tuesday 火曜日
Mercury Wednesday 水曜日
Jupiter Thursday 木曜日
Venus Friday 金曜日
Saturn Saturday 土曜日

Cool, huh?

Jenkins pipeline for Spring with beta and prod stages and deployment rollback

In the past couple of days, I’ve been experimenting a bit with the Jenkins pipeline plugin to create a code deployment pipeline with independent beta and prod stages for a Spring Boot app. I even managed to add rolling back a deployment in case a prod deployment fails! It took me a bit of time to Google my way through how to do everything, so I figure I just lay it all out here in case it helps other people do the same thing.

The nice thing about all of this is that I can push a code change to git, and Jenkins can build it, run through all the tests and then deploy to production automatically.

Layout of a pipeline script

Pipeline scripts are written in Groovy, which is a variant of Java. In general, a pipeline script is laid out like this:

node {
    def SOME_CONSTANT = "whatever"
    ...

    stage('some stage name like Build') {
        // Stuff to do as a part of this stage
    }

    stage('another stage') {
        // More stuff
    }

    ...
}

Each stage represents a stage in the pipeline (e.g. building, beta deployment, prod deployment etc.)

Defining some constants

First, a list of constants can be defined that can be used throughout the pipeline so that if the pipeline script gets reused somewhere, only these constants have to be changed. I’m not aware of anything that lets me reuse a pipeline in Jenkins without copying and pasting the code somewhere else so for now I’ll have to live with copying and pasting.

My project uses Maven so I got Jenkins to download its own copy of Maven that it can use to execute builds and have defined it as a constant. The name I’ve given it in the Jenkins Global Tool Configuration is “Maven 3.3.9” exactly. My project also uses Tomcat so there are some Tomcat specific things in there that may or may not be relevant to your use case.

    
def MAVEN_HOME = tool 'Maven 3.3.9'
    def WORKSPACE = pwd()

    def PROJECT_NAME = "name-of-project"
    def WAR_PATH_RELATIVE = "App/target/${PROJECT_NAME}.war"
    def WAR_PATH_FULL = "${WORKSPACE}/${WAR_PATH_RELATIVE}"
    def TOMCAT_CTX_PATH_BETA = "Tomcat-context-path-for-the-beta-stage"
    def TOMCAT_CTX_PATH_PROD = "Tomcat-context-path-for-the-prod-stage"
    def GIT_REPO_URL = "URL-to-git-repo-ending-in-.git"

Preparation Stage

First, the code has to be retrieved from the repository before it gets built. Jenkins allows storing username/password pairs so that they can be referenced without having to write out the password in plaintext. Jenkins uses a “credential ID” for this.

   
    stage('Preparation') {
        git branch: "master",
        credentialsId: "credentials-ID-stored-in-Jenkins-that-can-access-the-git-repo",
        url: "${GIT_REPO_URL}"
    }

Build Stage

Next, the code must be built and unit tested. “mvn clean install” will do just that (depending on what you want, you can always put in a different maven goal). The junit command is just there to take the resulting XML that gets generated during the build process and posts a graph of how many tests were run for each build.

    
    stage('Build') {
        sh "'${MAVEN_HOME}/bin/mvn' clean install"
        junit '**/target/surefire-reports/TEST-*.xml'
    }

Beta Stage

Once the build succeeds, you’ll want to deploy it to a beta environment, so integration tests can happen. The following happens at this stage:

  1. Get the right credentials to get permissions to Tomcat
  2. Call the deploy method to deploy the war file in Tomcat (more on that later)
  3. If the deployment fails for whatever reason, print out the deployment log for debugging and fail the build
  4. If the deployment succeeds, run the integration tests (the command to do this may differ based on use case)
    
    stage('Beta') {
        withCredentials([[$class: 'UsernamePasswordMultiBinding',
            credentialsId: 'credential-id-for-tomcat',
            usernameVariable: 'USERNAME', passwordVariable: 'PASSWORD']]) {
                // Password is available as an env variable, but will be masked 
                // if you try to print it out any which way
                def output = deploy(WAR_PATH_FULL, TOMCAT_CTX_PATH_BETA,
                        env.USERNAME, env.PASSWORD)
                if (output.contains("FAIL - Deployed application at context path " + 
                        "/${TOMCAT_CTX_PATH_BETA} but context failed to start")) {
                    echo "----- Beta deployment log -----"
                    echo output
                    echo "-------------------------------"
                    currentBuild.result = 'FAILURE'
                    error "Beta stage deployment failure"
                }
            }

        echo "Running integration tests"
        sh "'${MAVEN_HOME}/bin/mvn' -Dtest=*IT test"
        junit '**/target/surefire-reports/TEST-*.xml'
    }

The deploy method is what takes care of the actual deployment to Tomcat, which is how the Jenkins build tells Tomcat about the newly built war file. The deploy method is below (be sure to change the server IP and port). Alternatively, I could have built my project as an embedded jar file, but that has a different challenge in figuring out how to get Jenkins to tell the OS to execute the newly built jar file as a particular user.

  1. Based on the Tomcat context path, decide whether a beta or production deployment is happening
  2. Make a copy of the build war file and add .prod or .beta to the end of it so as to keep the original
  3. Set the Spring profile to use on the newly copied war file (more on that later)
  4. Call the curl command to do the actual deployment to Tomcat (more on that later)
def deploy(warPathFull, tomcatCtxPath, username, password) {
    def envSuffix = ""
    def isBeta = tomcatCtxPath.contains("beta")
    if (isBeta) {
        envSuffix = "beta"
    } else {
        envSuffix = "prod"
    }
    sh script: "cp ${warPathFull} ${warPathFull}.${envSuffix}" 
    setSpringProfile(warPathFull, isBeta)
    def output = sh script: "curl --upload-file '${warPathFull}.${envSuffix}' " +
            "'http://${username}:${password}@localhost:8081/manager/text/deploy" + 
            "?path=/${tomcatCtxPath}&update=true'", returnStdout: true
    return output
}

In the case of my project, I’m building a single war file that does not have a Spring profile (beta/prod) defined. This means that I have to manually define this before I deploy the app to Tomcat since there are some things that differ between beta and prod like database URL’s. To do this, I wrote a method that opens the war file like a zip (jar/war files are zip files) and adds a line to my application.properties to define a Spring profile.

Admittedly, doing this zip file manipulation seems kind of hacky. Alternatively, I could have defined my build such that I had a separate beta build and a prod build to avoid modifying the zip file, but the drawback is that I’d then have to build my code twice.

def setSpringProfile(warPathFull, isBeta) {
    def zipFileFullPath = warPathFull + "." + (isBeta ? "beta" : "prod")
    def zipIn = new File(zipFileFullPath)
    def zip = new ZipFile(zipIn)
    def zipTemp = File.createTempFile("temp_${System.nanoTime()}", 'zip')
    zipTemp.deleteOnExit()
    def zos = new ZipOutputStream(new FileOutputStream(zipTemp))
    def toModify = "WEB-INF/classes/application.properties"

    for(e in zip.entries()) {
        if(!e.name.equalsIgnoreCase(toModify)) {
            zos.putNextEntry(e)
            zos << zip.getInputStream(e).bytes
        } else {
            zos.putNextEntry(new ZipEntry(toModify))
            zos << zip.getInputStream(e).bytes
            zos << ("\nspring.profiles.active=" + (isBeta ? "beta" : "prod")).bytes
        }
        zos.closeEntry()
    }

    zos.close()
    zipIn.delete()
    zipTemp.renameTo(zipIn)
}

A curl command to Tomcat is what actually does the deployment. To deploy a file to Tomcat, do the following below. This will deploy the war file to Tomcat and instantly run it, thus it will be accessible at the given context path.

curl --upload-file 'path-to-war-file' http://username:password@server-address:port/manager/text/deploy?path=/tomcat-context-path&update=true

Prod Stage

The same kind of stuff happens in the prod stage as in the beta stage with a few exceptions. The following happens at this stage:

  1. Get the right credentials to get permissions to Tomcat
  2. Call the deploy method to deploy the war file in Tomcat (except this time it is prod)
  3. If the deployment fails for whatever reason, print out the deployment log for debugging and roll back the deployment
  4. If the deployment succeeds, save the build files, and then the pipeline is finished. Alternatively, smoke tests can be run at this point, but I did not implement this in my project

Rollback is important because if the deployment fails, you don’t want to be stuck with a broken environment. Since build artifacts are saved on successful deployments, these same artifacts can be brought back if future deployments fail. This means they can be redeployed so that the code can be fixed before another deployment happens.

    stage('Prod') {
        withCredentials([[$class: 'UsernamePasswordMultiBinding',
            credentialsId: 'credential-id-for-tomcat',
            usernameVariable: 'USERNAME', passwordVariable: 'PASSWORD']]) {
                // Password is available as an env variable, but will be masked 
                // if you try to print it out any which way
                def output = deploy(WAR_PATH_FULL, TOMCAT_CTX_PATH_PROD, env.USERNAME, env.PASSWORD)
                if (output.contains("FAIL - Deployed application at context path " + 
                        "/${TOMCAT_CTX_PATH_PROD} but context failed to start")) {
                    echo "Prod stage deployment failure, rolling back deployment"
                    echo "----- Prod deployment log -----"
                    echo output
                    echo "-------------------------------"
                    step([$class: 'CopyArtifact',
                            filter: "${WAR_PATH_RELATIVE}",
                            fingerprintArtifacts: true,
                            projectName: "${PROJECT_NAME}",
                            target: "${WAR_PATH_RELATIVE}.rollback"])
                    deploy(WAR_PATH_FULL + ".rollback/" + WAR_PATH_RELATIVE,
                            TOMCAT_CTX_PATH_PROD, env.USERNAME, env.PASSWORD)
                    currentBuild.result = 'FAILURE'
                    error "Prod deployment rolled back"
                } else {
                    archiveArtifacts artifacts: "${WAR_PATH_RELATIVE}*", fingerprint: true
                }
            }
    }

At the end you get to have something like this:

That pretty much sums up the whole Jenkins pipeline that I’ve been using lately for Spring projects!

© 2017 Henry Poon's Blog

Theme by Anders NorénUp ↑