Categorycomputer stuff

Running Selenium Webdriver on Bash for Windows

Bash for Windows has been working pretty great for me until I needed to run Selenium Webdriver on it. I quickly learned that it wouldn’t work just work right out of the box, and the set up for it is quite convoluted.

You will need

Bash setup

Install Firefox:

sudo apt-get install firefox

Export DISPLAY variable in ~/.bashrc. Just add this to the ~/.bashrc:

export DISPLAY=:0

Xming setup

Just download it from the link above and run it.

Download geckodriver

Download geckodriver from the link above and put in the /usr/bin/ folder in Bash for Windows.

Running Selenium

Here is a piece of sample Python code I used to setup the web browser. It will setup the browser, open Google, sit there for 10 second and then quit.

import time

from selenium import webdriver
from selenium.webdriver import DesiredCapabilities

def execute_with_retry(method, max_attempts):
    e = None
    for i in range(0, max_attempts):
        try:
            return method()
        except Exception as e:
            print(e)
            time.sleep(1)
    if e is not None:
        raise e

capabilities = DesiredCapabilities.FIREFOX
capabilities["marionette"] = True
firefox_bin = "/usr/bin/firefox"
browser = execute_with_retry(lambda: webdriver.Firefox(
    firefox_binary=firefox_bin, capabilities=capabilities), 10)

browser.get("https://www.google.com")

time.sleep(10)

browser.close()

Jenkins pipeline for Spring with beta and prod stages and deployment rollback

In the past couple of days, I’ve been experimenting a bit with the Jenkins pipeline plugin to create a code deployment pipeline with independent beta and prod stages for a Spring Boot app. I even managed to add rolling back a deployment in case a prod deployment fails! It took me a bit of time to Google my way through how to do everything, so I figure I just lay it all out here in case it helps other people do the same thing.

The nice thing about all of this is that I can push a code change to git, and Jenkins can build it, run through all the tests and then deploy to production automatically.

Layout of a pipeline script

Pipeline scripts are written in Groovy, which is a variant of Java. In general, a pipeline script is laid out like this:

node {
    def SOME_CONSTANT = "whatever"
    ...

    stage('some stage name like Build') {
        // Stuff to do as a part of this stage
    }

    stage('another stage') {
        // More stuff
    }

    ...
}

Each stage represents a stage in the pipeline (e.g. building, beta deployment, prod deployment etc.)

Defining some constants

First, a list of constants can be defined that can be used throughout the pipeline so that if the pipeline script gets reused somewhere, only these constants have to be changed. I’m not aware of anything that lets me reuse a pipeline in Jenkins without copying and pasting the code somewhere else so for now I’ll have to live with copying and pasting.

My project uses Maven so I got Jenkins to download its own copy of Maven that it can use to execute builds and have defined it as a constant. The name I’ve given it in the Jenkins Global Tool Configuration is “Maven 3.3.9” exactly. My project also uses Tomcat so there are some Tomcat specific things in there that may or may not be relevant to your use case.

    
def MAVEN_HOME = tool 'Maven 3.3.9'
    def WORKSPACE = pwd()

    def PROJECT_NAME = "name-of-project"
    def WAR_PATH_RELATIVE = "App/target/${PROJECT_NAME}.war"
    def WAR_PATH_FULL = "${WORKSPACE}/${WAR_PATH_RELATIVE}"
    def TOMCAT_CTX_PATH_BETA = "Tomcat-context-path-for-the-beta-stage"
    def TOMCAT_CTX_PATH_PROD = "Tomcat-context-path-for-the-prod-stage"
    def GIT_REPO_URL = "URL-to-git-repo-ending-in-.git"

Preparation Stage

First, the code has to be retrieved from the repository before it gets built. Jenkins allows storing username/password pairs so that they can be referenced without having to write out the password in plaintext. Jenkins uses a “credential ID” for this.

   
    stage('Preparation') {
        git branch: "master",
        credentialsId: "credentials-ID-stored-in-Jenkins-that-can-access-the-git-repo",
        url: "${GIT_REPO_URL}"
    }

Build Stage

Next, the code must be built and unit tested. “mvn clean install” will do just that (depending on what you want, you can always put in a different maven goal). The junit command is just there to take the resulting XML that gets generated during the build process and posts a graph of how many tests were run for each build.

    
    stage('Build') {
        sh "'${MAVEN_HOME}/bin/mvn' clean install"
        junit '**/target/surefire-reports/TEST-*.xml'
    }

Beta Stage

Once the build succeeds, you’ll want to deploy it to a beta environment, so integration tests can happen. The following happens at this stage:

  1. Get the right credentials to get permissions to Tomcat
  2. Call the deploy method to deploy the war file in Tomcat (more on that later)
  3. If the deployment fails for whatever reason, print out the deployment log for debugging and fail the build
  4. If the deployment succeeds, run the integration tests (the command to do this may differ based on use case)
    
    stage('Beta') {
        withCredentials([[$class: 'UsernamePasswordMultiBinding',
            credentialsId: 'credential-id-for-tomcat',
            usernameVariable: 'USERNAME', passwordVariable: 'PASSWORD']]) {
                // Password is available as an env variable, but will be masked 
                // if you try to print it out any which way
                def output = deploy(WAR_PATH_FULL, TOMCAT_CTX_PATH_BETA,
                        env.USERNAME, env.PASSWORD)
                if (output.contains("FAIL - Deployed application at context path " + 
                        "/${TOMCAT_CTX_PATH_BETA} but context failed to start")) {
                    echo "----- Beta deployment log -----"
                    echo output
                    echo "-------------------------------"
                    currentBuild.result = 'FAILURE'
                    error "Beta stage deployment failure"
                }
            }

        echo "Running integration tests"
        sh "'${MAVEN_HOME}/bin/mvn' -Dtest=*IT test"
        junit '**/target/surefire-reports/TEST-*.xml'
    }

The deploy method is what takes care of the actual deployment to Tomcat, which is how the Jenkins build tells Tomcat about the newly built war file. The deploy method is below (be sure to change the server IP and port). Alternatively, I could have built my project as an embedded jar file, but that has a different challenge in figuring out how to get Jenkins to tell the OS to execute the newly built jar file as a particular user.

  1. Based on the Tomcat context path, decide whether a beta or production deployment is happening
  2. Make a copy of the build war file and add .prod or .beta to the end of it so as to keep the original
  3. Set the Spring profile to use on the newly copied war file (more on that later)
  4. Call the curl command to do the actual deployment to Tomcat (more on that later)
def deploy(warPathFull, tomcatCtxPath, username, password) {
    def envSuffix = ""
    def isBeta = tomcatCtxPath.contains("beta")
    if (isBeta) {
        envSuffix = "beta"
    } else {
        envSuffix = "prod"
    }
    sh script: "cp ${warPathFull} ${warPathFull}.${envSuffix}" 
    setSpringProfile(warPathFull, isBeta)
    def output = sh script: "curl --upload-file '${warPathFull}.${envSuffix}' " +
            "'http://${username}:${password}@localhost:8081/manager/text/deploy" + 
            "?path=/${tomcatCtxPath}&update=true'", returnStdout: true
    return output
}

In the case of my project, I’m building a single war file that does not have a Spring profile (beta/prod) defined. This means that I have to manually define this before I deploy the app to Tomcat since there are some things that differ between beta and prod like database URL’s. To do this, I wrote a method that opens the war file like a zip (jar/war files are zip files) and adds a line to my application.properties to define a Spring profile.

Admittedly, doing this zip file manipulation seems kind of hacky. Alternatively, I could have defined my build such that I had a separate beta build and a prod build to avoid modifying the zip file, but the drawback is that I’d then have to build my code twice.

def setSpringProfile(warPathFull, isBeta) {
    def zipFileFullPath = warPathFull + "." + (isBeta ? "beta" : "prod")
    def zipIn = new File(zipFileFullPath)
    def zip = new ZipFile(zipIn)
    def zipTemp = File.createTempFile("temp_${System.nanoTime()}", 'zip')
    zipTemp.deleteOnExit()
    def zos = new ZipOutputStream(new FileOutputStream(zipTemp))
    def toModify = "WEB-INF/classes/application.properties"

    for(e in zip.entries()) {
        if(!e.name.equalsIgnoreCase(toModify)) {
            zos.putNextEntry(e)
            zos << zip.getInputStream(e).bytes
        } else {
            zos.putNextEntry(new ZipEntry(toModify))
            zos << zip.getInputStream(e).bytes
            zos << ("\nspring.profiles.active=" + (isBeta ? "beta" : "prod")).bytes
        }
        zos.closeEntry()
    }

    zos.close()
    zipIn.delete()
    zipTemp.renameTo(zipIn)
}

A curl command to Tomcat is what actually does the deployment. To deploy a file to Tomcat, do the following below. This will deploy the war file to Tomcat and instantly run it, thus it will be accessible at the given context path.

curl --upload-file 'path-to-war-file' http://username:password@server-address:port/manager/text/deploy?path=/tomcat-context-path&update=true

Prod Stage

The same kind of stuff happens in the prod stage as in the beta stage with a few exceptions. The following happens at this stage:

  1. Get the right credentials to get permissions to Tomcat
  2. Call the deploy method to deploy the war file in Tomcat (except this time it is prod)
  3. If the deployment fails for whatever reason, print out the deployment log for debugging and roll back the deployment
  4. If the deployment succeeds, save the build files, and then the pipeline is finished. Alternatively, smoke tests can be run at this point, but I did not implement this in my project

Rollback is important because if the deployment fails, you don’t want to be stuck with a broken environment. Since build artifacts are saved on successful deployments, these same artifacts can be brought back if future deployments fail. This means they can be redeployed so that the code can be fixed before another deployment happens.

    stage('Prod') {
        withCredentials([[$class: 'UsernamePasswordMultiBinding',
            credentialsId: 'credential-id-for-tomcat',
            usernameVariable: 'USERNAME', passwordVariable: 'PASSWORD']]) {
                // Password is available as an env variable, but will be masked 
                // if you try to print it out any which way
                def output = deploy(WAR_PATH_FULL, TOMCAT_CTX_PATH_PROD, env.USERNAME, env.PASSWORD)
                if (output.contains("FAIL - Deployed application at context path " + 
                        "/${TOMCAT_CTX_PATH_PROD} but context failed to start")) {
                    echo "Prod stage deployment failure, rolling back deployment"
                    echo "----- Prod deployment log -----"
                    echo output
                    echo "-------------------------------"
                    step([$class: 'CopyArtifact',
                            filter: "${WAR_PATH_RELATIVE}",
                            fingerprintArtifacts: true,
                            projectName: "${PROJECT_NAME}",
                            target: "${WAR_PATH_RELATIVE}.rollback"])
                    deploy(WAR_PATH_FULL + ".rollback/" + WAR_PATH_RELATIVE,
                            TOMCAT_CTX_PATH_PROD, env.USERNAME, env.PASSWORD)
                    currentBuild.result = 'FAILURE'
                    error "Prod deployment rolled back"
                } else {
                    archiveArtifacts artifacts: "${WAR_PATH_RELATIVE}*", fingerprint: true
                }
            }
    }

At the end you get to have something like this:

That pretty much sums up the whole Jenkins pipeline that I’ve been using lately for Spring projects!

Fixing T-Mobile international roaming

T-mobile has awesome phone plans for people living in America to get free international roaming in over 140 countries. For most people, all they’d have to do is flip the switch on their phones that enable international roaming. For a minority of people, there are a few more settings to mess with. In the past year, I’ve had two issues that resulted in me losing my data connection outside of USA and I’m documenting some things below to try since it worked for me – maybe it’ll help some people out.

Check that international data roaming is enabled

Try a different APN

I’m using “T-Mobile US LTE 260”. Not sure if changing this requires a phone restart. I restarted my phone anyway.

Set the APN protocol and APN roaming protocol to IPv4/IPv6

Just tap on the APN you want to change. Hitting the 3 dots will show the save button. Not sure if changing this requires a phone restart. I restarted my phone anyway.

Moving an OS to another disk and still have it boot with Linux

For the longest time, I’ve had an 80 GB HDD running my Windows partition (dual-boot setup with Ubuntu on a SSD), but now I’ve finally upgraded the Windows partition to an SSD as well. I looked into how to clone my Windows partition onto the SSD, such that I can still boot the disk.

I already have Ubuntu as my main OS, so copying the disk was easy using dd, which allows copying all the contents of one disk to another. This works well when the new hard drive is greater than or the same size as the current hard drive (I upgraded from a 80GB HDD to a 128GB SSD).

First I run this to see which disk I am copying from and to

fdisk -l

Then I run dd. If I am copying from /dev/sda to /dev/sda, then it’s:

dd if=/dev/sda of=/dev/sdb

But sometimes the disks don’t have the same size, so I used gparted to move/resize the partitions to make use of the extra space on the new disk. gparted complained that it might make my disk no bootable, but the disk was still bootable for me nonetheless. I didn’t even have to mess with any grub bootloader settings either. I simply unplugged the old disk, and left the new disk plugged in and booted into the new disk no problem.

Repairing corrupt PowerPoint files

A free tool called DiskInternals ZIP Repair was able to recover a corrupted PowerPoint file (*.pptx) that I had on my computer.  I learned that pptx files are actually zip files in disguise, and so using a utility to repair a corrupt zip archive could work.  At first I thought it was one of those sketchy bloatware programs, but Lifehacker has written an article about them before, so it should be fine.

Using the program was pretty simple.  I just opened it and told it which zip file (in my case the pptx file which I renamed to a zip file), and then the program did all the work and recovered everything.  It was like magic!

© 2017 Henry Poon's Blog

Theme by Anders NorénUp ↑