Categorycomputer stuff

Running Selenium Webdriver on Bash for Windows

Bash for Windows has been working pretty great for me until I needed to run Selenium Webdriveron it. I quickly learned that it wouldn’t work just work right out of the box, and the set up for it is quite convoluted.

You will need

Bash setup

Install Firefox:

sudo apt-get install firefox

Export DISPLAY variable in ~/.bashrc. Just add this to the ~/.bashrc:

export DISPLAY=:0

Xming setup

Just download it from the link above and run it.

Download geckodriver

Download geckodriver from the link above and put in the /usr/bin/ folder in Bash for Windows.

Running Selenium

Here is a piece of sample Python code I used to setup the web browser. It will setup the browser, open Google, sit there for 10 second and then quit.

import time

from selenium import webdriver
from selenium.webdriver import DesiredCapabilities

def execute_with_retry(method, max_attempts):
    e = None
    for i in range(0, max_attempts):
        try:
            return method()
        except Exception as e:
            print(e)
            time.sleep(1)
    if e is not None:
        raise e

capabilities = DesiredCapabilities.FIREFOX
capabilities["marionette"] = True
firefox_bin = "/usr/bin/firefox"
browser = execute_with_retry(lambda: webdriver.Firefox(
    firefox_binary=firefox_bin, capabilities=capabilities), 10)

browser.get("https://www.google.com")

time.sleep(10)

browser.close()

Jenkins pipeline for Spring with beta and prod stages and deployment rollback

In the past couple of days, I’ve been experimenting a bit with the Jenkins pipeline plugin to create a code deployment pipeline with independent beta and prod stages for a Spring Boot app. I even managed to add rolling back a deployment in case a prod deployment fails! It took me a bit of time to Google my way through how to do everything, so I figure I just lay it all out here in case it helps other people do the same thing.

The nice thing about all of this is that I can push a code change to git, and Jenkins can build it, run through all the tests and then deploy to production automatically.

Layout of a pipeline script

Pipeline scripts are written in Groovy, which is a variant of Java. In general, a pipeline script is laid out like this:

node {
    def SOME_CONSTANT = "whatever"
    ...

    stage('some stage name like Build') {
        // Stuff to do as a part of this stage
    }

    stage('another stage') {
        // More stuff
    }

    ...
}

Each stage represents a stage in the pipeline (e.g. building, beta deployment, prod deployment etc.)

Defining some constants

First, a list of constants can be defined that can be used throughout the pipeline so that if the pipeline script gets reused somewhere, only these constants have to be changed. I’m not aware of anything that lets me reuse a pipeline in Jenkins without copying and pasting the code somewhere else so for now I’ll have to live with copying and pasting.

My project uses Maven so I got Jenkins to download its own copy of Maven that it can use to execute builds and have defined it as a constant. The name I’ve given it in the Jenkins Global Tool Configuration is “Maven 3.3.9” exactly. My project also uses Tomcat so there are some Tomcat specific things in there that may or may not be relevant to your use case.

    
def MAVEN_HOME = tool 'Maven 3.3.9'
    def WORKSPACE = pwd()

    def PROJECT_NAME = "name-of-project"
    def WAR_PATH_RELATIVE = "App/target/${PROJECT_NAME}.war"
    def WAR_PATH_FULL = "${WORKSPACE}/${WAR_PATH_RELATIVE}"
    def TOMCAT_CTX_PATH_BETA = "Tomcat-context-path-for-the-beta-stage"
    def TOMCAT_CTX_PATH_PROD = "Tomcat-context-path-for-the-prod-stage"
    def GIT_REPO_URL = "URL-to-git-repo-ending-in-.git"

Preparation Stage

First, the code has to be retrieved from the repository before it gets built. Jenkins allows storing username/password pairs so that they can be referenced without having to write out the password in plaintext. Jenkins uses a “credential ID” for this.

   
    stage('Preparation') {
        git branch: "master",
        credentialsId: "credentials-ID-stored-in-Jenkins-that-can-access-the-git-repo",
        url: "${GIT_REPO_URL}"
    }

Build Stage

Next, the code must be built and unit tested. “mvn clean install” will do just that (depending on what you want, you can always put in a different maven goal). The junit command is just there to take the resulting XML that gets generated during the build process and posts a graph of how many tests were run for each build.

    
    stage('Build') {
        sh "'${MAVEN_HOME}/bin/mvn' clean install"
        junit '**/target/surefire-reports/TEST-*.xml'
    }

Beta Stage

Once the build succeeds, you’ll want to deploy it to a beta environment, so integration tests can happen. The following happens at this stage:

  1. Get the right credentials to get permissions to Tomcat
  2. Call the deploy method to deploy the war file in Tomcat (more on that later)
  3. If the deployment fails for whatever reason, print out the deployment log for debugging and fail the build
  4. If the deployment succeeds, run the integration tests (the command to do this may differ based on use case)
    
    stage('Beta') {
        withCredentials([[$class: 'UsernamePasswordMultiBinding',
            credentialsId: 'credential-id-for-tomcat',
            usernameVariable: 'USERNAME', passwordVariable: 'PASSWORD']]) {
                // Password is available as an env variable, but will be masked 
                // if you try to print it out any which way
                def output = deploy(WAR_PATH_FULL, TOMCAT_CTX_PATH_BETA,
                        env.USERNAME, env.PASSWORD)
                if (output.contains("FAIL - Deployed application at context path " + 
                        "/${TOMCAT_CTX_PATH_BETA} but context failed to start")) {
                    echo "----- Beta deployment log -----"
                    echo output
                    echo "-------------------------------"
                    currentBuild.result = 'FAILURE'
                    error "Beta stage deployment failure"
                }
            }

        echo "Running integration tests"
        sh "'${MAVEN_HOME}/bin/mvn' -Dtest=*IT test"
        junit '**/target/surefire-reports/TEST-*.xml'
    }

The deploy method is what takes care of the actual deployment to Tomcat, which is how the Jenkins build tells Tomcat about the newly built war file. The deploy method is below (be sure to change the server IP and port). Alternatively, I could have built my project as an embedded jar file, but that has a different challenge in figuring out how to get Jenkins to tell the OS to execute the newly built jar file as a particular user.

  1. Based on the Tomcat context path, decide whether a beta or production deployment is happening
  2. Make a copy of the build war file and add .prod or .beta to the end of it so as to keep the original
  3. Set the Spring profile to use on the newly copied war file (more on that later)
  4. Call the curl command to do the actual deployment to Tomcat (more on that later)
def deploy(warPathFull, tomcatCtxPath, username, password) {
    def envSuffix = ""
    def isBeta = tomcatCtxPath.contains("beta")
    if (isBeta) {
        envSuffix = "beta"
    } else {
        envSuffix = "prod"
    }
    sh script: "cp ${warPathFull} ${warPathFull}.${envSuffix}" 
    setSpringProfile(warPathFull, isBeta)
    def output = sh script: "curl --upload-file '${warPathFull}.${envSuffix}' " +
            "'http://${username}:${password}@localhost:8081/manager/text/deploy" + 
            "?path=/${tomcatCtxPath}&update=true'", returnStdout: true
    return output
}

In the case of my project, I’m building a single war file that does not have a Spring profile (beta/prod) defined. This means that I have to manually define this before I deploy the app to Tomcat since there are some things that differ between beta and prod like database URL’s. To do this, I wrote a method that opens the war file like a zip (jar/war files are zip files) and adds a line to my application.properties to define a Spring profile.

Admittedly, doing this zip file manipulation seems kind of hacky. Alternatively, I could have defined my build such that I had a separate beta build and a prod build to avoid modifying the zip file, but the drawback is that I’d then have to build my code twice.

def setSpringProfile(warPathFull, isBeta) {
    def zipFileFullPath = warPathFull + "." + (isBeta ? "beta" : "prod")
    def zipIn = new File(zipFileFullPath)
    def zip = new ZipFile(zipIn)
    def zipTemp = File.createTempFile("temp_${System.nanoTime()}", 'zip')
    zipTemp.deleteOnExit()
    def zos = new ZipOutputStream(new FileOutputStream(zipTemp))
    def toModify = "WEB-INF/classes/application.properties"

    for(e in zip.entries()) {
        if(!e.name.equalsIgnoreCase(toModify)) {
            zos.putNextEntry(e)
            zos << zip.getInputStream(e).bytes
        } else {
            zos.putNextEntry(new ZipEntry(toModify))
            zos << zip.getInputStream(e).bytes
            zos << ("\nspring.profiles.active=" + (isBeta ? "beta" : "prod")).bytes
        }
        zos.closeEntry()
    }

    zos.close()
    zipIn.delete()
    zipTemp.renameTo(zipIn)
}

A curl command to Tomcat is what actually does the deployment. To deploy a file to Tomcat, do the following below. This will deploy the war file to Tomcat and instantly run it, thus it will be accessible at the given context path.

curl --upload-file 'path-to-war-file' http://username:password@server-address:port/manager/text/deploy?path=/tomcat-context-path&update=true

Prod Stage

The same kind of stuff happens in the prod stage as in the beta stage with a few exceptions. The following happens at this stage:

  1. Get the right credentials to get permissions to Tomcat
  2. Call the deploy method to deploy the war file in Tomcat (except this time it is prod)
  3. If the deployment fails for whatever reason, print out the deployment log for debugging and roll back the deployment
  4. If the deployment succeeds, save the build files, and then the pipeline is finished. Alternatively, smoke tests can be run at this point, but I did not implement this in my project

Rollback is important because if the deployment fails, you don’t want to be stuck with a broken environment. Since build artifacts are saved on successful deployments, these same artifacts can be brought back if future deployments fail. This means they can be redeployed so that the code can be fixed before another deployment happens.

    stage('Prod') {
        withCredentials([[$class: 'UsernamePasswordMultiBinding',
            credentialsId: 'credential-id-for-tomcat',
            usernameVariable: 'USERNAME', passwordVariable: 'PASSWORD']]) {
                // Password is available as an env variable, but will be masked 
                // if you try to print it out any which way
                def output = deploy(WAR_PATH_FULL, TOMCAT_CTX_PATH_PROD, env.USERNAME, env.PASSWORD)
                if (output.contains("FAIL - Deployed application at context path " + 
                        "/${TOMCAT_CTX_PATH_PROD} but context failed to start")) {
                    echo "Prod stage deployment failure, rolling back deployment"
                    echo "----- Prod deployment log -----"
                    echo output
                    echo "-------------------------------"
                    step([$class: 'CopyArtifact',
                            filter: "${WAR_PATH_RELATIVE}",
                            fingerprintArtifacts: true,
                            projectName: "${PROJECT_NAME}",
                            target: "${WAR_PATH_RELATIVE}.rollback"])
                    deploy(WAR_PATH_FULL + ".rollback/" + WAR_PATH_RELATIVE,
                            TOMCAT_CTX_PATH_PROD, env.USERNAME, env.PASSWORD)
                    currentBuild.result = 'FAILURE'
                    error "Prod deployment rolled back"
                } else {
                    archiveArtifacts artifacts: "${WAR_PATH_RELATIVE}*", fingerprint: true
                }
            }
    }

At the end you get to have something like this:

That pretty much sums up the whole Jenkins pipeline that I’ve been using lately for Spring projects!

Fixing T-Mobile international roaming

T-mobile has awesome phone plans for people living in America to get free international roaming in over 140 countries. For most people, all they’d have to do is flip the switch on their phones that enable international roaming. For a minority of people, there are a few more settings to mess with. In the past year, I’ve had two issues that resulted in me losing my data connection outside of USA and I’m documenting some things below to try since it worked for me – maybe it’ll help some people out.

Check that international data roaming is enabled

Try a different APN

I’m using “T-Mobile US LTE 260”. Not sure if changing this requires a phone restart. I restarted my phone anyway.

Set the APN protocol and APN roaming protocol to IPv4/IPv6

Just tap on the APN you want to change. Hitting the 3 dots will show the save button. Not sure if changing this requires a phone restart. I restarted my phone anyway.

Moving an OS to another disk and still have it boot with Linux

For the longest time, I’ve had an 80 GB HDD running my Windows partition (dual-boot setup with Ubuntu on a SSD), but now I’ve finally upgraded the Windows partition to an SSD as well. I looked into how to clone my Windows partition onto the SSD, such that I can still boot the disk.

I already have Ubuntu as my main OS, so copying the disk was easy using dd, which allows copying all the contents of one disk to another. This works well when the new hard drive is greater than or the same size as the current hard drive (I upgraded from a 80GB HDD to a 128GB SSD).

First I run this to see which disk I am copying from and to

fdisk -l

Then I run dd. If I am copying from /dev/sda to /dev/sda, then it’s:

dd if=/dev/sda of=/dev/sdb

But sometimes the disks don’t have the same size, so I used gparted to move/resize the partitions to make use of the extra space on the new disk. gparted complained that it might make my disk no bootable, but the disk was still bootable for me nonetheless. I didn’t even have to mess with any grub bootloader settings either. I simply unplugged the old disk, and left the new disk plugged in and booted into the new disk no problem.

Repairing corrupt PowerPoint files

A free tool called DiskInternals ZIP Repair was able to recover a corrupted PowerPoint file (*.pptx) that I had on my computer.  I learned that pptx files are actually zip files in disguise, and so using a utility to repair a corrupt zip archive could work.  At first I thought it was one of those sketchy bloatware programs, but Lifehacker has written an article about them before, so it should be fine.

Using the program was pretty simple.  I just opened it and told it which zip file (in my case the pptx file which I renamed to a zip file), and then the program did all the work and recovered everything.  It was like magic!

Migrating a WordPress blog from WordPress.com to WordPress.org

Since a few months ago, I migrated my blog from the previous free installation of WordPress that WordPress.com offered, to the self-hosted installation offered by WordPress.org (note that one is .org and the other is .com).  Here are just a few of the key differences stated briefly:

  • Self hosted installations allow greater control over what plugins the blog uses, what domain name to use, monetization, among other things
  • The cost is the added time required to tweak to do the setup and customisation

After I chose to switch to the self-hosted installation, I faced the problem of having to migrate all of my content, site stats, and followers.  I also had to set up proper URL redirection to the new website.  Most of the guides I read did answer how to migrate content, and set up URL redirects, but it was not so clear as to how to migrate my site stats and followers.

Content Migration

For migrating content, the WordPress.com blog allows exporting blog content into a file to be imported into the new WordPress installation.  Steps 1 to 3 in this guide illustrate the process: http://www.wpbeginner.com/wp-tutorials/how-to-properly-move-your-blog-from-wordpress-com-to-wordpress-org/

Migrating Site Stats and Followers

Migrating site stats and followers requires installing the Jetpack addon and linking it to the WordPress.com account (so that the self-hosted WP and the existing WP blog are connected).  Afterward, a support thread has to be made on WordPress.com to ask WP staff to do the rest of the migration (there seems to be no other way).  I found the information on how to do this at this site: http://wordpress.stackexchange.com/questions/161633/migrating-stats-from-wordpress-com-blog-to-self-hosted-wordpress-org-blog.  The post I made for my own site migration is here.

Setting Up URL Redirection

This part costs money unfortunately.  I paid ~$17 CAD for this so that my own URL would be redirected for the new one.  If redirection isn’t required, then there is no need to buy anything.  The setup is pretty straight forward.  After buying the redirect, it is just a matter of setting the destination WordPress site to go to.  Reference link here: https://en.support.wordpress.com/site-redirect/

Running League of Legends in PlayOnLinux (Ubuntu 14.04)

There are various guides floating around on the Internet for running League of Legends on Linux, and no single guide worked for me, but after piecing the information together from various places, I managed to get it to work on my system.

My computer specifications:

  • Intel Core i5-4570
  • 16 GB Memory
  • Radeon HD 5770

I followed the instructions here, with these changes:

  • Using video driver “fglrx-updates” (the tutorial talks about NVIDIA cards)
  • Did not install TuxLoL
  • Did not do anything regarding the “Maestro error” since it only applies to Optimus Notebook users
  • Did not follow step 6 because I did not run into the problem for big item icon text for the item shop
  • EDIT: Thanks to Ingvar’s comment, the installation progress for the game can be viewed like so: open terminal (Ctrl+Alt+T) and execute:
    # tail -f ~/.PlayOnLinux/wineprefix/LeagueOfLegends/drive_c/Riot\ Games/League\ of\ Legends/Logs/Patcher\ Logs/*.log

There are also other steps that I had to do, which I read from here:

  • Click configure for the “League of Legends” entry in “PlayOnLinux” and find the “Display” tab and then choose the following options:
    • Direct Draw Renderer – gdi
    • Video memory size – 4096 (or something else depending on graphics card)
    • Offscreen rendering mode – fbo\0
    • Everything else on default
  • Create a file called “game.cfg” in the directory “/home/your-username-here/PlayOnLinux’s virtual drives/LeagueOfLegends/drive_c/Riot Games/League of Legends/Config”. Below is what I have in my “game.cfg”:
[General]
EnableAudio=1
GameMouseSpeed=10
UserSetResolution=1
BindSysKeys=0
SnapCameraOnRespawn=0
OSXMouseAcceleration=1
AutoAcquireTarget=0
EnableLightFx=0
WindowMode=0
ShowTurretRangeIndicators=0
PredictMovement=0
WaitForVerticalSync=0
Colors=32
Height=1080
Width=1920
SystemMouseSpeed=0
CfgVersion=5.3.296
x3d_platform=1
[HUD]
CameraLockMode=0
MiddleClickDragScrollEnabled=0
KeyboardScrollSpeed=0.5000
ChatScale=50
ObjectTooltips=0
AutoDisplayTarget=0
ShowAllChannelChat=1
ShowTimestamps=1
ItemShopPrevY=39
ItemShopPrevX=106
NameTagDisplay=1
ShowChampionIndicator=0
ShowSummonerNames=1
ScrollSmoothingEnabled=0
MiddleMouseScrollSpeed=0.5000
MapScrollSpeed=0.5000
ShowAttackRadius=0
NumericCooldownFormat=1
SmartCastOnKeyRelease=0
EnableLineMissileVis=0
FlipMiniMap=1
ItemShopResizeHeight=0
ItemShopResizeWidth=164
ItemShopPrevResizeHeight=1080
ItemShopPrevResizeWidth=1920
ItemShopItemDisplayMode=1
ItemShopStartPane=1
[Performance]
CharacterInking=1
ShadowsEnabled=1
EnableHUDAnimations=0
PerPixelPointLighting=0
EnableParticleOptimizations=0
BudgetOverdrawAverage=0
BudgetSkinnedVertexCount=0
BudgetSkinnedDrawCallCount=0
BudgetTextureUsage=0
BudgetVertexCount=0
BudgetTriangleCount=0
BudgetDrawCallCount=0
EnableGrassSwaying=0
EnableFXAA=0
AdvancedShader=0
FrameCapType=6
GammaEnabled=0
Full3DModeEnabled=0
AutoPerformanceSettings=0
CharacterQuality=4
EffectsQuality=4
EnvironmentQuality=4
ShadowQuality=4
GraphicsSlider=6
paths=0
[FloatingText]
EnemyTrueDamageCritical_Enabled=1
EnemyMagicalDamageCritical_Enabled=1
EnemyPhysicalDamageCritical_Enabled=1
TrueDamageCritical_Enabled=1
MagicalDamageCritical_Enabled=1
PhysicalDamageCritical_Enabled=1
Countdown_Enabled=0
EnemyTrueDamage_Enabled=0
EnemyMagicalDamage_Enabled=0
EnemyPhysicalDamage_Enabled=0
TrueDamage_Enabled=0
MagicalDamage_Enabled=0
PhysicalDamage_Enabled=0
Score_Enabled=0
QuestComplete_Enabled=0
QuestReceived_Enabled=0
Disable_Enabled=0
Level_Enabled=0
Dodge_Enabled=0
Heal_Enabled=0
Special_Enabled=0
Invulnerable_Enabled=0
Debug_Enabled=0
Absorbed_Enabled=0
OMW_Enabled=0
EnemyCritical_Enabled=0
MagicCritical_Enabled=0
Critical_Enabled=0
[Volume]
SfxVolume=0.5
MasterVolume=0.5

 

Deleted IME’s still appear in the list even though they have been removed in Ubuntu 14.04

I had this weird problem when my Japanese and Chinese IME stayed in my IME list even though I had removed them.  It was driving me nuts, but it turns out that they can be disabled through the dconf-editor

First, run:

sudo apt-get install dconf-editor

Navigate to:

org/gnome/desktop/applications/input-sources

Bold options are options that have been modified.  The interesting one is the row labeled “sources”.  Clear the entries of the IME’s that should not appear in the list and that should remove the IME’s that shouldn’t be appearing.

Running LINE Messenger in Ubuntu 14.04

LINE is a messaging program kind of like WhatsApp, but it has a desktop client.  However, this desktop client only runs on Windows.  On Linux, it has to run under Wine, which allows running Windows programs (some) inside the Linux environment.  Caveat: sending stickers does not appear to work.

UPDATE 7. JULY 2015: Since this post, my version of LINE has been updated to 4.0.3.367, and it looks like stickers work now!

UPDATE 31. JULY 2016: Looks like in July 2015, LINE released a Chrome app.  Users of Chrome can just use that extension instead of using this method: Link

Running LINE in Ubuntu 14.04

  1. Install Wine
sudo apt-get install wine
  1. Install VC++ 2008 Redistributable
winetricks vcrun2008
  1. Download and install LINE for Windows

Without installing the VC++ 2008 Redistributable, LINE would intermittently crash (even though it was able to start correctly).  This worked on my Ubuntu 14.04 x64 system.

LINE Version: 3.9.1.188
Wine Version: 1.6.2

Time required for badblocks -svn for 1.8TB drive under RAID0

I was curious to see if I had any bad sectors on a set of drives (3x640GB = 1.8TB) I’ve owned since 2010, so I ran “badblocks -svn” on that drive and it took a little over three days to complete.

Some info about the drive setup:

  • 3x 640GB Western Digital Caviar Black
  • SATA 3.0 Gb/s
  • RAID0

Some info on the badblocks command specifically:

  • s – shows scan progress (percent completion)
  • v – verbose mode
  • n – non-destructive read-write mode and requires the drive to not be mounted (more info about the test mode below)
    1. block is first read
    2. test pattern then written into that block
    3. test pattern then read from that block
    4. compare the results from 2-3 to see if they’re the same
    5. put back the original block

Hopefully, this gives a reference for others who want to run this test so that they have an idea at least for how long the test would take.  For a less thorough, but faster test, one can opt to do a non-destructive read test (leave out the “n” option) and it also does not care whether the disk is mounted or not.

© 2017 Henry Poon's Blog

Theme by Anders NorénUp ↑