At Frankfurt Airport, when Edward and I lined up to check in, we were the only two Chinese people in line.  We seemed very out of place.  Some of the people gave us these stares that seemed to ask “why are you flying to Kiev?” We wanted to visit Chernobyl and shoot AK-47’s.

Our experience on the plane was dreadful.  Four crying babies/toddlers plus jet lag plus one hour delay was not a good combination. No sleep for me I guess.  Worst flight ever.

Awkward Car Ride

The hostel arranged transportation for us and so when we arrived, I looked for someone holding a sign with my name on it (first time ever!).  The driver was not happy at how we were late (not that we could do anything about it).

Probably since he was late, he drove like a maniac.  He drove way faster than the posted limits and passed every car on the highway.  The expected 45 min ride was reduced to almost 20 minutes.  The driver didn’t speak English.  No one said a word during the entire ride.  Awkward.

I had read before that people sometimes get driven to random places by taxi drivers before they get mugged.  By getting the hostel transportation, we minimized that risk.  Despite that, I was at peak alertness the whole time, making sure he was going the right general direction (I vaguely remembered the directions from the airport to the hostel).

Dream House Hostel

This is by far one of the hostels I’ve ever stayed at.  The staff were extremely helpful.  We asked them like a thousand questions and they happily answered them every time.  They treated security very seriously.  Kiev might not be safe, but I felt safe leaving my passport and cash in the locker in the room.

Security Measures

During our trip planning, we had read about the potential dangers in going to Kiev from the Canadian travel report about Ukraine.

The report tells travelers to “exercise a high degree of caution”.  It warns of dangers such as pickpockets, passport scams, armed robbery, racially motivated violence, muggings, etc.

Seeing how the hostel’s security was pretty good, we left our valuables in the room.  We each carried a decoy wallet with nothing in it except a scan of our passports and the cash we needed for the day.  No credit cards, no real ID, no cards at all.

If anybody steals a wallet, no big.  Better to lose $30 than all our money.  Nobody can scam us for our passports if we aren’t carrying it.  I didn’t even put on a hidden money belt.  The less I brought the better.  We felt pretty prepared, and luckily nothing happened.

Language

People in Kiev generally speak Ukranian or Russian.  I know neither.  I also don’t know the Cryllic writing system.  Even if I wanted to ask in English how to get somewhere, I can’t even pronounce the name of the place.  When I went to Greece, I could at least pick out the greek alphabet (side effect of studying math and physics), but in Kiev, I can’t even read anything.  Too bad.

Food

The first night we arrived in Kiev, we had no idea what kind of food was around here.  We couldn’t even read the menus anyway.  Picking a restaurant was a matter of randomly walking into a restaurant.  To our surprise, our random pick turned out to be Japanese food.  Interesting.  No pictures needed.  Vancouver has an abundance of Japanese restaurants.  Beer is worth taking a picture of though.  I can’t even read the label.

One of the hostel staff had recommended to us the Golden Dukat Coffee House.  I wish I took more pictures here, but I found someone else’s travel blog with good pictures of it!  Good coffee, good dessert.  And they have an English menu!

We were told that Borscht is one of the traditional soups in Ukraine.  When I tried it, its taste was so familiar and I had to think about where I had tried it before.  It tasted a lot like the vegetable soup that people can get at Hong Kong style cafes (羅宋湯).  I Googled it later, and apparently the Hong Kong recipe (while a little bit different) is a derivative of it.  What a coincidence.

Independence Square

By far the largest city square I have been to.  For me this was the place to see in Kiev.  The pictures here don’t do justice to how epic it was.

At the square, there would always be these people carrying pigeons around.  They kept coming up to me with me and I had no idea what they wanted.  Later I found out that apparently they’re trying to get tourists to take a picture with them holding a bird.  Not sure why that’s cool but okay.

There were also people dressed up like Mickey Mouse or Shrek trying to get little kids to take pictures with them for money.  They did this even when the temperature was above 30 degrees Celsius.

St. Andrew’s Church

Nearby our hostel, there was a nice church.  It’s architecture was quite different from anything I’d seen before.

Friendship Arch

This arch was built to commemorate the unification of Ukraine and Russia when Ukraine was still part of the Soviet Union.

Street Market

On the way from our hostel to Independence Square, we walked along a long stretch of road with many street vendors selling souvenirs, trinkets and other random stuff.  People there even sold gas masks, replica German WWII medals, and fake guns.

Trip Ideas for Next Time

After we came back to Vancouver, we found out that the hostel started offering tours to abandoned missile silos, helicopter rides, and Yak plane acrobatics!  If only the hostel were offering those tours.  We would have totally gone to those.  There’s always next time.

Links to photo albums:

• Kiev
• Kiev (Edward’s Photos)

Here is my next travel blog entry about my visit to Berlin. In a short three days, we managed hit up quite a few places in the city, especially the historical areas about the Cold War and the Second World War. I would have liked to have posted this sooner, but my computer refused to cooperate.

Massive Train Delay

I was always under the impression that the Deutsche Bahn was always on time, but I guess that’s not true anymore. The City Night Line was supposed to leave at 12 AM from Mannheim and arrive in Berlin at around 7:30, but due to a train crash causing a chemical spill in Switzerland, our train took a detour. It arrived three and a half hours late. At first it was supposed to only be a 170 minute delay, but as time went on, the delay went longer and longer. To pass the time, we walked around Mannheim late at night, and pretty much saw the entire city, AKA the schloss, in about an hour. We spent the rest of the time just moping around the station. It was cold at night too and I never expected to be waiting so long so I ended up freezing when I was outside. I tried sleeping on the benches, but that didn’t really work for me, and instead ended up sitting there eating snacks until the train came. Maybe we should have chilled at a bar instead. Because of the long wait for the train, I didn’t sleep too well. It also didn’t help that I forgot to reserve a spot that had a bed, and I instead reserved a reclining seat instead. Because of the late train, the conductors handed out a giant form in pure German for people to fill out to get part of their money back.

Berlin Hauptbahnhof

We arrived in Berlin at about 10:30 in the morning, 3 hours later than scheduled. We all got off the train, still super tired, and started our journey in Berlin. The Berlin main station was the biggest station I’d seen so far. It had a really modern look using glass everywhere they could. Its sheer size allowed for a lot of businesses to open inside the station. The station had three floors with two floors used as train platforms. Coincidentally, it was also the station’s fifth birthday. When we got outside, we could see the German Reichstag within a walkable distance.

German Reichstag

Seeing that we could already see the Reichstag when we walked out of the Hauptbahnhof, we proceeded there first. Since I played so many World War II games, I already knew what the Reichstag looked like. Over the years, the Reichstag underwent a few renovations such as the glass dome that was added after the war. The rest of the building looked so similar to what I had seen in video games and historical photos that as I walked up the stairs, it briefly reminded of me of the guy with the flamethrower in the cut-scene in Call of Duty 5. I also remembered some of the iconic photographs I’d seen such as the soldier waving the flag of the Soviet Union on the roof.

Prior to going to Berlin, I had reserved spots for the roof terrace and the dome (according to their website, people couldn’t go up there without a reservation). Before going in, I had to put my stuff through their X-Ray machine (maybe they’re afraid of terrorists or something). They had confiscated my Swiss Army Knife for the duration that I was up there. At the roof terrace, the staff issued audio guides to every guest. While walking to different areas of the dome, the audio guide would automatically begin talking about parts of the city that one could see from that part of the dome. I didn’t really pay attention to the boring voice of the audio guide and the information that it had wasn’t all that interesting (it really didn’t matter to me if I knew how many rooms were in a particular building or the land area it took up).

Brandenburg Gate

From the top of the Reichstag, we could see the one of the symbols of Germany, the Brandenburg Gate. It is located at one end of the street Unter den Linden, and was used as a sign of peace during the Prussian Empire. Since it is also one of the most famous landmarks in Europe, there were naturally a lot of tourists here. So much so that there were people dressed up in weird costumes to earn money from people taking pictures with them. There were a few people holding American and USSR flags and even people dressed up as Star Wars characters. Near the gate were also a embassies of various countries such as France and the United States. One of the buildings even proudly waved the flag of Quebec (I don’t know why and it wasn’t the Canadian embassy either).

Eastener Hostel

By now, we were already super tired from the lack of sleep and from carrying our bags around all day. We took the city train to our hostel for a quick rest and to drop off some of our stuff. The man at the hostel was super thorough in explaining the rules of the hostel (with his East German accent) and even took the time to give us a map and point of the major tourist destinations on it. Using this map, we were able to find all the places we wanted to go to. By the end of the trip, the map had tears in several places and random folds here and there, clearly showing how intensely I used it.

Ampelmann

While walking around the city, I noticed that the pedestrian traffic light signals had a different design than the rest of Germany. Apparently, the traffic light man used in some places in Berlin were based off of the design used in former East Germany. I thought this Ampelmann guy was pretty cool.

The Remains of the Berlin Wall near Nordbahnhof

One of the locations that the map pointed us to were to some of the remnants of the Berlin Wall. Before coming to Berlin, I had a thought of posing at the wall with a hammer, but I never got one and I don’t think the people would have been very happy with me doing that. The segments of the wall here showed a lot of graffiti along its entire length. The city also preserved the guard tower and showed some of the methods of preventing escapes. Also in the area was a memorial for those who died while trying to flee to West Germany.

East German Cars

One of the peculiar things about Berlin was the fact that tourists can still take rides in cars (known as a Trabant or Trabi for short) that the GDR once used. I looked inside one of them and it just seemed so old and run down. I figure the only reason that these cars are still around are because of their historical/moneymaking value.

Really Cool Old-Style German Font

When I walked through some of the old train stations, I came across a really cool looking vintage font that I felt was so characteristic of Germany. Maybe I watched too many documentaries or something.

Checkpoint Charlie

Taking the train from Nordbahnhof, we made our way to Checkpoint Charlie, one of the border crossings once used before the unification of East and West Germany. The old border crossing booth and the sign telling people they’re leaving the American Sector still remains today. Standing outside are also two people holding American flags. Near the border booth, there is a giant sign with an American soldier on it and on the reverse side, there is a Soviet soldier. On the street, street vendors sold random things like fur hats with the logo of the Soviet Union on it, gas masks, and flags of the GDR and USSR.

Haus am Checkpoint Charlie

Adjacent to the border crossing, there was a museum dedicated to the documenting the border security between East and West Germany. The museum documented various methods of escape such as hiding people inside the hood of a car and using a hot air balloon. Also displayed were tools that people used to escape like special wall climbing footwear and stories of escapes. Seeing all these different methods made me realize how smart these people were. I would have never thought of something like that. Sadly, I was already super tired while walking through this museum and I was ready to just go back to the hostel and sleep. It was still light out at the time. Unfortunately, we weren’t allowed to take photographs in this museum.

DFB Cup Final

While in Berlin, we saw a lot of people walking around wearing stuff from football teams and apparently the DFB Cup Final was to be played in the Berlin Olympic Stadium. To catch the football game, we went to a bar to tune in. While at this bar, I got to try Berlin-style wheat beer. Theirs was different than the wheat beer I usually drink because here, they had beer cocktails. I tried the red and green Berliner wheat beer. The red one had a raspberry flavour while the green one had a woodruff flavour.

The Fernsehturm

The Fernsehturm is the TV tower where people can go up to see a view of the city. It is also one of the tallest buildings in the European Union. We didn’t go here since we didn’t really care so much about the view and we went to see other landmarks instead.

Potsdamer Platz at Night

According to my boss at work, Potsdamer Platz was a must see at night, and after seeing it for myself, I totally agree with him. The beauty of this place was the architecture that looked so beautiful in the night, especially the Sony Center. The roof of the Sony Center was shaped like an umbrella and its color would gradually change between blue and purple.

Französische Straße

Located on this street were the Französischer Dom, Berlin Concert House, and Deutscher Dom. Each of these three buildings have quite a bit of history and are all have amazing architecture. This place is definitely the place to be to see old-style architecture in Berlin.

Fassbender & Rausch Chocolatiers

This chocolatier is quite well known and their store is frequently visited by tourists. What makes it so special is that they built many of the landmarks in Berlin out of chocolate. Anything made of chocolate is cool. The longer I stayed in this store, the bigger craving for chocolate I had.

Führerbunker

During the Battle of Berlin, Hitler stayed in his bunker beneath his New Reich Chancellery. I went to the location of where the bunker once was only to find the area replaced by pretty regular things like shops and apartments. The side of the street here had plaques explaining the history of this area and its use for Hitler’s Bunker.

Memorial to the Murdered Jews of Europe

Near Hitler’s Bunker was a memorial for the Jewish victims of the holocaust. The memorial had thousands of concrete blocks, or stelae, of varying heights arranged in a grid pattern. According to one of the designers of the memorial, Peter Eisenman, he stated that “the stelae are designed to produce an uneasy, confusing atmosphere, and the whole sculpture aims to represent a supposedly ordered system that has lost touch with human reason.” Walking through the memorial reminded me so much of Bomberman. The memorial had the grid pattern just like in the game. It would probably also be really fun jumping from one block to the next, but it would have been really disrespectful.

Potsdamer Platz during the Day

To take a break and relax during the day, we went back to Potsdamer Platz for a beer. Now I understand why my boss told me to come here at night instead of during the day. It doesn’t look as nice at this hour.

Victory Column

Similar to the Arc de Triomphe on Avenue des Champs-Élysées in Paris, the Victory column stands in the end of the street Unter den Linden and is similarly surrounded by a giant roundabout. The column serves to commemorate the Prussian victories in the Danish-Prussian War, Austro-Prussian War, and the Franco-Prussian War. Surrounding the column are other smaller statues. One of the statues, I thought I saw my last name on it, but looking more closely, I realized it said “Roon” instead of “Poon”. What a bummer.

Schloss Bellevue

Located in the Tiergarten District of Berlin is the official residence of the President of Germany. We sat on the grass outside the residence to take a rest and to admire the beauty of the schloss and to watch the bike race that was happening just on the adjacent road.

Soviet War Memorial

Built using stone from the Reich Chancellery not long after the end of the Second World War, this memorial remembers the war dead of the Soviet Union, particularly the 80 000 soldiers who died in the Battle of Berlin. The memorial is of a statue of a Soviet soldier and on its left and right sides are two 152 mm Howitzer artillery pieces and two T-34 tanks (for some reason, birds like to go inside the barrel of the tank’s main gun). When I looked at the artillery piece, I tried to open the chamber for where the rounds go and tried to change the angle of the gun, but none of that worked. I’m pretty sure they didn’t want people messing with it. Behind the building is a small museum documenting the memorial’s construction and the Battle of Berlin. The memorial used to be guarded by Soviet troops during the Cold War even though it was in the British zone of occupation and now it is managed by the city.

Hackesche Höfe

One of my friends suggested that we go here since this place is known for its beautiful courtyards. It had eight interconnected courtyards, each with its own unique design. Some of the courtyards had tables for people to relax with coffee or beer while others had gardens.

Berliner Dom

Located on Museum Island, the Berliner Dom is a cathedral that belongs to the Evangelical Church. What was weird was that when we arrived there, we saw a group of people there having a cosplay party. There was one guy waving around a flag of the Empire of Japan. I never knew Anime was that big in Germany. Even weirder is that some of these kids were frolicking in the fountain in front of the Dom.

Museum Island

Another well known landmark in Berlin is Museum Island, which is an area with five internationally well known museums. They are the Altes Museum, Neues Museum, Alte Nationalgalerie, Bode Museum, and Pergamon Museum. We didn’t go in any of these museums since none of us were that interested in what the museums had to offer. The Alte Nationalgalerie had a look that seemed to have been inspired by the Greek Parthenon (but I’m not sure). While we were there, the sky started raining for about 15 minutes. It was quite refreshing since it was really hot earlier in the day.

Food

Although we didn’t try any food that were unique to Berlin (such as currywurst), we did splurge for one of our meals, and I found the food we had to be really delicious. Since Berlin is a lot closer to the coast than Stuttgart, I got to enjoy better quality seafood.

New Synagogue

Prior to the Second World War, the synagogue was badly damaged during Kristallnacht and the synagogue could no longer hold any services. Since the war, the synagogue was reconstruct and it is now the main synagogue for the Jewish community in Berlin.

Olympic Stadium

The current stadium was originally built for the 1936 Summer Olympics. Since the stadium was built during the Nazi period, the architecture of the building follows the Nazi style. Apparently, it was at these Olympic games that the tradition of the torch marathon began.

Potsdamer Platz for the Third Time

On our second night in Berlin, we went back to the Sony Center to relax for a little bit again. This time I got to try new kinds of beer such as mango and passion fruit flavoured wheat beer. It had an interesting flavour, but I thought the regular kind tasted a little better.

The Brandenburg Gate and the Reichstag at Night

When we arrived here at night, we found this place to be deserted even though the landmarks here looked amazing in the night. It was a good thing for us since there weren’t any tourists around. It allowed me to get really clear shots of these places.

The East Side Gallery

This area contains a continuous segment of the Berlin Wall that is 1.3 km in length. Along the wall are paintings from artists around the world to promote freedom in the world. The gallery also features the iconic political painting of Leonid Brezhnev and Erich Honecker kissing during the days of the GDR. There was also a souvenir shop here that stamped people’s passports with the GDR stamp for one euro. I thought that was a cool gimmick.

The Topography of Terror

To document the terrible deeds that the Nazi regime and the SS perpetrated between 1933 and 1945, an open-air museum was set up at the site of an old SS installation. It talked about how the Nazi’s transported people from conquered areas to the concentration and extermination camps set up across the continent. There was even a special exhibit just on the war crimes perpetrated by Adolf Eichmann. Near the museum were also SS buildings that have been converted to government offices.

Going Home

After three days of travelling in Berlin, we were all very tired and we were ready to go home. We went to our train platform for the ICE Sprinter about five minutes before the departure to find out that our train wasn’t there. We had no idea were it was and we didn’t want to miss the train. Apparently, some arsonists damaged some parts of the railway so the train ended up stopping on a different platform. Luckily, we found out in time and we managed to get on before it left. On the way back, I got to see some fantastic scenery of the German country side.

Souvenirs

I found that it was this trip that I spent the most money on souvenirs. I ended up buying Ampelmann themed cell phone accessories and a full sized flag of the USSR (not that I’m a communist or anything).

More photos here

• Introduction
• Motivation
• Newton’s Method
• Steepest Descent Method
• Limitation
• Sources

Introduction

In many sciences, the problem of non-linear optimization appears quite often.  What happens if they’re non-linear?  Sometimes, the system of equations remains easy to solve even given non-linearity, but other times, the possibility of solving the problem analytically may not exist if the algebra gets too complicated.  The solution lies in using numerical methods (approximations).  Here, I present two common methods: Newton’s Method, and the Steepest Descent Method. 

Each method has its own limitations and the choice of which method to use depends on the problem.  These methods involve using iterative computation in order to reach a solution.  Luckily, computers can prove useful in providing solutions very quickly due to its quick processing power for numerical computation.

In order to understand some of the concepts and terms used in this article, the reader has knowledge in multivariable calculus.  Also, equations in picture form made the line spacing look funny, so I’m going to do my best write the equations as text.  I should also point out that I couldn’t write the vector arrow on top of the variables in text form so vectors are written in bold.

Motivation

In calculus, the idea of optimization involves taking the derivative of the function and equating it to zero.  In multivariable calculus, the same idea applies except, we set all the partial derivatives to zero and solve the system.

So in general, if we are given a function:

ƒ(x), where x = [x₁, x₂, … x_n]^T

The maximum or minimum can be found by taking a partial derivative with respect to the vector x.  This gives the following equations:

∂ƒ/dx₁ = 0

∂ƒ/dx₂ = 0

…

∂ƒ/dx_n = 0

A short-hand way of expressing the above equations is by using the gradient vector, giving∇ƒ(x) = 0.  By definition, the gradient is defined by:

∇ƒ(x) = [∂ƒ/dx₁, ∂ƒ/dx₂, … , ∂ƒ/dx_n].

An analytical solution is not always possible, hence the motivation for approximating the solution using numerical methods.

Newton’s Method

In general, this method is written below.  Explanation of each step will follow.

1. Given the equations to solve, derive equations for the approximating line or plane, depending on whether the problem is 2D, 3D, nD, etc.
2. Using the linearized equations, solve the system of equations
3. Using the results from step 2, generate new equations for a new approximating line or plane, and repeat step 2.

This method also involves starting with a guess, denoted by a variable with a subscript of 0.  A guess is required, because this is the starting point for this method.  Using this guess, each successive solution should converge (more on that later).

Step 1

In order to solve the equations ∇ƒ(x) = 0 (see Motivation for more information), we can apply multivariate Newton’s method to approximate the solution.  Like the single-variate (two-dimensional) version, a linear approximation is required.  However, instead of having an approximation in the form of a line, the approximation is a plane instead.

2D (line): ƒ(x) ≈ ƒ(x₀) + ƒ’(x₀)(x – x₀)

3D (plane): ƒ(x, y) ≈ ƒ(x₀, y₀) + ƒ_x(x₀, y₀)(x – x₀) + ƒ_y(x₀, y₀)(y – y₀)

nD (nD plane): ƒ(x) ≈ ƒ(x₀) + ∇ƒ(x₀)(x – x₀)

Since we are trying to optimize, we have the partial derivatives of the function we are trying to optimize, but in order to apply Newton’s Method, we need a linear approximation of the partial derivatives.  This means that in order to approximate the partial derivatives, we need to take another derivative in order to get the slopes of the approximating planes.  If that description didn’t make sense, try the following.  For an equation with two independent variables, there will be two equations for the partial derivatives, one for x and one for y.  Taking partial derivatives from each of those equations again will give the slopes of the plane that approximate the first partial derivative.  So if we have two independent variables, we end up getting four equations: ƒ_xx, ƒ_xy,ƒ_yy, and ƒ_yx (ƒ_xy and ƒ_yx should be the same).  In general, for n partial derivatives, we would have n² equations if we took another set of derivatives.  The approximations for both of the partial derivatives would be:

ƒ_x(x, y) ≈ ƒ_x(x₀, y₀) + ƒ_xx(x₀, y₀)(x – x₀) + ƒ_xy(x₀, y₀)(y – y₀)

ƒ_y(x, y) ≈ ƒ_y(x₀, y₀) + ƒ_yy(x₀, y₀)(x – x₀) + ƒ_yx(x₀, y₀)(y – y₀)

Plug in the values for the second derivatives, and the values from the initial guess (the x y values and what function value you get using those values).

Step 2

Although the equations generated from step 1 is yet another system of equations, the main difference is that the equations from step 1 are linear.  This means that the system can be solved analytically.  Solve for the solution of the system using row reduction methods, or other methods.  The solution from solving the system will be the values used for the next iteration and will replace the guessed values.  The solution may or may not be close to the actual solution.  More iterations need to be done to see if the answer changes with each iteration.  Usually if the guess is really far away, then it takes a lot of iterations to get it really close.  However, sometimes, a bad guess may cause the solution to diverge, meaning that the solution will be further and further off.

Step 3

Generate new equations for the approximating planes using the solution from step 2.  Using these equations, go back to step 2 and solve it again.  This method is super tedious if done on paper.  Matlab can help, but then again, Matlab can do the entire approximation with just a few commands, but not everyone has access to it unfortunately.

Hopefully after a few iterations, the solution stops changing.  When that happens, you can substitute the final solution back into the original equation to see if it works!

Steepest Descent Method

Here is the general method:

1. Find the direction of the maximum or minimum directional derivative
2. Follow it until a maximum or minimum is reached

Newton’s Method is sufficient in a lot of cases, but sometimes it may be too hard to go beyond first derivatives.  This main idea of this method is to use the first derivative and follow the direction in which the function gets larger/smaller.  A fundamental concept used in this method is the directional derivative.  Since in 3D (or higher dimensions), we are not restricted to movement along the x axis.  This means we could move in any random direction, and in each direction the rate of change is going to be different.  To calculate the max or min, we need to calculate where the directional derivative is largest or smallest.  Just like a boulder rolling down the steepest slope on the hill.

Starting with a guess, we can find out the value of the function at the point and the direction to travel in.  Take a step in that direction and see how the value of the function changed.  Keep doing that until a max or min is reached.

Step 1

The definition of the directional derivative is the dot product of the gradient and the unit vector specifying the direction.  Here is the definition of the directional derivative:

D_uƒ = ∇ƒ(x)·u

Since the definition involves a dot product, it is true that if the two vectors are orthogonal, the resulting dot product is zero.  The direction that this derivative is in is called the level direction.  But if the two vectors are parallel, we get the highest value of the directional derivative.  For a vector that is parallel in the same direction, the directional derivative will be a maximum, but if the vector is antiparallel (parallel but both pointing opposite ways), the directional derivative will be a minimum.

Since the maximum value of the directional derivative is parallel to the gradient, we can calculate the unit vector of the gradient and use it as the unit vector u.  In order to get the minimum, simply flip the direction of the vector by multiplying it by –1.

u = ∇ƒ(x)/|∇ƒ(x)|

Although not necessarily required, substituting this value into the definition of the directional derivative, and doing some simplification with algebra, the maximum and minimum directional derivative is given by:

Max D_uƒ = |∇ƒ(x)|

Min D_uƒ = -|∇ƒ(x)|

Using the initial guess, calculate the direction of to travel in at that point.

Step 2

Knowing the direction to travel choose a desired step size “h”.  This means that if the equation will be traversed in a particular direction, we need to know exactly how far to go.  Choosing smaller step sizes gives better computational accuracy but increases computation time.

To get to the maximum, use the direction for the maximum directional derivative.  To get the minimum, use the direction for the smallest directional derivative.

Starting with a guess, calculate the function value at that point.  Then take a step in the direction given by Step 1 using the step size.  The variable “u” here is a component of the unit vector

x_n+1 = x_n + u_xh

y_n+1 = y_n + u_yh

At this new value, calculate the function value again.  Keep repeating this until the maximum or minimum is reached.

Limitation

The approximation is generally only as good as the initial guess.  Sometimes, a bad guess can lead to a solution that never converges.  In that case, a different value must be used in order to find a converging solution.  For example, if we have a fourth order polynomial function, a local extreme will exist.  As x approaches –∞ or ∞, y will do the same.  If I use the steepest descent and give a bad guess, my solution will go to ∞ instead of the local extreme.

Sources

1. Calculus by Gilbert Strang
2. Applied Regression Analysis by Norman Draper and Harry Smith