Saturday, April 05, 2008

System update kicks Heathrow baggage system offline | IT Project Failures | ZDNet.com

This is probably one of the highest profile computer/infrastructure failures in the last few years

The error occurred, “[A]fter an update to the system made it reboot….” Additionally:

BAA told Computerworld UK that a computer software upgrade, conducted Monday night, was the cause of the failure. Despite testing the software, the “glitch crashed both baggage sorting machines….”

System update kicks Heathrow baggage system offline | IT Project Failures | ZDNet.com

Computer glitches were part of the issue, but the key problems appeared to be logistics.  Staff couldn't find parking spaces in the morning, and couldn't log in to the system.  After that, it's no wonder they were having a bad day.

But British Airways added that staff log on issues were also part of the problem. “The log on system initially didn’t work,” a BA spokesperson told Computerworld UK. “There was an issue with that for some reason.”

The airline said that “the car parking problems meant the staff were not in place.”

Who designed the baggage system? 

The high tech baggage system, was heavily featured in the Terminal Five pre-opening publicity. Created through 400,000 man hours of software engineering, it was designed and supplied by Vanderlande Industries in conjunction with IBM technology and Alstec, who are operating the system.

Didn't George from Seinfeld work for Vanderlande Industries?

British Airport Authority (BAA) is ultimately responsible for the operation of the terminal and it's technology.

The technology investment is immense. T5 boasts 17 kilometres of baggage conveyor belts in a system designed to handle as many as 12,000 bags an hour. Over 400,000 man-hours have gone into developing the system's software, which has the ability to prioritise late bags. The terminal has involved 180 IT suppliers, runs 163 IT systems, manages 546 interfaces, more than 9,000 connected devices and 2,100 PCs. The building has 96 self check-in kiosks, 54 traditional check-in desks and 90 self-drop baggage depots.

This release from February 14, 2008, seems to be a preclude to the disaster.

Airports authority BAA has begun looking for a service provider to work with it to help develop consistent IT systems at Heathrow and across the authority’s airports.

It appears that 50 of the IT staff for BAA transferred to IT services company LogicaCMG, when BAA signed a 5-year outsourcing contract with them.

LogicaCMG will manage more than 400 different applications, including Oracle Enterprise Suite, as well as aircraft stand allocation and staffing.

Other components of the luggage handling system include wireless connectivity.

The network will have 800 access points connected to two separate redundant Aruba Mobility Controller wireless switches, in two separate locations in Heathrow. The APs used are Aruba's 802.11abg devices. The new 802.11n specification was too risky and early at the time the network was designed – "but we can just clip in 802.11n when we need it", says Newbold.

The wireless LAN will be used for the baggage-handling system. Engineers with laptops and PDAs can manage the infrastructure and check barcodes on luggage anywhere in the building.

However, Newbold made a deliberate decision not to focus on making the WLAN "seamless". He explains, "It's not a contiguous medium, it's not connection oriented." For instance, when staff step from the shopping mall to "back of house", it is very important that they experience a transition, and some resources are not available the other side of the line. Sometimes this is enforced by excluding radio signals from other parts of the space. "There are a lot of Faraday cage materials in the building."

That would tick me off if I was a baggage handler having to setup connectivity every time I leave an area.

Here's a question that shouldn't have to be assumed... it should be proven in a lab when dealing with this kind of scale.

Will it work?

Despite careful planning, Newbold can't be sure the WLAN will work straight away, because the building has not yet seen a full complement of visitors. "Optimisation is the biggest challenge," he explains. On an average day, the building expects to see 53,000 people, all of whom will have a definite effect on the propagation of radio signals, since they are mostly made of water.

Could login issues be as simple as wireless connectivity?

And with ever-increasing numbers of people wishing to fly, the things that can turn airports into nightmares are getting bigger and more frightening.

The line above was written almost 3 years ago.

Passengers arriving very early, say seven hours before their scheduled departure, will also be able to check their bags into the system rather than wait for check-in to open and add to the queues.

Doesn't sound like Just in time baggage check-in to me...

I had my luggage "delayed" somehow on a trip to NY last year.  It was shipped to my hotel 7 hours after the flight, however by then I had already purchased a new outfit.  (So I guess it wasn't really a bad thing. )

There was one other person with me waiting anxiously at the baggage carousel for bags that would never show up.

That was only 2 people and the person manning the desk for lost luggage still looked stressed.  I wonder how the staff at new Heathrow dealt with 12,000 "delayed" bags.

If my reaction was to go out and buy a new outfit and toiletries, then I guess that losing people's luggage is one way of indirect economic stimulation.