Ah well, not to be too precious, I thought it was time to put on my nostradamus hat and take another look at what the iPhone will look like next.

Let’s start with my last set of predictions. I said:

June 2009: tablet iPhone. This is where the iPhone and the Computer merge. This is 3G and it is the ubiquitous Internet device. Always on, this is the internet in your pocket, plus being able to work & play. This will have MS office or an equivalent. Expect tighter integration with iTunes (through which you’ll be able to buy TV shows & movies). Google will pay massive amounts to Apple for access to search. Expect a full software keyboard presented with a reasonable viewable area. First release June 2009. Sell ridiculous numbers

Stupidly, I’d forgotten about a part of my prediction number 2, in which I predicted Apple will need to build a network infrastructure like they did for music with iTunes, but for email management. I got that right:  they’ve done so with MobileMe. But what didn’t click for me is that they don’t need iPhone 3 to have the office suite as an app, they’ll just put it in the cloud.

MobileMe is a revelation. And the thing is, it is so compelling that people are actually paying for it. As a service. This is so mind boggling to me that I can’t say it enough. People, as in consumers, are paying for a software service.

And this is the essence of my prediction for iPhone version 3. iPhone 3 is the mobile front end to MobileMe. Suddenly we can have a reasonably dumb device. All the smarts are in the cloud. And I can have my version 3 iPhone in a year.

Oddly, this is what we were trying (miserably) to do with Bullant 8 years ago. Of course this was before ubiquitous javascript, so we had to develop a screen painter, but the concept was the same.

So what will Apple do in the next year?

Firstly, they’ll add services to MobileMe. Expect a word processor and a spreadsheet app. Also a way to at least view presentation files (powerpoint, keynote). Expect also a ‘Back to My Mac’ capability so that I can do some stuff (like printing etc) away from the office.

I still expect a larger form factor for iPhone 3. A fold out design most probably, where the thing is usable closed but has a touch screen both sides. Folded out it is a double wide touch display.

And the thing with Google? Will Google still be willing to pay Apple for all the search clicks if Apple are competing with rich internet apps? Still valid. In fact Google will love it – even though it blows Google’s office suite out of the water. Because Google aren’t trying to monetize (hate that word) their apps. They’re just desperately trying to make sure that the internet remains HTML. Because if other web formats (Silverlight, Flash) become the basis of Web 3.0, then Google’s search ad revenue disappears. By giving away free HTML based services, Google are trying to preserve the status quo. And Google will keep their apps and even improve them, as they don’t want the folks that won’t pay for these services going to non HTML formats.

And remember my other failed prediction: Apple won’t build chips as they’re too keen to look at an enormous cash reserve? I’ll bet the folks from PA Semi are right now working on ARM improvements for iPhone version 3. My days as a pundit are obviously numbered.

Unfortunately, the process seems to be awfully manual, as the dates keep getting screwed up. Anyway, I’m back-filling like mad, so I’ll keep my other blogs around until this has all the content.

I’ll have to work on moving the comments as well. Unfortunately I don’t have access to the raw data from the other blogs, so I’m not entirely sure how I’m going to do it. I’m hoping I don’t have to write a screen scraper.

We’ve experimented with a number of different approaches, none of which have worked well.

The current approach is the ‘cafe society’, in which some individual (whom I shall call ‘the sap’) is given a task. Theoretically the sap then calls together a group of people who have domain expertise and they produce a project plan and then execute on it flawlessly.

In fact what happens is the sap calls on people that they know will turn up and then tries to dump as much work as possible on to these people before giving up and doing the whole thing themselves.

The problem is accountability. Because the sap is not part of the line management of the team members, there is no benefit to the team members in doing the work. In fact it is quite the opposite, as their line management will be measuring them on the work that they do as directed by the HOD. The more work they do as an ad-hoc team member, the less time to do formally directed work.

I’m proposing a review process in our organisation to try to take this into account. I’m proposing that part of the debrief is to review the performance of the members of the team. And that this be taken into account in the 6 monthly review process. At the same time I’m proposing that the existing feedback mechanism for the team members to be able to report concerns be strengthened.

The review process would report on the overall outcome of the project and the individual participation within the project as well as issues arising. The brief to the line management is to look at the number of projects in which an individual is participating (spread too thin? Just a talker?) as well as their participation and project success rate.

Then just before the associates 6 monthly review process, a meta review would gather all line management together and review all project participants as a group. The results of the meta review are included in the associate review.

Any reason this won’t work?

I attended the ‘This Is Not Art’ festival last weekend. Great vibe, lots of cool stuff and a host of talks on interesting things. However – it was also a bit frustrating as almost all the talks that I attended missed the central point of their respective argument.

For example, the blog.

I heard that the blog was different to a website because:

• It is ‘interactive’ which I guess means that you can comment
• It is easier to write than a web page (well I guess they haven’t tried to use a third generation CMS like steer)
• It is easier to read than a standrd webpage (ie most webpage designs are crap)

All of this is missing the point of a blog. A blog is temporal. It has a history, a timeline, a context.

Bloggers (at least the ethical ones) don’t go back and correct blogs. You don’t revise and republish. The primary navigation for a standard blog is time – either via a ‘previous’ ‘next’ decoration or directly via an archive.

And this central idea gives the benefits above. It is interactive because it is fixed in a context. That later posts can contradict earlier and it is understood that they do. You can influence a blogger via a comment system because it is a historically recorded conversation.

It is easier to write than a web page (ad easier to navigate) because there are fewer choices. I don’t need to worry about where to post this rant because it will be posted under ‘today’. I don’t need to worry about if it belongs under /rants/philosophy/web or under /general/discussion/meaning/blogosphere. I just post it and everyone understands the primary navigation.

Of course the temporal nature of the blog is also its biggest weakness. Because I only have some simplistic keyword tagging tools, this post will lose a chunk of semantic information by not being located under one of the categories above.

Which is the subject for a different post.

I’ve just had a very impressive experience with my new hosting company.

Many moons ago I started writing code. And some moons later  (now I don’t want to paint myself as a grumpy old codger, but for those in the know I might point out that it was so long ago that my aunic handle was cy-au) I put together a few web sites. We’re talking late 80s here and these things were absolutely static

Navigation was done via includes, all code was written by hand and the idea that a web page might do something for the user… well let’s just say that at that point yahoo was some bloke’s link list.

Contrast that with the experience on dreamhost. I signed up, paid via google, and in the past 2 hours I’ve had 10 emails showing various status as it has:

• Registered a new (or actually re-registered an ancient) domain
• Setup http, ftp, webdav, mail, wordpress, joomla, mailman, mysql and jabber
• Provided a mechanism for me to add ‘one-touch’ software – being a bunch of open source goodness preconfigured and just waiting to be installed
• Provided a mechanism to manage all of the above and then some
• Provided a mechanism for me to set up 3000 email boxes, unlimited subdomains (though I’d like to see apache parse some of the subdomains I can think up) and will allow 200GB of storage space.

And did I mention it’s costing me about $10/month?

There is something scary in the economics of the web. The infrastructure is essentially costed at arbitrary amounts. I am paying (in my professional life) ungodly amounts* to host a single simple corporate website while I pay nothing for a far more sophisticated personal site.

How come the front end is at web 2.0 and the infrastructure is still feudal?

* And I have a high threshold of ungodlyness. I am talking serious ungodlyness – like several salaries ungodlyness. That’s ungodly.

We’ve known for ages that Moore’s law will break for single core processors. We’ve also known for ages that multi-core programming is really hard. And by hard I mean impossibly difficult. So when the future of silicon is multi-core, but the developers aren’t smart enough to take advantage then Intel has an issue.

So what to do?

If I were Paul (and I can call him Paul as I once saved him from serious embarrassment at an IDC keynote) I’d be looking very hard at projects such as LLVM. Using LLVM, Intel could concentrate on developing compilers that optimise the bejasus (technical term that one) out of everything including using hardware that may not be installed on the end user’s machine. LLVM comes along at run time and removes unnecessary static branches.

This would allow Intel to do some interesting things for their higher margin items such as motherboards. As they build the compilers, they could optimise for their own layout and specific implementations and then have LLVM strip for ‘less capable’ devices from other manufacturers.

As a side benefit, some of those optimisations might be to send types of computations to specific types of hardware without troubling the CPU. For example, what if the compiler had determined that a chunk of code could execute on the GPU? If a compatible GPU existed then it would use that branch, if not then use the CPU.

Of course, this kind of work would be brilliant for Apple, where they control the hardware, but I think they have other fish to fry.

Update: Colour me embarrassed.

Scott has just informed me that not only are Apple heavily involved in LLVM, but they’re developing a new c front end to it. I guess I should scrub out Paul Otellini’s name and put Steve Jobs. Apparently there’s a section of core graphics that is built using LLVM so that it can run on a bunch of architectures including older non-OpenGL optimised cards.

Seems a little small scale to me. What do you bet that the OS past leopard includes generalised LLVM switchable compiler optimisations.

I still think Intel could do similar things.

After staring at the numbers long and hard, I think he may be right. So – either Apple will get in to the hardware design business (and won’t Intel be upset about that!) or I’ll need to push the Tablet out a bit.

Now if only Apple would use some of its enormous cash reserves to buy a semiconductor company to develop a high performance embedded chip then I could have my tablet. But they love to look at their bank balance and gloat so I think it is unlikely.

Sigh.

Let’s start with the pundits. My prediction is that the general set of commentary will go down one of three ways.

Commentary one: Apple know computers, they don’t know phones.

Actually Apple are brilliant hardware resellers. They are very good at packaging together a bunch of commodity components and selling it for an enormous markup. They do this by building hardware that everyone wants to have. And they do that by concentrating on the experience.

The iPod is a great example of this. All the components was there for anyone to build a player with. Apple just decided to build an experiential infrastructure.

And if there is one thing that mobile phone engineers are terrible at it is an experience. I’ve generally used Nokia phones – generally because they have come free with my plan. I think I’ve had perhaps 5 phones in the past 10 years and I still couldn’t tell you how to find the calculator on any of them. Everytime I use the interface I have to discover everything anew.

These are obviously phones that are marketed by listing features. Matricies of little xs arranged in grids. I don’t love my phone. I’m going to predict that everyone will love their iPhone.

And let’s think about what it will need to do to succeed:

1. Make phone calls
2. Make you feel special for using it

The actual applications that come with the phone are almost immaterial.

Commentary 2: It’s not 3G

I was almost hesitant to separate this from the above, but I think it is important enough to sit on its own. Ready, I’m going to say something that is probably not at all contentious: There is no benefit to 3G on a phone.

There are a couple of local telcos in Australia that have 3G networks. And they’ve tried to market 3G services (TV, video calls etc) and nobody cares. There is absolutely nothing that is compelling about 3G on a small interface.

So the only thing that differentiates 3G from slower networks at the moment is cost. And not only in components but also in power consumption. I predict that Apple won’t have 3G in the iPhone until the version 3 revision in 2009. (See below).

The iPhone connectivity via GSM / Edge will be slow and problematic. But it won’t matter to people because no one knows a different experience. And the cool folks will have access to WiFi in many places so they won’t care.

Commentary 3: It’s expensive and it’s locked

This is the biggie. And I must admit this is troubling for me. $500 minimum plus a two year contract is expensive for a phone that won’t be touched by corporates until the version 2 revision (see below again). Apple do have a habit however of gouging early adopters, so don’t be surprised if they drop the price if it sells well enough. Note the ’sells well enough’ caveat there (firmly covering my arse). Apple must be taking a proportion of the contract spend, there’s no way they would be content with the $500. Therefore I’d argue that they have built in to the contract with their telco partners that the amount that they get per contract increases with popularity. I’d suggest that if the adoption velocity reaches a certain point (and I’m guessing it will) then they’ll drop the price by $100 or so within the first 3 months.

When Apple make their corporate play, expect the cost of the device to move from capex to subscription.

Predictions:

1. Everyone will want one and sales will be astronomical. There will be queues on the first day and Apple will not run out of stock.
2. ‘Unlock iPhone’ will be within the top 100 most googled phrase for the first 3 months after launch
3. The iPhone will drop in price by $100 within 6 months of launch. Probably January 2008 when the 4GB version is dropped and the 16GB model is released, but if sales are very strong by October.
4. Apple’s stock price will hit $220 by December this year (today’s open $85.60)

Now for the ‘If I Were Apple’:

Smart phones are not big in the US. Smartphone penetration in the US is about half that of Europe. And the brands are: Symbian, Palm, Microsoft and RIM.

Palm are a joke. Let’s ignore them.

Microsoft and RIM are in play because of Exchange. Take away corporate email and they’d be as much of a joke as Palm. Really the only decent smart phones at the moment are Symbian based. And even they are not compelling enough to buy one. I think the proportion of phone to smart-phone is currently running at more than 10 to 1, which is a pretty small proportion. This begs the question:

So what are Apple doing in this market?

I think they have a three step plan:

First, get the early adopters and premium buyers. These are Apple’s traditional market, and the reason that they do so well on margin. These folks will be happy with a user experience + phone + iPod in a single device. They’ll also be happy paying the premium to own one. Sell 1 million phones – primarily in the US. Release June 2007 (announced).

Second, replace the blackberry. This is going to take two things. Firstly very tight exchange integration and secondly a significant price drop. This is a corporate play so we’ll need management tools, such as VPNs etc. We’ll also need to open the market to other providers, but that can come a little later. Apple are also going to have to build infrastructure again (like RIM have done) to support push email. This will still be GSM / Edge, but will probably add GPS. This is the phone for the masses. Sell 20 million iPhones worldwide. Watch RIM’s share price drop somewhat. Release June 2008

Third, tablet iPhone. This is where the iPhone and the Computer merge. This is 3G and it is the ubiquitous Internet device. Always on, this is the internet in your pocket, plus being able to work & play. This will have MS office or an equivalent. Expect tighter integration with iTunes (through which you’ll be able to buy TV shows & movies). Google will pay massive amounts to Apple for access to search. Expect a full software keyboard presented with a reasonable viewable area. First release June 2009. Sell ridiculous numbers.

I’ll be introducing TDD and some agile development tools, but there is nothing in Agile that inspires a desire to help out your replacement. Agile is about fixing the problem in front of you. Most of the development teams that I’ve been involved with that wrote useful documentation had it as a culture. It is going to be difficult to inspire that culture in both new and existing developers at the same time.

So what I’m going to do is to build in a process of silent code reviews. The team will all get together and review a single piece of code. But the developer is not allowed to speak. If the in place documentation is not good enough for the rest of the team to understand what they were doing, then the documentation must be improved on the spot.

I’ve done the first round with myself as the developer and the team is getting into it.

Perhaps the chocolate biscuits as reward for best commentary helped

We had a simultaneous UPS failure at 2:33PM on the 22nd (which I am going to call H hour). This caused the ERP production cluster and storage to go down hard. Cause of the UPS outage is at the moment unknown. They seem to have been out for a little under a second. The effect was – how can I put this – bad.

Firstly, I should mention that we have had fairly significant outside assistance in setting up our production hardware. Everything is in a redundant configuration. Battery backed up disk controllers, RAID up the wazoo. A nice setup that is – or should be – resilient to failures.

Unless of course it has been mis-configured.

A little history. About 8 months ago (6 months before I joined the company) there was a disk outage on the storage cluster. Not a problem, except that the hardware support vendor (who I’m not going to name as we haven’t decided on our litigation strategy yet) could not source appropriate replacements as the storage cluster is pretty old. Instead of replacing the single drive that had failed, they replaced one of the controllers and the set of disks on that controller. They then re-setup the RAID 0+1 configuration across the old and new controller.

This took them about 15 hours to do and caused a significant outage. As you may imagine, this cost them (and us) a large amount of money. Us in lost sales and them in contract penalties. I would imagine that the on-site engineers were under a fair amount of pressure. And probably because of this they missed something.

The new controller’s battery was not installed. And the monitoring system wasn’t updated to register a battery failure on the new controller type…

So when the power outage occurred, one half of the RAID 0 had the cache written to disk and the other didn’t. And this continued across the RAID 1. And the result is that about half our filesystems wouldn’t even mount. Enormous corruption in the database tables that were open.

It took us about five hours to work this out – mostly because the build documentation was not up to date. We’d spent a lot of time trying to understand the source of the corruption and to fix it with volume management tools and (eventually) fsck and related tools. It was about three hours in that we decided that we should simultaneously start work on a recovery plan that did not include data from the un-mountable drives. Unfortunately, the limited resources available meant that we had to exhaust the easy options first.

Our recovery scenario therefore was fourfold:

1. Install a battery into the controller and update the monitoring system
2. Continue to try to recover the un-mountable volumes
3. Recover the previous days backup onto the standby storage
4. Work out if we could recover the transactions between the last backup and the failure

A little background. We have three major types of transactions:

1. Customer management (add, change, delete)
2. Order management (add, pick, ship)
3. Inventory management (bulk movement, allocate stock to order)

We first set out to identify what information was critical to recovery. We decided to ignore customer changes and deletions, as the number would be small and not business critical.

Bulk movements have physical implications (pallets) so we had already asked Operations to start identifying these. And the carrier had not yet turned up to collect the 1,400 or so orders that had been picked on the day. By H+1 we had asked Operations to send the carrier away empty in case we needed to use the physical orders as an information store.

Then we tried to identify the data that had survived. Due to an idiosyncrasy of the storage design, it was more efficient to have multiple mount points and segment our data across these. This meant that by losing about half the file systems, we still had many of our critical files. Unfortunately not the primary orders file, but we did have a transaction audit file and an inventory file. We also had a payments log file for all credit card transactions. Our customer file was also gone.

So, by H+6 we had a plan. The development team started to write code to identify orders that were in the current transaction logs but not in the restored transaction logs. Also to identify stock movements and orphaned payments. Unfortunately the payment transaction did not have the customer attached to it, but we could make an educated guess by comparing payment amounts, times and credit card names. We could also link inventory movements (stock allocation) to the times of order entry. So by H+11 we had a suite of order recovery tools ready to test.

We trashed the orders in the order file in our QA system for June 20 and started to test the recovery process. Unfortunately this was not a good test as we could not (at least without recovering the trashed files from backup) ensure that the processes were consistent. However it gave the QA team some hints as to what to look for when the real data was available.

Unfortunately the data recovery was going slowly. Our tapes are stored off-site and the security company had problems sourcing keys (arg!). By H+9 our recovery started and we began the process of restoring to offline storage. The restore took 4 hours. Then there was a further 2 hours of replaying transaction logs to get the offline storage to a consistent state to start our testing.

Testing began at about H+15, and we had the usual bugs. While these were being identified, the QA team started to look at the resurrected orders to see if they looked reasonable. This was a critical test as if the approach that we had used was invalid then we would have no choice but to discard the days trading.

Sunrise was about 7AM (H+16). A misty morning.

H+17 Fortunately, the results from QA looked consistent. The development team had a new release, and while this was being tested, we trashed the current production storage, inserted the battery and rebuilt the RAIDs.

H+19 we had moved the data from offline storage to online storage and started recovery.

Our DR plan (which I was in the process of re-writing as we were also in the middle of a SOX audit), mandated a communications plan that had been put in place at the outset. We had company wide updates each hour and the president and cfo were woken at regular intervals (they probably didn’t need to be, but they hadn’t yet signed off on the new DR plan). By H+18, we had most of the company at work, so we also had PA announcements on progress. A lot of staff were diverted to customer service lines to handle the increased load.

H+20 the production system is rebooted and all partitions are visible. Disable user logins to the system to avoid any stupidity.

H+21 our data updates were complete and we had a lit of problematic payments and orders. That is, payments that we could not match with orders. We also had a set of orders for which there were no customers (new customers that must have been created on the day). We then set off a data consistency check prior to our final data manipulations.

H+23. Everything is set to go back into production. We incremented the order and customer table’s primary indexes to leave room for any orders or customers that we are able to manually recreate. We also gave the list of orphaned orders (338) and the list of customer referrals (78) (due to our business model, we generally know who a new customer was referred by as there is a commission system and the payment of the commission is audited and therefore we know who referred a new customer even if we don’t know who the new customer is). Customer service then began the process of ringing customers and finding out who they had referred.

We also printed the list of orders for which we could not identify payments and passed that to Customer service to verify (27).

We also printed a list of the orders that we had successfully rebuilt (1,283) and asked Operations to go to their picked list of orders and identify which had been processed already (to avoid double shipping). Operations found about 200 rebuilt orders that had already been picked. These picked rebuilt orders were also used to give us confidence that the item lists for an order were correct, as we also had Operations do spot checks on 30 of them at random.

We also gave operations the list of orphaned orders – as this would give us customer details that we could use.

Operations were then able to identify the orders that were left over – by definition, these were orders from previous days. We could bulk update these order to status ‘picked’ and ship them (we’d lost the changes to order status as they’re on the orders file).

We then briefed each of the warehouse supervisors (5 warehouses) on all the extra checking procedures that they would need to ensure that orders were not double shipped. 

H+25 The restored ERP back into production.

H+25.1 sleep.

Results after 1 week

1. We have two orphaned orders. Customer service have made 823 calls.
2. We have about 80 double shipped orders that we can identify. All customers either called us or were contacted. Most have been returned.
3. No clue about the UPS. Have scheduled an outage for a UPS engineer to check both devices.

What we would do differently:

1. Get more business involvement. Particularly from operations as they ended up having nothing to do for the first 20 hours and then incredibly rushed for the next 10. Also this would have freed up ICT resources as the local operations leadership could have briefed the remote warehouse management.
2. Our outsourced support teams have been allowed to be slack. SLAs were not met by Technical support, Tape Security service or Vendor support. We need to ensure that these services are kept to account and regularly tested.
3. Do more scenario testing. Play ‘what if’ games with the team. We started the process with a very small team. Had we begun with the entire team we may have been able to do more in parallel.
4. We’re short in broad technical personnel