Tim Berners Lee recently raised concern that thanks to the spread of apps for newspaper delivery, more of the web is becoming inaccessible to search engines.
I believe he is right to be concerned and that this is part of a larger problem – that much of the web is already locked in walled gardens and in danger of being lost to future generations.
And that makes me think that even this is only part of a bigger problem still. Thanks to Tim et al, we started off with a fantastic opportunity in the world wide web, but we have really messed it up.
A few days ago, I wanted to find the opening times for a Chinese restaurant, a pretty everyday task. You might think with a mature web that I would be able to go straight to the website of the restaurant just by Googling it. In fact, the first few pages Google returned were all intermediaries, all getting in my way instead of helping.
Far from creating a nice streamlined web where you can do business direct with suppliers and cut out the middleman, we have done pretty much the opposite, creating layer upon layer of intermediary where none is needed.
If you don’t know the URL of the company, it can be near impossible to find it now, buried under hundreds or thousands of other companies hoping to use their company to squeeze commission or advertising revenue from your transaction. We badly need a ‘Google Direct’ or the like that allows you to search for the company and exclude all the other sites.
Walled gardens are a problem too. If material isn’t searchable to outsiders, its impact is limited and it is also in danger of being lost when the company owning the garden disappears or changes the system.
Proprietary software and standards make data more vulnerable to change. But the bigger problem is that they restrict people to a small area of the web.
A lot of people spend most of their web time inside Facebook for example, and see the online world via a corporately owned and controlled filter. iPad users also live in an attractive but mostly restricted world.
Some people want to make this way of being the norm, especially Facebook and Apple, of course, but it is hard to see how a cage is better than freedom, however gilded.
Social search will make it even worse and even Google is heading down this route, filtering what we see according to what they know about us to maximise their revenue. The web offered us freedom but it seems that we have chosen captivity.
A whole generation is growing up thinking that Facebook is the universe. It is like those Americans who never feel any desire to step outside of the US, imagining that everything worth doing or seeing is there.
I am not sure whether it is worse that we aren’t getting the web we should have, or that people seem happy with it, blissfully unaware of how they have been short-changed. They booked the Royal Suite but got put in the broom cupboard but because they don’t know what the Royal Suite looks like, they don’t know anything is wrong.
Even if we tried to make a new web to fix it, it could just happen again the same way. We need to make search mechanisms that let people see everything, even if they can’t fully access it without a subscription, and to be able to easily filter out intermediaries.
Google won’t be allowed to do the first of these and won’t even want to do the second, nor will Microsoft or Facebook. Regulation won’t work – it rarely does, so a technology platform that is appealing to all parties may be the answer.
All these competitors use the same IP so why not also the same overall platform for search? And if we can do that, then we can also look at data formats and make sure that we use systems that are OS independent and app independent so that we don’t lose everything when a company goes under.
If there is sufficient volume of data in a format, then making sure that the compression or conversion algorithms are absorbed into future standards would be basic common sense to most people so it isn’t too much to ask that the IT industry catches up and adopts such a common sense approach for everyone’s benefit. How this is done is anyone’s guess.
It will take a while, but it needs doing and quickly. At the moment we are relying too much on luck and goodwill and neither of those can ever be assumed to have a long life.