The Next Internet, How to Build It?

-

Researchers believe it could be time to build a successor to the internet.

-

With online crime rising and traffic increasing rapidly, some academics believe it is time to have a serious discussion about what succeeds today’s internet.

-

“There’s a real need to have better identity management, to declare your age and to know that when you’re talking to, say, Barclays bank, that you’re really doing so,” said Jonathan Zittrain, professor of internet governance and regulation at the Oxford Internet Institute.

-

Many ideas revolve around so-called “mesh networks”, which link many computers to create more powerful, reliable connections to the internet. By using small meshes of many machines that share a pipeline to the net instead of relying on lots of parallel connections, experts say they can create a system that is more intelligent and less prone to attack.

-

Millions of pounds are being pumped into academic research, bringing to mind the early days of computer networking such as Arpanet.

American computer scientists in the past relied on government money, they have had less support from the Bush administration, which has substantially reduced funding and channelled money instead into homeland security projects.

-

“There’s a risk in doing completely blue-sky research that fixes a problem but then turns out to be useless at the things the internet did well,” he said. “There aren’t that many who can do a clean-slate design – and you don’t want to throw the baby out with the bathwater.” [ source ]

Read Full Post »

YouTube Architecture

YouTube grew incredibly fast, to over 100 million video views per day, with only a handful of people responsible for scaling the site. How did they manage to deliver all that video to all those users? And how have they evolved since being acquired by Google?.

-

Platform:

* Apache
* Python
* Linux (SuSe)
* MySQL
* psyco, a dynamic python->C compiler
* lighttpd for video instead of Apache

-

What’s Inside the Company ?
The Stats 

* Supports the delivery of over 100 million videos per day.
* Founded 2/2005
* 3/2006 30 million video views/day
* 7/2006 100 million video views/day
* 2 sysadmins, 2 scalability software architects
* 2 feature developers, 2 network engineers, 1 DBA

-

Recipe for handling rapid growth

while (true)
{
identify_and_fix_bottlenecks();
drink();
sleep();
notice_new_bottleneck();
}

This loop runs many times a day.

-

Web server

* NetScalar is used for load balancing and caching static content.
* Run Apache with mod_fast_cgi.

* The Python web code is usually NOT the bottleneck, it spends most of its time blocked on RPCs.
* Python allows rapid flexible development and deployment . This is critical given the competition they face.

-

Lesson’s Learned

* Creative and risky tricks can help you cope in the short term while you work out longer term solutions.

* Know what’s essential to your service and prioritize your resources and efforts around those priorities.

* Pick your battles. Don’t be afraid to outsource some essential services. YouTube uses a CDN to distribute their most popular content. Creating their own network would have taken too long and cost too much. You may have similar opportunities in your system. Take a look at Software as a Service for more ideas.

* Keep it simple! Simplicity allows you to rearchitect more quickly so you can respond to problems.

* Sharding helps to isolate and constrain storage, CPU, memory, and IO. It’s not just about getting more writes performance.

* Constant iteration on bottlenecks:
- Software: DB, caching
- OS: disk I/O
- Hardware: memory, RAID

* Have a good cross discipline team that understands the whole system and what’s underneath the system. People who can set up printers, machines, install networks, and so on. With a good team all things are possible.

[source]

Read Full Post »