专访Twitter工程师：SNS产品发展途中的那些事儿

Nick Kallen原本是一位软件咨询师，同时也是Rails3框架的基础Arel、NamedScope、分布式缓存框架Cache Money和JavaScript行为驱动开发框架Screw.Unit等多个开源项目的作者。两年前他应邀解决Twitter的可伸缩性问题，并由此加入了Twitter的技术团队。对于现在Twitter的规模而言，可高效查询的可伸缩式数据模式是最为紧迫的任务之一，而Nick现在的主要关注方向之一就是一个通用的分布式数据库。

Nick的Twitter是@nk。

Twitter的发展早期

做为流行度增长最快的SNS服务，Twitter常常被拿来与2004年创建的Facebook作比较。与Facebook日益剧增的新功能与新应用形成鲜明对比的是，Twitter在诞生之后的三年内一直鲜有新特性的加入，直到最近这一年才逐渐加入了图片支持、地理位置显示等新功能。根据Nick的介绍，早期的Twitter团队对Twitter的定位是以“最小化服务”为目标的，当时团队认为添加新的功能将会对Twitter“最小化服务”的定位产生不好的影响。

不过对当时常年被Fail Whale所纠缠的Twitter团队而言，的确也抽不出足够的人力和精力放在开发新功能上面。Twitter诞生后的三年时间中，用户数量一直飞速发展，数据量骤然攀升。Twitter最开始以LAMP架构（Linux+Apache+MySQL+PHP）创建，这个系统很快便不堪重负。Nick十分坦诚的向我们描述了Twitter团队早些年的状况：

“系统的可伸缩性一直是如此紧迫的任务，导致我们的工程师们几乎没有什么机会考虑新特性方面的事情。所有的工作就是维持服务运作，以及将其扩展。……在一开始的LAMP架构下，通常就是一个MySQL主数据库来做垂直伸缩。这当然不是不可伸缩的设计，不过无法满足我们的需求。”

Twitter的可伸缩性

在用户接触不到的后台，Twitter在四年间其实一直经历着很多改变。比如在应用层，服务器后台处理这一块，Twitter在08到09年间使用Scala语言重写了后台应用，大大增强了多进程异步处理的能力，提升了性能。

这期间最大的变化，应该就是数据层的变革了。2009年开始掀起的“NoSQL革命”在Web领域造成了极大的反响，Twitter也在今年年初开始对其Tweets数据类型引进新生的NoSQL数据库Cassandra。而根据Nick的介绍，Twitter早在一年半之前便开始专注于高效的分布式数据存储解决方案，该方案的重点在于Partitioning策略，即数据的分割。

“以前我们将所有的数据和服务存储在一个组件上。而数据分割的做法，就是将数据分割成小块，然后存储在多个组件之上。因为大块的数据被切割成了小块，我们就可以并行的、以小任务的方式完成查询和操作的工作。无论是我们开始发一个推，还是我们开始一个社交图（social graph），还是我们开始一个搜索，每一个主要组件都在过去的一年半中通过不同的策略完成了数据分割。这就是现在Twitter可伸缩性的实现。”

近一年多持续增长的时间内，Twitter Fail Whale的出现次数已经降低了很多，应用层与数据层改进可谓是卓有成效。

Twitter API与新功能

相比Twitter的流量和用户数，现在的Twitter技术团队仍然显得人手不那么充足。不过Twitter有一个很有意思的地方，就是在于其第三方应用的流行。2009年7月的统计显示Twitter的第三方桌面应用、手机应用、Web应用和浏览器扩展总数刚刚过万，而截止到现在，这个数量已经超过了十万。虽然Twitter的核心功能一直没什么重要改变，但在全世界上万开发者的摆弄之下，这个社交平台显示了惊人的生命力与创新能力。像是TwitPic图片存储服务和iPhone客户端这种应用都是来自第三方开发者，而这与Twitter的开放API是密不可分的。可以说，Twitter的大部分活跃和创新都应该归功于它的开放API。

Twitter开发者大会

对于Twitter API的由来，Nick做了一点简单的介绍：

“其实一开始的API开发，就是因为一个以前在Twitter的工程师离职去了德国，而他想要把Twitter集成到自己的聊天机器人里面。最初的API就是为了这位老兄的小玩具而设计的，不过我们很快就看到，将有更多人使用API来创造更多的东西。因此我们在很早的时候就开始在API上投入精力。”

和Facebook一样，Twitter对于开发者社区非常关注。Twitter刚刚在4月14日和15日举办了Twitter开发者大会Chirp，刚好在Facebook的F8开发者大会之前一周。Chirp大会上刚刚公开了Twitter开发者网站的上线，相信对于Twitter开发者而言是个令人兴奋的消息。

更多的变革

正如同Nick所介绍的那样，Twitter已经发生了转变。与之前的“最小化”定位不同，Twitter已经开始积极的推出新的特性。而对于开发者社区，Nick也表示了自己的期待：

“开发者社区的一大挑战便是，我们构建的产品核心功能要如何通过API来实现更多的创新——那些我们将不会在核心中包括的创新，而不仅仅是Web的一个替代品。程序使用API的方式和人使用网站的方式是有着极大区别的。程序总是不停地问Twitter：有更多数据么？有新的信息么？如此这般。不过人是不同的，他们在一天当中的特定时间段查看页面，比如午饭的时候。所以很有意思的是，所有的API使用都很同质化——功能相似，速度很快，重复性强。如何在这种方式下进行高效的访问，是工程师需要思考的问题。而每个用户不同的、不规范的使用习惯，也是另一个需要思考的问题。”

附录：Nick Kallen专访文字整理

（右为Twitter系统工程师Nick Kallen，左为51CTO开发频道编辑杨赛）

NK: I joined twitter about 2 years ago. Originally I consulted for them to help with scalability issues, and I really enjoyed working there. They wanted to hire me, they made me an offer and I accepted, that’s how I originally joined. I believe there is 180 employees now, approximately. As for how many engineers are there, I think there is about 40% or 50% of engineers, so about 70 to 80.

NK: There is a lot of reasons why twitter has in the past been cautious adding new features. For the first couple of years of twitter history, and until recently, scalability has been such an urgent concern that there hasn’t been as much opportunities for the engineers to work on any new features, they’ve been so busy keeping the site up on, making it scale. I think also early on, twitter is sort of a minimal service, I mean, many people contrast it to Facebook. Facebook is a rich set of features like photos, all sorts of things. And traditionally twitter has been very minimal, hasn’t added like extensive conversation functionality abilities like photo features. And so, the culture for a while, we have been reluctant to add features, which distracts the minimal of twitter. I think that is changing now, though, I think we are pretty aggressively adding new features, and riching new feature sets has been rather experimental. So, vary to the minimalism we used to be.

NK: Well it definitely wasn’t designed for scalability from the beginning. It was designed using kind of the traditional LAMP style – Linux, Apache, MySQL, PHP architecture – usually the single MySQL master database, vertically scaled. That is kind of how the original version of twitter was architectured. That is definitely not unscalable design.

Last year and a half, we were focused on basically partitioning strategies for our data storage. That means, instead of storing all the data or service in one component, you take that data and divide it into small pieces, and you store it across multiple components. So you can answer queries and manipulate it in parallel and in smaller jobs, because you’ve take a huge amount of data and divided it into pieces. So every major components, from how we start tweets, to how we start a social graph, to how we start search indices, has basically been partitioned using different strategies over the last year and a half. That’s what makes twitter scale now.

NK: Originally the API was developed because, when the first engineers of twitter left and moved to Germany, and he wanted to integrate twitter with an IRC bot, and the original API was designed to support him doing his little toy, and it quickly became apparent that people could have created things using the API, so we early on invested on the API functionality. That gives twitter a main advantage since we have been a small engineering team for a long time, and by opening the API we allow other people to build core functionality for us, an obvious example would be twitpic, where we didn’t have the resources to build photo storage/services, because there weren’t enough engineers. But by having the API, those core services could be built by other people, including like an iPhone client these days.

The challenge in the community now, though, is as we are able to build central parts of the products for people to have more creative uses of the API, and use it for not just as an alternative to the web, but creative kinds of things that we are not going to build into the core product. For scaling, APIs vs. the web, there is a big difference between the way software queries APIs vs. the way human beings use the websites. Programs would constantly pull – they keep pinning twitter, can I get more data, is there anything more recent, etc. The human beings don’t behave that way, they check it a few times, during lunch or something like that. So the interesting thing about API usage is that it’s very homogeneous, very similar, and is very high velocity and repetitive. And so you need to engineer your system to support that style of access efficiently. And that’s a different problem in supporting the kind of varied and irregular use cases of human beings.

51CTO专访Nick Kallen视频请见下一页

视频采访实录