May 17, 2011

Thoughts on Twitter's rate limit

This weekend I read an article about Twitter that, among other things, mentioned the limit in the number of tweets per user and the restriction that it creates on timeline searches. It made me think: really? Come on, it's just 140 characters! I have some space left in my drive, if they need it! :P

According to Wikipedia, Twitter has today around 200 million users that together post 65 million tweets a day, with a limit of 140 characters per tweet. This way, I decided to do some math to find out how much storage space this number of daily tweets requires and the first step was to learn how the 140 characters are counted.

Counting characters

Basically, Twitter uses the NFC form of the text to count the length of a tweet, which favors the combined character form. Special characters are represented by one codepoint, that is encoded as two bytes in UTF-8. You can find a lot more detailed information about it following their tutorial on counting characters and the references in it.

The math

Assuming the worst case scenario, in which all the 140 characters used and are special ones, a single tweet would take 280 bytes of storage space. If 200 million users generates 65 million tweets a day, they require almost 17GB of disk space a day. If a user is limited to post 1000 in their timeline, that means the 200 million users require something around 500TB of data.

Conclusion

Twitter has to manage 500TB of data, of which 17GB is daily modified. Considering that 75% of the requests to twitter.com are API calls, that means a lot of work on querying all that data. OK, leave my drive out of it!

Do you feel that this isn't much and they should raise the limits? Follow me on twitter and make me post more! :)

May 12, 2011

Integrating Android App Inventor apps with DB2

In the end of my last post, Easy Android Development with App Inventor, I was wondering about App Inventor integration with external data from different databases. How about using the TinyWebDB component for that? IBM recently released a new version of DB2 Express-C, Fix Pack 4, so I think it would be a good idea to try it out!

DB2

DB2 Express-C is the free version of DB2, IBM's database server. It's free for development, production and even distribution. The new features introduced in this version, as well as the ones introduced in previous Fix Packs, are described in What's new with DB2 Express-C 9.7.4, Raul Chong's article on developerWorks.

If you run Ubuntu and feel like trying it out, I recommend reading Installing and Testing DB2 9.7.4 on Ubuntu 11.04, where you can find tips on going through the installation process.

Sinatra

As I described in the last post, both TinyDB and TinyWebDB use a simple key-value based database concept. There are no different tables, extra columns, functions or queries. Given the key, the API allows the user to store and retrieve values. When storing a value, if the key doesn't exist, a new entry is created. If it does exist, the old value is overwritten. Simple like that!

That way, it's impossible to just connect App Inventor with a DB2 database, there has to be an interface for that. An easy-to-implement alternative would be a simple Sinatra app! Sinatra is a "DSL for quickly creating web applications in Ruby with minimal effort". It allows developers to quickly write Web services, which are very useful for an App Inventor/DB2 interface, once TinyWebDB component sends REST requests to a server, that can be specified, and gets JSON objects back with the appropriate content.

Implementation

Here's my idea: put up a DB2 instance, create a Sinatra server mapping Web services to DB2 operations and adding a TinyWebDB component to an App Inventor application, setting the service URL to point to the Sinatra server. Sounds complicated? Well, it isn't, keep reading! :)
  1. Setup DB2

    Get to your DB2 instance and create a new database, called APPINV. In this database, create a table DATA with two String columns, KEY and VALUE. To do that, launch a CLP and issue the following commands:


  2. Setup the Sinatra server

    First you need to install some gems, as you can see in the first lines of the code below. They should be pretty straight-forward to get. You may have issues with the DB2 driver gem, ibm_db. If you do, check the tutorial Installing Ruby on Rails and DB2 on Ubuntu 11.04, it's definitely going to help you out.


    After that, you'll have to type the correct password in the the IBM_DB.connect command and launch the server:


    This should launch a new server listening to 0.0.0.0:4567, which is where you should point your App Inventor app to!

  3. App Inventor application

    Finally, go to App Inventor! Get one of your existing applications, or create a new one, and launch the Designer interface. Once there, you can find the TinyWebDB component (still) under the 'Not ready for prime time' section, on the left panel. Drag it to you app, select it and, in the Properties column, set theserviceURL to http://0.0.0.0:4567.

    App Inventor: TinyWebDB configuration

After following these 3 steps, you should have your TinyWebDB working with your DB2 instance! This is a really simple example, but it's definitely a good way to integrate your app with existing DB2 databases. What you have to do next is customize the Sinatra server to interact specifically with your data.

That's it! If you have issues playing around with this tutorial, write a comment, I can help with it and give more details.

May 4, 2011

Easy Android development with App Inventor

Dear diary,

lately I have been messing around with App Inventor. It's a really cool Web IDE, maintained by Google, that allows non programmers to quickly and easily build and deploy Android applications on their phones. Check their short video presenting the idea.

The App Inventor Designer is a HTML-based drag-and-drop interface that offers a bunch of different components. From the basic ones, such as buttons, checkboxes and labels, to camera integration, media playback, sensors integration and local and remote databases components. In the 'coding' perspective there is actually no code. All the logic is put together using a intuitive and easy to use block diagram, called Block Editor, which is based on the Open Blocks Java libra ry.

Google provides good support, including a good reference documentation and a number of tutorials that walk you through getting started with App Inventor. This information can be found on the Learn section page.

Among the advanced components, you can find TinyDB and TinyWebDB, which, as the names sugest, are local and remote database components. Both of them implement the same functionality, a simple key-value based database and the user is able to store and retrieve data. The next pictures show the blocks available in the Block Editor, representing the operations for them.

TinyDB operations (synchronous)

TinyWebDB operations (asynchronous)

The basic difference between them is that the remote database, TinyWebDB, is implemented asynchronously, in a way that the application doesn't have to wait for remote answers. Therefore, there specific blocks to capture those answers, as shown below.

TinyWebDB remote events

This way, the block

    call TinyWebDB.GetValue triggers when TinyWebDB.GotValue

and the block

    call TinyWebDB.StoreValue  triggers when TinyWebDB.ValueStored,

assuming that the operations were executed successfully. If not,

    when TinyWebDB.WebServiceError is triggered.

For the server side, the actual database, Google provides a simple Web service written in Python, that runs on the App Engine. Instructions to create a custom Web service are provided, but for testing purposes, there is also a public and free version available.

Now, how about using different databases to integrate an App Inventor app with some other existing applications and data sets? :)

October 19, 2010

CloudCamp Toronto

Dear diary,

today I subscribed to the Toronto's CloudCamp! Acording to the official website, it is an unconference where "early adopters of Cloud Computing technologies exchange ideas". Sounds pretty cool, I'm in! :)


There is no agenda for the event! OK, there's an opening session, but after that, all sorts of topics are discussed in an open format, without scripts or preparation, and attendees are free to walk by and find something they consider useful.

I've been diving into cloud computing lately, playing around with Amazon EC2 instances and mobile platforms, using DB2 in the cloud as a persistence layer. This might be a good learning opportunity! The CloudCamp will take place next Tuesday, so be sure to follow me on Twitter to stay tuned!