My realization with synchronization

Recently I had to do major code refactoring on a component to make it thread safe and also to improve its performance. The component is a persistence layer that is capable of persisting and querying data. The component was already thread safe but the synchronization levels were on rather higher level. Older synchronization levels would block the query if the component is already executing some other query. So despite being multi threaded, queries executed from multiple threads will only execute one after the other. What we wanted now was to be able to execute queries simultaneously. Regardless of how many queries are executed in parallel. Two queries executed in parallel should block each other at rather micro level (like record level).

This component involves extensive use Java Collections. And at most of the places (while querying) you need to copy from these Collections. A particular Collection can contain more than a million object sometimes. So copying naturally takes up a lot of time in this case. The bad part is you have to synchronize on the Collection for the time you are copying it. So you are stopping any inserts and updates into the database (add, remove, on Collection).  When you need to synchronize access to the Object you always have to synchronize on all accesses to the Object. Similarly you have to synchronize all the actions on the Collections. To make the synchronization simpler, one can always use synchronized versions of Collections. You can obtain synchronized version of the underlying Collection instance by using methods Collections.synchronizedXXXX(). These synchronized versions of Collections can get rid of hassle of explicitly synchronizing on every access to the Collection. But there is a thing to remember with the synchronized Collections. Wherever you obtain any sort of Iterator, you need to externally synchronize the Collection. If you iterate on Map.keySet of synchronized map or you iterate on a synchronized Collection, you must externally synchronize the Collection (More explanation of why, coming later)

So in our case when I am copying from the collection I am actually iterating on the Collection and adding items to the other Collection. This access is synchronized. Which avoids any other access to the underlying Collection. Which means only one thread can iterate on a Collection at a time (There can be multiple iterators on a Collection at a certain time. But then we assume that no one modifies the Collection). To avoid all this blocking iteration, I came up with a rather stupid idea of using a synchronized List instead. The idea behind synchronized List was to avoid iteration (Remember you can do index base access to the List) So,

Collection src = Collections.synchronizedCollection(new ArrayList());
Collection dest = new ArrayList();
synchronized (src) {
        Iterator itr = src.iterator();
        while (itr.hasNext()) {
                 dest.add(itr.next());
        }
}

will be replaced with

List src = Collections.synchronizedList(new ArrayList());
List dest = new ArrayList();
for (int i = 0; i <= src.size(); i++) {
         dest.add(src.get(i));
}


With this approach, the synchronization is even finer now. We dont lock the Collection for all the time we spend iterating it. But we just lock it for the period we are doing src.get(i) operation. This means individual List.get() operations block each other instead of whole iteration process blocking the other one. For a while I considered this as a fabulous idea. But if you have noticed, we are breaking synchronization here. We can get into all sorts of problems using this approach.

For example:

Thread 1 : starts iterating List calculates List size as 10
Thread 2 : Removes an item from the List
Thread 1 : Reaches step List.get(i) where i=9. This will result into ArrayIndexOutOfBoundsException.

Since Thread 2 has removed one object from list by the time Thread 1 reaches 9th iteration of its for loop, we have broken the synchronous access to the List.

This lead me to an obvious conclusion that, Iteration is one logical operation on the Collection and it should block to all add, remove, get operations on corresponding Collection.

Well, thats not all! This rule can be generalized for all the objects. If you are making certain object as completely synchronized internally then the same rule applies to all such objects. Any logical operation to such object should block other logical operation (Maybe not always but most of the times). I encountered the same problem with a custom object. Which is suppose to a database index. I tried to synchronize on the entry level in index. The entries are actually stored on a Map. IndexEntry is mapped with its key. This did not help me because by the time i get an IndexEntry (Map.get(key)) out and as I operate on it. Some other thread can completely remove the entry from the Map (Map.remove(key)). Then all the operations by original thread on this IndexEntry are invalid. Same principle of the logical operation applies here. Each logical operation like read, add, update on index should be synchronous.

The other catch with synchronization is, Anything going out of scope of internally synchronized object needs to be externally synchronized. As we saw in iterators of synchronized collections. Even if the object is internally synchronized, if we return part of it as reference (internal structure, object), this part breaks the synchronization limits of the object and then needs explicit external synchronization from the user of this part. Most of the times we can avoid this scenario by making defensive copies of the internal structures before returning. But sometimes this can hit your performance.

Quite a long one! What do you guys say?

kdesu, su and sudo

Recently I upgraded to SUSE 10.2 First thing I did after install was configured sudo. I find it handy when doing operations requiring root privileges. But I always faced problems running GUI applications which require to connect back to X server. For example sudo /sbin/yast2 would always revert to the text based yast in suse, and others would give up with error. After I had done my installation I  tried running yast (this time clicking on a link in launch bar.) It showed me the routine pop up asking for root password. And to my surprise, root password wouldn't work. So I typed in my password and Bingo! it worked! Now I was puzzled why would it not accept root password (despite asking me for root password) and would succeed using my password.

So I looked up for the link launching the yast. I found out that the application which asked for root password in GUI mode was kdesu, and the command looked like kdesu /sbin/yast2. After reading brief documentation of kdesu, I learned that kdesu is kde equivalent for su. But then why would it not take my root password and still accept my password? Then I realized that I configure my sudo with targetpw attribute disabled. This will ask me for my password instead of root password to get root privileges. So for my curiosity I ran command kdesu /sbin/yast2 again, and did pstree. The tree showed that kdesu spawned sudo (and not su) This clarified why my password was accepted and not root password.

kdesu is very handy running KDE / GUI applications. You can also run applications as some other user (as in sudo) with -u switch. Say you want to run konqueror as user bozzo you can run kdesu -u bozzo konqueror from Run Command dialog (pressing Alt + F2).

Operating System in the browser.

In my previous post I had mentioned about Operating System being provided as service in future. In fact some steps towards it are already taken. YouOS has started a project which runs whole operating system inside your browser. This is the most innovative use of Ajax I have seen till date. It has sure made me rethink about my opinions about JavaScript.

Go "Find Bugs" in your software.

Life was going smoothly for me last couple of weeks. Hardly anything to
do. So I started reading articles and blogs and I ended up on a blog on
The Serverside which mentioned about the tool called "Find Bugs" I found
the name very interesting and downloaded that tool. The very first thing
I did was that I ran it on the code base of software I was working on. I
found it funny that Find Bugs reported around 700 bugs. Obviously I had
at least some faith in my code which made me believe that it was just crap!

Next thing I did was I started looking at each issue individually and
read through the explanation on why that thing was reported as a Bug.
Suddenly it started making a lot of sense to me. Believe me, 80% of them
were bugs! It efficiently detected some blunders like empty catch blocks
with catch(Exeption e) Assignments to static in non static methods of
class, Dead local store, Class casting problems, Possible Null pointer
dereference etc. And at last I found some real work to do! A lot of
work! It taught me a lot about good coding.

Find Bugs is a must have tool in your swiss army knife for code review.
It takes off a lot of hassles of manual code reviews. It comes with a
Eclipse plugin too which works well but the swing front-end provided by
Find Bugs is really good. It lets you address problems by category.
Where as in Eclipse you have to go to individual source file to find out
bugs in your code. Eclipse also has summary view to see the problems but
it looks really cluttered. Its a must use tool for all Java programmers.

My "dream" development environment

My boss keeps telling me to leave my laptop and get onto a desktop
machine. I agree, desktops are more powerful and rather sturdier than
laptops. But my laptop gives me freedom to work from anywhere. I can
work from home if i want (Well.. maybe at least sometimes! :D) I have
all the resources that i need wherever I go. But its really frustrating
when I try running heavy applications on my laptop. I feel I really need to get
on to a powerful machine. But then I loose all my freedom of working
from anywhere.

Recently I just gave this whole idea a thought! Well, what do i really
need is a powerful hardware! What if I can carry my whole Operating
System with me wherever I go? such that I use only the computing power
of the hardware but the operating system (and the whole environment) is
what remains the same. Then I came across a USB disk, one sent to me by
my Dad. The specialty with this USB disk is it actually uses memory
card. So that you can plug in any memory card into it and use it as a
pen drive. For long I have known a Linux distribution aimed at being
really small. Its called Damn Small Linux (DSL). It has also got XFace
windowing system and is really damn small in size. Major feature of DSL
is that it can start from within Windows or even within Linux. So what
has this all to do with my dream development environment?

Well, if I can put all these pieces together, I can build my "dream"
development environment! Say I have a memory card of 5G. I use my funky
USB Disk to connect that to my computer. I can install DSL or in fact
any other distribution on this USB disk. I can setup my whole
development environment on the same. Since I have got 5G which I suppose
is plenty of space, (Even if you dont agree) I can setup all the stuff I
want like Java, Eclipse, Browser etc. I just have to make sure that I
have driver for most of the hardware. I take this little USB disk to any
machine which can boot from USB (Not necessary if I install DSL), I boot
to my very own portable development environment!

Sounds amusing isn't it? But reading or writing stuff to this memory
card maybe very slow and loading, saving applications can also be
considerably slow. But I wouldn't bother to compromise on that if I have
so very portable environment with me. Surely comfortable and easy to
carry. Well only thing which can bother me now is my USB disk crashes. I
think we are not very long from an era where computing power will be
provided like service, and your software (my USB disk in this case) will
be something you store on Internet. The computer will be a very light
front-end on top of all these services.

Why do we get infected...

Whats the golden rule to stay away from viruses? What do you do to make
sure you dont get infected? Dont install software from untrusted
source, run Antivirus, dont execute attachments unless they are from
trusted source or without scanning them through Antivirus etc? Do you
think this will ensure that you would never get infected? Answer is No!

I personally dont prefer running antivirus on my machine. It makes your
system crawling slow. But when you are running windows, you have to make
sure you run one. Ever wondered how these viruses get into your systems?
Most of the viruses get into the systems through vulnerabilities in
software. A vulnerability is a bug in software of which you can take
some advantage. Vulnerabilities are more vulnerable if they can be
remotely exploited. Vulnerabilities can be of any sort. But most
dangerous of them are some which can let attacker execute some code on
target machine by exploiting them remotely. And one major category of
the remotely exploitable vulnerability is "Buffer Overruns".

Buffer Overrun is a bug in which you allocate some buffer (memory) and
while filling it up you forget to check bounds of it. In a way allowing it
to overflow. The matter is made worse when the allocated buffer is used
to store some externally provided entity. Something which user provides
to software. This can allow the attacker to execute some arbitrary code
on the system running such software. Detailed description of how buffer
overruns work can be found in a classical paper published in a Phrack
magazine called "Smashing the stack for fun and profit"

Buffer overrun is a very common error which you can make while coding in
C, C++. 90% of such errors maybe exploitable. And if they are remotely
exploitable, then you have a virus targeted at that vulnerability.
Buffer Overruns are scarier than this! So, now what are the things that
can get you infected?

You tried to view an image sent you by your friend. The image was
specially crafted to exploit the problem with your jpeg rendering
engine. You get infected.

You tried to open short movie clip you downloaded from server, The movie
had exploited bug in your movie player's codec. You get infected.

You visited a web-page which exploited some javascript vulnerability,
You get infected.

You play safe and disable javascript and visit a web-page, page exploits
vulnerability with HTML rendering engine. You get infected.

Worst! you plug in your network cable, attacker sends a vulnerable
packet to you that exploits some vulnerability in your kernel networking
code. You get infected.

Whatever you do you cannot make sure you dont get infected. But sure,
prevention is better than cure! Either use antivirus or dont use Windows :D

Linux and Viruses

Why Linux doesn't have problem of viruses? Is Linux most secure Operating System? No! Believe me, there is nothing like perfectly secure in the world. But then why does Windows only suffer with such a huge problem of viruses?

The fact being that Windows is most widely used desktop OS. Most of the virus writers prefer to write viruses for windows. Writing virus for windows guaranties you mass spreading and infection. But that is not the only reason. How many people do you know always use windows with a non privileged account? Most of the Windows users log in with Administrator account. Dont be surprised if people laugh on a Linux / Unix user logging on as always as root. Windows does not follow good practices in this sense to discourage people for using Administrator account for normal usage. Where as there are certain Linux programs which will warn you if you are trying to run them as root. When you have logged onto your system with privileged user and when you are attacked by a virus you are bound to cause more damage to the system since they execute code as a privileged user.

Also there are certain areas where Windows does not follow best of the security practices. Its possible for a non privileged windows users to create or register DLL files and also cause some damage to the system. Where as in Linux anything else than root is largely restricted to their home directory and maybe some shared data. Enforcement of security by default is far better in Linux.

One of the important factor about Linux is it runs on large number of platforms. Linux as an Operating System is not standardized. All distributions are different from each other. There are various number of programs aimed at certain purpose. Where as Windows has standard mail client MS Outlook / Outlook Express,IE as the browser. Which makes virus writers life easy, as every Windows machine will have all these programs. Linux users use variety of mail client, so virus aimed at particular mail client may not harm all Linux systems. This makes virus writing for Linux rather difficult (But not impossible!).

Super User Do! (sudo)

sudo (Super User do) is an excellent and yet extremely simple utility.
sudo lets ordinary user to execute commands as super user (root)
Any user can execute privileged program using sudo command.
sudo will ask for that users password before executing any command
(Note it doesn't ask you root password) sudo can be configured to
gain finer control over who can execute what. People can be restricted
to execute certain commands such as halt, reboot etc. Yet they can
execute other commands without providing root password.

sudo configuration can also be stored in LDAP. This allows finer control
over distributed set of machines. All machines can read configuration from
LDAP server which localizes the configuration and management of sudo
for multiple machines.

Ubuntu comes with preconfigured sudo. Ubuntu has null root password
that is no one can log into root account directly. First user who registers on
the ubuntu box is given all privileges through sudo on that machine.
If the privileged person wants to login as root he can do so using
command

sudo su

The most important feature of sudo is that it enables logging. sudo logs
the commands you execute with the user name. And this is what many
people use it for.

This sounds very good approach in terms of security, but I don't totally
agree with it. Say there are 3 users who have got full privileges with sudo.
All of them can execute any command by providing their own password
to sudo. This means that we now have nearly 3 root equivalent accounts
and 3 root equivalent passwords to protect! Also the configuration is
rather trickier. When I configured sudo I started with exclusion principle.
Users were disallowed to execute certain programs, rest of the programs
can be executed by users. This is really dangerous as the programs which
you are allowing are virtually infinite set, so its always better to use inclusion
principle here. Only allow certain programs can be executed as root.
Maybe rest of the programs can be allowed to run as some less privileged
user or the same user (As pointed out by Milan :D)

All I can say its something very useful for people using it for personal
desktops or in the environment where there are limited users.

links:
Sodores Manual

Building cheapest gateway machine.

What would you like your gateway machine to have? I consider
most people will want at least following things in a gateway,

1. Firewall
2. DHCP Server
3. Easy administration tools
4. Monitoring Tools

And also there may be less essential things like

1. Caching DNS Server
2. Caching Proxy Server
3. HTTP Server

Most latest router products can give all above things out of the
box. But most of the routers are considerably expensive. Now
what if you have ran out of budget to buy such funky routers?
Well you dot need to worry! You can build a cheapest gateway
machine with cost as less as FREE!

There are plenty of Linux distributions (Open Source and
Commercial) available aimed for this purpose. These distros
provide out of the box services which a gateway or a server
machine might need.

Most of these distributions provide you excellent monitoring and
administration tools. Some have web based administration interface.
There are good bandwidth management tools available. Firewall is
conventional iptables / ipfilter based firewalls and also custom and
open source front-ends for configuring the same.

Major feature about these firewalls is that they also provide you Live
CD distributions. Configuration files can be saved to floppy discs
or also on the USB disks. You can also create your own custom CD
with configuration burnt on it. So, in case if your gateway machine
goes down, pickup this CD put it into other machine and then that
machine can readily start functioning as the gateway. Effective way
to handle fail-over! Most of these distros provide good support for
hardware (As Linux does anyway!) one can run it on any machine like
old Pentium machine to any new high-end machine. But you ideally
don't need a powerful machine. In case of Live CD distos, you don't
even need any hard-drive.

Some of the firewall distros are -
Devil Linux
IP Cop Firewall
Smoothwall
Coyote Linux
ClarkConnect Gateway/Server

Static Inner classes?

Ever tried serializing the inner class object with non serializable
outer class? We cannot do this! By default compiler adds a reference
to the outer class in the inner class. Remember! we need to access
some attributes of outer class at times. This explicit referencing helps
at that time. When you try to serialize the inner class object you will
get an error.

Now what do we do if we want an inner class object to be serialized?
There is a way to do it. This is where you can use static inner class!.
A static inner class doesn't get a reference to the outer class when
compiled. Its treated as any other standalone class all together. When
an inner class is compiled its actually compiled into a class file with
name OuterClass$InnerClass.class where as the static inner class gets
its own seperate InnerClass.class file. This also meas that you cannot
access OuterClass members from static inner class since we no
longer have any reference to the outer class.