IEBC Tech Kenya

Google statement on Japak GIS

“Google provided technology and tools that the IEBC is using on their official website to ensure Kenyans have access to important information. Google, working through our partner Japak GIS, trained IEBC and provided technical support for them to make the results available on their website, including visualization on maps. Google was not involved with results collection, transmission, tallying, or storage. Google technology has helped provide open and accurate information to voters in elections around the globe, including in France, Italy and Mexico, among others.”

Dorothy Ooko Communications & Public Affairs, East & Francophone Africa

IEBC Tech: Audit Results

There’s an interesting piece in the Standard where the person who was hired to do an audit of the failed IEBC RTS system speaks about what happened.

Also, note that they say that the “IEBC tallying system was attacked twice on poll day”.

“Attacked” does not mean “hacked”. In an earlier report here, you’ll note that the servers were defended successfully.

Main points:

Multiple companies involved, duplicating effort, added vulnerabilities
Systems by JapakGIS and IFES, which had failed weeks earlier, were still used
Delays in the delivery of phones had a ripple effect
It supposedly cost 96m Ksh ($1.1m) for the RTS system

Full article after the jump below.

A Clear Definition of the IEBC Tech Failure

This information comes from members of a team that worked with the RTS system. The following is in his words:

RTS

As you stated – RTS was a slick design, it was a system that was to run on 339 servers across the country and over 33k phones and at least 26k users logged in to the production system. It was a shame that issues outside the main RTS software denied it the limelight. The visualization and transmission aspects were not part of the RTS system and thus the RTS system comprised of:

mobile phone software – a J2ME application
the web service processing the request – a Servlet running on Glassfish
Memcache to cache data that was not changing
the database – running on Mysql

All of which were based on tried and tested open technologies.

The Failure

The truth is that around 8PM Monday is that the /var partition on the provisioning server (running CentOS not Windows) got filled and thus the underlying RDBMS failed. It was a shame because there was so much space on that server but not in the correct (needed place). I can state that there was no hacking (nothing points to it). I can also state that RTS was not creating files and thus the partition was not filled by RTS data but rather by Mysql binary logs that were being generated in situ due to database replication which was switch on. Thus this meant that if the provision server went down - no new logins and requests for candidate data for that polling station could not be serviced. However, those individuals who had logged in at least once before in accordance to the procedure were able to send results to the other servers that were up. This explains the “slow down” experienced after the provisioning server went down.

The following are some figures of the success the system achieved when it was up:

16,617 polling stations had reported the presidential race,
8,130 polling stations had reported the governor
8,229 polling stations had reported the senator race,
11,020 polling stations had reported the national assembly race
8,613 polling stations had reported the women rep race
9,228 polling stations had reported the county assembly ward reps.

It was a shame that the technical team could not deduce that the issue that took down the server was a simple problem of disk space because the team spent time searching for the problem elsewhere (concurrency, number of threads being spun, max number of connections on mysql). What makes the team sad is that RTS missed it’s the golden window to shine due to non programming errors and the time wasted before we were able to detect that trivial issue for that matter.

Other issues that have been eclipsed due to the nature of the of the issues experiences were procedural and process issues namely –

The delivery of phones and configuration of the same was not done in time. This had a negative effect on the rest of the process.
Presiding officers should have logged in, downloaded the candidates and sent a dummy result on the eve of the election to prove it works. This was not done in all polling stations.
The issuing of usernames and passwords was not as in some places – users were still asking for passwords to be reset 2 days after the election.

The late delivery of the visualization software (election day) ensured that no real stakeholder testing happened and thus the spoilt, rejected and disputed vote figures were inflated by a factor of 8 for the presidency (8 candidates) due to a join error that could have been detected if the visualization was delivered on time. I

So while RTS the system failed the software to transmit and receive data was fine and stuff around it messed it up and still believe that it is a great piece of software that run under conditions that were not exactly optimal for its correct performance.

Pentesting the IEBC Tech side

@Wanjiku has a new post on her blog titled, “Was the IEBC network compromised, an insiders view….” (now linking to cached version as the original was deleted), go there for the full story. The long and short of it is that the system could be compromised, but was defended by the guy who actually figured it out. Well done!

A couple exerpts:

So the first step was to get a VPS servers and upload a Malware client and prep a Malware server, tested it on Virtual Machines at the comfort of my home for around two weeks, until i got the dropper working right. I needed to make sure I own atleast one workstation, then propagate my attack from there.

Am not going to bore you on how I got my malware into the IEBC internal network and I had a lot of time scanning and probing for services on this infrastructure, but I was in, and fully in change of the machine I had hacked into. Luckily this workstation I had got into wasn’t getting switched off at night, but I worked hard to make sure that in case the owner decides to reboot, my code would still connect back via port 443 to my VPS server in UK. A few days I was able to update my Malware server, and I was ready to test a reboot, and I ran it.

Further along:

Just by luck, I bumped into an email, via *.PST, that had an attachment with information about the RTS, and the name of the machine in IEBC infrastructure and also the one in BOMAS and DR. I wont share the IPs of these initial machines, but the systems hostname as known by the network was results.iebc.or.ke.

Breaking into this was easy, via php vulnerabilities and also that it was still using simple passwords, and also the developers common mistakes of leaving SQL dumps on the webroot. Got in added a user on the system by end of February. Then I started going for the system at BOMAS. Lucky me, it was windows, much more easier. By this time it was almost at the start of March, and an IEBC official called me and told me to send my CV over for recruitment as a contractor.

Defending the system at IEBC:

Now, due to sensitivity of issues, I will not copy the incident report here or with any details, but I wish to specify we had two attacks, all of them ran on the initial step, reconnaissance and by the time scanning had started, I had hammered the attack, and all was okay.

…

So, a lot of rumors spread by some countrymen, meant to bring up chaos, (sorry it dint work, Kenyans, we are peaceful), saying that the systems were hacked were completely ridiculous, since we monitored every traffic, any binary and service that ran on both Servers and the major machines, We also monitored any laptop, computer that got hooked on the network, for any type of polymorphic type of attack, network scanning or a form of Advance Persistent Threat etc

It was a hell of a week, but we did it.

Why do I ask these questions?

[I posted this on Tuesday, but reposting so those questioning my motives can find it]

The reason I’m trying to figure out the data flow (of the IEBC RTS system) and parties involved, is because I’m curious as to what the IEBC’s technical difficulties have been today. There hasn’t been much information on the actual details behind their inability to get provisional polling data from around the country, from the IEBC, or any of the corporate partners.

I’m looking for clarity on something that should already be documented, but which I can’t find. Something that should be openly communicated, but hasn’t been. We all benefit from a transparent electoral process, and this is a key process for all of us to understand.

If it’s out there already, please shoot me the link.

Additional Thoughts from today (3 days later)

If you don’t know me from my avatar, I’m Erik Hersman (@WhiteAfrican on Twitter). Feel free to ping me with any thoughts comments.

This all started with me genuinely being curious as to how the system worked and who was involved in the steps that data would take going from the polling station to the IEBC API. I searched online, but only found a couple articles, then I tweeted some questions and had some people on Twitter send in some information and further links.

At that point I didn’t have enough information to write anything, nor to speculate. Instead I started this Tumblr as a way to aggregate the links, comments, sources and all other information I could get on the RTS system. I did it openly and online so that more people could find it and help answer some of the questions, and so that there would be a centralized place to find the some facts about the system.

So, why now? Why not wait a week until the process is over? It’s been very troubling for me to see people speculating on social media about the IEBC tech system, claiming there have been hackers and all types of other sorts of seeming misinformation. Those of us in the technology space were looking to the IEBC and its partners for the correct information so that these speculative statements could be laid to rest. I deeply want the legitimacy of this election to be beyond doubt. The credibility of the electoral system was being called into question, and clear, detailed and transparent communications were needed in a timely manner. These took a long time to come, thus my search.

Interestingly, Safaricom came out with a very clear statement on what they were responsible for and what they did. Google was good enough to make a simple statement of what their responsibilities were on Tuesday. both of these companies helped answer a number of questions, and I hoped that the other companies would do the same. Even better would have been a clear and detailed statement from the head of IEBC’s ICT department to the public. Fortunately they did provide some general tech statements, claimed responsibility, refuted the hack rumor, and made the decision to go fully manual.

(Sidebar: it should be noted that the IEBC’s RTS system was a slick idea and if it had worked we’d all be having a much more open and interesting discussion. Remember though, the RTS system was an add-on for additional transparency and credibility, and that the manual tally we’re doing now was always going to happen and was the official channel for the results.)

My assumption was that since this was a public service for the national elections, that the companies involved would be publicly known about as well. This wasn’t true, it took a while asking around to get an idea of who did what. On top of that, In a country that has been expounding on open data and open information, I was surprised to find that most of the companies didn’t want to be known, and that a number of people thought it was a bad idea to go looking for who they were and what they did. I wasn’t aware that this information was supposed to be secret, in fact I assumed the opposite, that it would be freely announced and acknowledged which companies were doing what, and how the overall system was supposed to work.

I’ve spoken directly to a number of people who are very happy that I’m asking questions and putting the facts I find in an open forum, and some that are equally upset about it. Much debate has been had openly on Skunkworks and Kictanet on it this, and when we debate ideas openly we fulfill the deepest promise of democracy. My position remains that this information should be publicly available, and the faster that it’s made available, the more credible the IEBC and it’s partners are.

Understanding the infrastructure, what went wrong, and some more questions

Update: This information came from a source sitting in the server room at Bomas of Kenya. Like any information here, if you have verifiable facts that dispute it, do get in touch so I can update it.

The IEBC servers:

20 Processors, 128GB RAM, 18 TeraByte Hard discs.

Data:

All the country’s data was text-based and was less than 200MB in a My-SQL database. The entire system was running on a Windows Server stack.

The JapakGIS system was being hosted by Google and so it resided on Google servers. That is where vote.iebc.co.ke is pointed at.

What do we currently understand of what went wrong?

One of the main system failings was on the visualization layer made by JapakGIS company, an approximately 1 year old company that was put together by an ex-ESRI East Africa employee. JapakGIS is in charge of visualising info in the database from the gadgets in the field.

There was only one program running on the server, the webserver xamp.

Questions to ask:

How did JapakGIS get awarded the tender?
Was there ever an audit of the JapakGIS system before the elections, as we know it failed in the tests?
In 2010, Next Technologies was the company contracted to do the system for the Referendum and it worked flawlessly. But they got kicked out, citing problems. Can we know what those issues were?
Were the mock elections held on Sunday Feb 25th ever visualised?