In my last blog we saw how to manage the private cloud, and now the unthinkable happend, suddently our management system called FiFo stopped working all by it self...When clickking on the tabs it does not display any content, and below is a screenshot of what the js console reports:
Fortunately the underlying SmartOS and all the Virtual Machines keept on working as if nothing happend at all.. But now our users are unhappy because they can't manage their Virtual Machines through a GUI, I think the below picture tells more than a thousand words :-(
That's me on the right, trying to comfort one of our dearest customers...
Obviously we sought help in the FiFo googlegroups, and the guys there are pretty fast to respond and offer suggestions. We started to debug as per their suggestions and this checklist, but as we soon discovered, it was an insurmountable task due to the complexity of FiFo, even the log files did not give any hints as to the root cause of the problem.
So we abandoned the attempts to recover FiFo and decided to go for a new install, but unfortunately the FiFo database would also be overwritten by a new install, so this was not a viable option as we have a lot of user information in the database. Subsequently the new install attempt was also abandoned.
As a last resort we followed the upgrade guide from a-z, the upgrade was successful - but FiFo refused to restart after the upgrade, and our customers remains unhappy, what to do? Obviously FiFo is time being not the ideal way for us to pursue, we have to look into alternatives.
So what are the alternatives? well, if we look in the market there are OpenNebula, OpenStack, CloudStack and obviously SmartDataCenter. SmartDataCenter is not really an option, since it costs a lot of money, and we can't justify such expenses at the moment. What concerns the first three alternatives there are no support for the SmartOS hypervisor and there are no current plans to support SmartOS.
We could contribute to the efforts by Hendrik Volkmer to port openStack to SmartOS, but completion is no where to be seen as people also have dayjobs to care for and the task is complex, and then there are all of the subsequent maintenance to secure continued compatibility.
OpenNebula, OpenStack, CloudStack all support the AWS EC2 API, which means that we could write our own web service that runs on SmartOS and is EC2 compatible. Again the task is complex, and then there are all of the subsequent maintenance to secure continued compatibility.
We could also contribute to the FiFo project - but again, it's complex and FiFo tries to be omnipotent just like the other four alternatives, and we don't have time to wait for a workable product when our customers demand it now and not tomorrow!
Stay tuned for the next exiting part in this series, to see how we solved the problem.