[ubuntu-cloud] Cloud debugging

[Marius Seritan]

Most of my code nowadays is ruby, by integration testing I mean testing at the  controller or API/request level. I also deployed java in data centers and EC2. The usual debugger works in production on both platforms but I do not see how you could use it without impacting your customers. Staging of course is perfect for debugging.

If there is a problem in production I revert to the previous version code and database, if needed. It is crucial to be able ro revert fast, the standard ruby way is to have separate folders and softlinks. Then you download the logs and either fix the bug or add more logging and deploy to just one server if you really can't figure it out. Better to not paint yourelf in that corner too often :-(

So handling production problems is more about the process than the tools.

Curious to hear other opinions :)

Marius


On Feb 8, 2011, at 6:49 PM, Brian Fromme <brian.fromme at canonical.com> wrote:

> Thanks Marius,
> 
> This is excellent advice.  So, on your integration-level testing, for example, what happens when a bug is seen there?  Does any debugging go on at the cloud, or is it all done in a simulated local environment while looking at various log files?
> 
> I know for me, I would want to have some sort of tools in the staging environments that I could use for diagnostics and debugging of the real execution.
> 
> thanks,
> Brian
> 
> On Feb 8, 2011, at 5:23 PM, Marius Seritan wrote:
> 
>> Brian,
>> 
>> I handle the complexity of the cloud as the next level of complexity in the application. I do as much testing as I can at the unit level, then integration, then QA on staging. If the code makes it to production with a real bug all the bets are off. Generally what ever error is left gets reported by  CPU monitoring on each system (monit) and through centralized error reporting (hoptoad). Deploying in stages also helps to contain what ever problem appears.
>> 
>> As the code matures I add custom munin plugins to monitor specific processes and see trends.
>> 
>> Marius
>> 
>> -- 
>> Marius Seritan                        http://www.de-co-de.com
>> mseritan at de-co-de.com      (415) 409-9277
>> 
>> 
>> 
>> 
>> 
>> 
>> On Feb 8, 2011, at 3:43 PM, Brian Fromme wrote:
>> 
>>> Hi,
>>> 
>>> I'm researching tools for cloud development.  Can anyone comment on tools/methods for debugging their scalable application in the cloud?
>>> 
>>> thanks,
>>> Brian
>>> 
>>> 
>>> -- 
>>> Ubuntu-cloud mailing list
>>> Ubuntu-cloud at lists.ubuntu.com
>>> Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-cloud
>> 
> 
> 
> -- 
> Ubuntu-cloud mailing list
> Ubuntu-cloud at lists.ubuntu.com
> Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-cloud