Ceph - a long debug
Hello again! Yet a longer time between posts, but at least my colleagues on https://pouta.blog.csc.fi are holding the fort!
Generic disclaimer: Ceph Hammer/Jewel, CentOS7
Most of the work was done by our Ceph team, yours truly was partially involved.
Background
We've been using Ceph for a long time, and . . .
Resize problems
It's a while since the last blog. For some reason the technial blog posts stopped at the same time I took the product owner hat. Interpret that as you may.
Normal disclaimer: CentOS 7, RDO on OpenStack Newton
I can't resize!
Resizing and migrating VMs are some of the standard tools in the OpenStack arsenal . . . .
Rally and Tempest
Or: your cloud probably has problems
When you operate a complex system, like OpenStack, one of the most important things is to know the state of the system. What works, what's broken and what's broken for a known reason. Without this information it's really hard to make changes to the system.
I think pretty much all OpenStack installations have their own way . . .
Liberty to Newton
Sorry Mitaka
Here's my traditional OpenStack upgrade blog post. I usually write these after all of our major OpenStack upgrades.
Our stack: CentOS 7, RDO, Puppet + Ansible, Linuxbridges + VLAN, Ceph
We're a dinosaur who still has monolithic API nodes (Most services running on a pair of VMs), which is relevant for this procedure.
. . .Packets - Part 2
Some time back, I wrote a post about hunting down dropped packets on our network nodes. Here's a small follow up with some performance hints.
More problems
My happy glow after fixing the previous issue didn't last for long. Our network nodes had hit the next problem.
Our network nodes are virtualized for . . .
Logical Volume Management
Or: Paying the Price for Technical Debt
We have run OpenStack for a while. I think we went into production around Grizzly. Back then we were new to running production OpenStack systems (who wasn't?). This means we made some decisions back then, which we are still paying for.
Credit where credit is due: Almost all the real work described in this post was done by my . . .
Packets, Drop by Drop
I originally started writing this blog to document all the tricky bugs I encountered, and google couldn't answer.
My, did I ever find one. And it wasn't one of the issues that makes you go beat yourself for not figuring it out sooner.
Network, gross work
We had a simple change in our cloud infrastructure. We . . .
