In the “Developer Interview” series we talk to developers from some of the companies using Semaphore to find out how they work and share their insights with you. This time we chatted with Devon Noel de Tilly, QA Engineer at 500px, a photo community for discovering, sharing, buying and selling inspiring photography powered by creative people worldwide.
Tell us about your role and responsibilities at 500px.
I’m a QA Engineer at 500px. I look after our CI setup which runs on Semaphore, maintain our suite of automated tests, run manual tests in a pre-production environment, manage deployment, find and fix bugs, and work closely with our developers and sysadmins. My role is something of a hybrid between testing, automation / DevOps and sometimes development. I work primarily on our web apps and supporting services.
Your products serve an established community of users. What does your workflow typically look like at this point, from an idea until it’s shipped?
A lot of what we do at 500px comes out of ideas from our developers and designers. We hold monthly hack days, and a lot of our new features have come from that, as well as just discussions amongst team members. Of course we get ideas from our users, what they want and what is important to them. And sometimes our executives will come to the team with what they’d like to see.
We hold monthly hack days, and a lot of our new features have come from that.
Usually after someone gets an idea, they’ll make a small proof of concept, often for hack day but sometimes they’ll just pitch it to anyone who’ll be involved in making it a reality. If people like the idea and are on-board for it, then the developers or designers will start working on it. Our technical staff is separated into teams at 500px, so we have a web team, a platform team, an iOS team, etc. that are all largely self-contained, but collaborate closely when necessary. Usually a project will be scoped to one team, with maybe a bit of support from some of the other teams, so planning is generally pretty minimal.
After a team has decided they’d like to do something, and our project managers have split the work up into tickets and slotted it in for a sprint, it’s largely up to the individual teams how they want to proceed. For example, our platform team likes to do things in small bite-sized tickets and work either in pairs or alone, hammering out a new feature a little bit at a time, while our web team prefers to get together in a big room with their laptops and all work on a new feature together.
Once the initial design and development is done for a ticket, the developer will open a pull request, and it’ll go through code review, where other designers and developers on the team will point out flaws and make suggestions for improvement. After code review, it’ll enter QA, where we’ll run the full suite of automated tests through Semaphore against the branch, as well as do some manual and load tests in a pre-production environment. If there are failed specs, or if we find bugs through manual testing, QA will gather any relevant logs, errors and steps to reproduce, sometimes make suggestions, reject the ticket, and developers will work to fix it. After a ticket has been accepted by QA, we’ll merge it into the master branch and deploy the code to users.
Did you have times when technical debt slowed down this process? If so, how did or do you overcome it?
Yeah, technical debt I think is a problem for any code base once it’s become sufficiently large. We try to be proactive about it as much as possible, but there have been times when it’s come back to bite us a bit, or when its slowed down our development cycle for sure.
It’s not so much a problem we’ve overcome as a problem we’re always overcoming. It’s something we have to be constantly mindful of. How we deal with it is again largely on a team by team, or even individual by individual basis. We use Code Climate to analyze our code base, and that helps us find some things we’ve overlooked. Sometimes members of the team who find themselves with time will spend it combing over the code base and addressing some of our technical debt.
For me, when I started at 500px, a big part of the technical debt we had at the time was the state of our spec suite, which wasn’t really being maintained and was a bit of a mess.
What was your main guideline in getting to a green build?
I wouldn’t say I really had one main guideline, unless you count the vastness of the internet as a whole. I drew from a variety of resources. But I approached the problem knowing where our weaknesses were (mostly race conditions and inefficiently loading test resources), and keeping best practices in mind, found our pain points and corrected them a little at a time.
I used Relishapp’s Rspec documentation and Betterspecs as a guideline for best practices, as well as a variety of great tech blogs like Airbnb’s blog and Semaphore’s own blog.
There’s an idea that if you care about something, and you want to work on it, then work on it and its yours.
Given the amount of tests in our suite and the amount of race conditions that needed addressing, I also realized early on that I’d have to make a couple compromises. I found a great little gem called respec that we use to rerun failed specs exactly once at the end of our test build, and that helped us a lot in the beginning.
Do you release new features to all users immediately?
Often but not always. Small new features and bug fixes we generally push to all users right away, but some of our bigger features we’ll protect behind rollout flags and rollout internal first, so we can all get acquainted with it and find any problems or things that we might’ve overlooked. If something has the potential to really hammer our API, and we’re not sure how it’ll preform, we’ll sometimes do a staged rollout to percentages of our users also. And we also do A/B tests when we’re unsure how users will respond to a new feature or layout.
How do you manage the code for your main application on 500px.com? Is it pretty much a classic MVC web app, or you’ve branched out into something more custom?
At one point our main site was basically one huge Rails app (which we call The Monolith internally), but as our platform got bigger it became harder and harder to maintain. So we’ve started splitting everything up into a microservices architecture. The idea is that any logically distinct part of code should be self-contained.
Ruby, and also Rails, are designed primarily to be easy to use and pleasant for the programmer, that’s a big philosophy in the Ruby and Rails community. That things should Just Work. Which is awesome, but once you start getting into big computation on the backend, it can be pretty slow and inefficient. So we started splitting out some of our really slow code, our pain points, into their own Go services, which is much more performant. Things like search and photo conversion.
What’s your favorite part in working at 500px?
There are a lot of things I like about working at 500px. The opportunities to work on things that interest me, and to grow my skills and knowledge. The great benefits and team outings. The people I work with, they really make it a great place to work. But if I had to pick something, it would probably be the attitude towards ownership of the product. The autonomy I’m given is great, I don’t feel like I have people looking over my shoulder and telling me what to do all the time.
At 500px, there’s an idea that if you care about something, and you want to work on it, then work on it and its yours. That we own the things we’re working on, that we’re ultimately responsible for the things we decide are important. Of course there are disagreements sometimes about what we should be working on, of course there’s pressure from other team members to focus on something, but that’s the ideal we’re working towards. And I feel like I can really get behind that message.