MySQL Questions

Skills Questions

1. Why are SQL queries so fundamental to database performance?

This is the one question which a DBA should have an answer to. If they can’t answer this question, they’re unlikely to be a good candidate.
After a MySQL server is setup and running, with many of the switches and dials set to use memory, and play well with other services on the Linux server, queries remain an everyday challenge. Why is this?
SQL queries are like little programs in and of themselves. They ask the database server to collect selections of records, cross tabulate them with other collections of records, then sort them, and slice and dice them. All of this requires MySQL to build temporary tables, perform resource intensive sorts and then organize the output in nice bite size chunks.
Unfortunately there are many ways to get the syntax and the results right, yet not do so efficiently. This might sound like a moot point, but with modern websites you may have 5000 concurrent users on your site, each hitting pages that have multiple queries inside them.
What makes this an ongoing challenge is that websites are typically a moving target, with business requirements pushing new code changes all the time. New code means new queries, which pose ongoing risks to application stability.

1. Indexes – too many, too few; what’s the difference?

Indexes are very important to the smooth functioning of a relational database. Imagine your telephone book of yore. I can look up all the people with last name of “Hull” in Manhattan because I have the proper index. But most yellow pages don’t include an index for *first* names even though they might occaisionally come in handy, for example with the names “Star” or “Persephone”.
You can imagine that, if you had a phone book which you maintain and update, everytime you add or remove a name you also have to update the index. That’s right, and the same goes for your relational database.
So therein lies the trade off, and it’s an important one. When you are *modifying* your data, adding, updating or removing records, you must do work to keep the index up to date. More indexes mean more work. However when you’re looking up data or *querying* in SQL speak, more indexes mean more ways of looking up data fast. One more trade off is that indexes take up more pages in your phonebook, and so too they take up more space on disk.

1. Backup & Recovery – explain various types & scenarios for restore

Backups come in a few different flavors that the DBA should be familiar with.
Cold backups involve shutdown down the database server (mysqld) and then backing up all the data files by making a copy of them to another directory. To be really thorough, the entire datadir including binlogs, log files, /etc/my.cnf config file should also be backed up. The cold backup is a database in itself, and can be copied to an alternate server and mounted as-is.
Logical backups involve using the mysqldump tool. This locks tables while it runs to maintain consistency of changing data, and can cause downtime. The resulting dump file contains CREATE DATABASE, CREATE TABLE & CREATE INDEX statements to rebuild the database. Note the file itself is not a database, but rather a set of instructions which can tell a MySQL server *HOW* to reconstruct the database. Important distinction here.
Hot backups are a great addition to the mix as they allow the physical database data files to be backed up *WHILE* the server is up and running. In MySQL this can be achieved with the xtrabackup tool, available from Percona. Despite the name, it works very well with MyISAM and InnoDB tables too, so don’t worry if you’re not using xtradb tables.
There are a few different restore scenarios, and the candidate should be able to describe how these various backups can be restored, and what the steps to do so would be. In addition they should understand what point-in-time recovery is, and how to perform that as well. After restoring one of the above three backup types, the DBA would use the mysqlbinlog utility to apply any subsequent transactions from the binary logs. So if the backup was made at 2am last night, and you restore that backup, the mysqlbinlog tool would be used to dig up transactions since 2am, and apply them to that restored database.

1. Troubleshooting Performance

Since this is an ongoing challenge with relational databases, a good grasp of it is crucial. One way to challenge the candidate would be to describe a recent performance problem you experienced with your infrastructure, and ask them how they would go about resolving it.
If they struggle with the particulars of what you ran into, ask them to describe a big performance challenge they solved, what the cause was, and how they performed analysis.
Typically, first steps involve mitigating the immediate problem by finding out what changed in the environment either operationally or code changes. If there is a bug that was hit, or other strange performance anomaly, the first stop is usually looking at log files. MySQL server error logs, and the slow query log are key files. From there, analyzing those files during the timeframe where problems occurred should yield some clues.
You might also hope to hear some comment about metrics collection in this discussion. Tools such as cacti, munin, opennms, or ganglia are invaluable tools for drilling down on a past event or outage, and sifting through server stats to find trouble.

1. Joins – describe a few kinds and how the server performs them

A basic understanding of INNER JOIN and OUTER JOIN would be a great start. A simple example might be employees and departments. If you have four employees and two departments, an INNER JOIN of these tables together will give you the departments employees belong to. Add another employee without assigning her to a department, and the inner join won’t display her. Further adding a new department which doesn’t yet contain employees won’t display either. However performing an OUTER JOIN will give you those matches with null in the department field, and null in the employee field respectively.
Thought of with another example, take a credit card company. One tables contains cardholders identity, their number, address, and other personal information. A second table contains their account activity. When they first join, they don’t have any monthly statements, so an INNER JOIN of cardholders with statements will yield no rows. However an OUTER JOIN on those two tables will yield a record, with a null for the statements columns.

1. Disk I/O

Disk performance should be an ever present concern to a DBA. So although they don’t need to be a storage specialist, they should have a working knowledge. Ask them about RAID versions, mirroring versus striping, and so forth. Mirroring combines two disks as a unit. Every write is duplicated on both disks. If you lose one disk, you have an immediate copy. Like a tandem truck that has spare tires running in parallel. Lose one, and you don’t have to pull over immediately to replace it. Striping spreads I/O over multiple disks so you on the one hand increase throughput linearly as you add disks. That’s because you have more disks working for you. At the same time you increase risk with each new disk you add, because the failure rate is then the sum total of all those disks.
For relational databases the best RAID level is 10, which is striping over mirrored sets. You use more disks, but disks are cheap compared to the hassle of any outage.
If you’re deploying on Amazon, your candidate should be familiar with the Elastic Block Storage offering also known as EBS. This is virtualized storage, so it introduces a whole world of operational flexibility. No longer do you have to jump through hoops to attach, add or reconfigure storage on your servers. It can all be done through command line API calls. That said EBS suffers from variability problems as with any other shared resource. Although Amazon guarantees your average throughput, the I/O you get at a given time can swing wildly from low to high. Consider Linux software RAID across multiple EBS volumes to mitigate against this.

1. How would you setup master/slave & master/master replication?

A basic replication setup involves creating a full dump of the primary database, while it’s tables are locked. The DBA should capture the master status, logfile & position at that time. She should then copy the dump file to the secondary machine & import the full dump. Finally the CHANGE MASTER TO statement should be run to point this database instance to it’s master. Lastly START SLAVE should be issued. If all goes well SHOW SLAVE STATUS should show YES for both of these status variables:
Slave_IO_Running: Yes

Slave_SQL_Running: Yes
Master-master replication is similar, except one additional step. After the above steps have run, you know that your application is not pointing at the slave database. If you’re not sure, verify that fact first. Now determine the logfile name & position on the slave with SHOW MASTER STATUS. Return to the primary box, and run the CHANGE MASTER TO command to make it slave from the secondary box. You’ve essentially asked MySQL to create a circular loop of replication.
How does MySQL avoid getting into an infinite loop in this scenario? The server_id variable must be set, and be unique for all MySQL instances in your replication topology.
For extra credit, ask the candidate about replication integrity checking. As important as this piece is to a solid reliable replication setup, many folks in the MySQL world are not aware of the necessity. Though replication can be setup, and running properly, that does not mean it will keep your data clean and perfect. Due to the nature of statement based replication, and non-deterministic functions and/or non-transactional tables, statements can make their way into the binary logs, without completing. What this means is they may then complete on the slave, resulting in a different row set on the same table in master & slave instance.
Percona’s pt-table-checksum is the preventative tool to use. It can build checksums of all your tables, and then propagate those checksums through replication to the slave. An additional check can then be run on the slave side to confirm consistency, or show which rows & data are different.

1. How are Users & Grants different in MySQL than other DBs?

Creating a grant in MySQL can effectively create the user as well. MySQL users are implemented in a very rudimentary fashion. The biggest misunderstanding in this area surrounds the idea of a user. In most databases a username is unique by itself. In MySQL it is the *combination* of user & hostname that must be unique. So for example if I create user sean@localhost, sean@server2 and sean@server3, they are actually three distinct users, which can have distinct passwords, and privileges. It can be very confusing that sean logging in from the local command line has different privileges or password than sean logging in from server2 and server3. So that’s an important point.

1. How might you hack a MySQL server?

This is a good opportunity for the candidate to show some creativity with respect to operations and Linux servers. There are all sorts of ways into a database server:
a.bad, weak or unset passwords

b.files with incorrect permissions – modifying or deleting filesystem files can take a database down or corrupt data

c.intercepting packets – could reveal unencrypted data inside the database

d.unpatched software – bugs often reveal vulnerabilities that allow unauthorized entry

e.moving, disabling or interrupting the backup scripts – a possible timebomb until you need to restore

f.DNS spoofing, could allow login as a different user

g.generous permissions – may allow an unprivileged user access to protected data
There are endless possibilities here. Listening for creative thinking here, reveals how much that person will think thoroughly and effectively about protecting your systems from those same threats.

1. Brain teasers, riddles and coding problems

Google for a long time was a fan of these types of tests at interviews, but I’m not at all. For one thing you filter for good test takers, and for another the candidate has no resources – either books or the internet at their disposal.
Why not instead ask them to tell a story. Storytelling conveys a lot of things. It conveys a bit of teaching ability, which extends far beyond internalizing some multiple choice questions. It tells you more about their personality, which as I’ve said is very important. It shows how they solve problems, as they’ll take you through their process. And gives them an opportunity to tell you a real world triumph they presided over.